Rating vs. Personality Quiz based Preference

Paper: A Comparative User Study on Rating vs. Personality Quiz based Preference Elicitation Methods

Written by: Rong Hu and Pearl Pu

Comments: Jacob

Summary:

This paper compared a rating based recommender system with a personality quiz based recommender system. Both systems rated movies and made suggestions to the users. The rating based system was called MovieLens and the personality quiz system was called Whattorent. There were 30 participants that took part in this study.

The graph shoes the differences between all of the participants. The recommendation system interface for MovieLens had to be changed up a little bit to match with Whattorrent in order to only recommend one movie at a time. The way they tested the similarity and which system the user likes better was by:

1. Perceived Accuracy: measured by the participants' ratings to the recommended movies and their responses to the post-questionnaire at the end of the study.
2. User Effort: calculated how much time people contemplated their answers and how much effort the user feels that they have used by doing the rating or quiz questions.
3. User Loyalty: asked the participants if they would be likely to recommend a system to their friends.

For perceived accuracy the the participants were asked a question stated positively, and the same question stated negatively. In this section Personality quiz-based system scored higher than the rating system.

This graph shows the result's of the users'ratings on recommended movies. The people who were given recommendations through the personality quiz-based system were more interested in the movies recommended to them then the rating based system.

The graph above shows the effort the participants felt like they were giving when taking part in either system. Most users felt that the rating system took more effort than the quiz system because they didn't feel like they could accurately evaluate a movie that they hadn't seen in a while. They couldn't remember the plots of all of them, and when other people only gave the movie an average rating they weren't sure how to rate them.

This last graph shows how likely the participants would be to recommend the system to their friends to use. The personality quiz-based system had more loyalty then the rating system did. This paper concluded that the personality quiz-based system has the potential to be a great tool and an alternative to the existing methods.

Discussion:

I think this is rather interesting because people are always trying to make recommender systems better so that users will be happy with what movies they are recommended. I think that if the personality quiz is going to have better accuracy, and the users feel like they don't have to much effort into it then they will probably use it more. Since they couldn't really say which system was the best I think that the future may hold a better combination of the two systems for users to use.