The Strengths and Weaknesses of Ratings Online.doc

The Strengths and Weaknesses of Ratings OnlineNathanaël CorreSubmitted in partial fulfilment of therequirements of Napier Universityfor the degree ofMasters of Science in Multimedia TechnologySchool of ComputingSeptember, 2002Authorship declarationI, Nathanaël Corre, confirm that this dissertation and the work presented in it are my own achievement. Where I have consulted the published work of others this is always clearly attributed;Where I have quoted from the work of others the source is always given. With the exception of such quotations this dissertation is entirely my own work; I have acknowledged all main sources of help;If my research follows on from previous work or is part of a larger collaborative research project I have made clear exactly what was done by others and what I have contributed myself.I have read and understand the penalties associated with plagiarism.Signed: Date:15 August 2002 Matriculation no: 99077221AbstractRatings are increasingly used on the web and they help filtering an abundance of products and information. This project investigates the nature of users' rating and their utility in the study case of an online collaborative questions & answers database about the Middle East conflict. In other words, it deals with information management according to users' interaction with the database.In its first part, this report looks at related works in the areas of computerised recommendation system, the use of rated reputation online, and the structure of a questions and answer database (chapter 1). Then, psychology concepts that could be useful for understanding the way users rate online and use ratings´ results are investigated. These concepts concern decision making and social influence on judgement. This information is also meant to provide hints for the design of the collaborative system (chapter 2).The second part (chapters 3 and 4) focuses on the design, the development, the implementation and testing of See through, a collaborative questions & answers system accessible through a web interface. The testing phase involved one database about the Middle East conflict. This was targeted at any adult person interested in the topic and it was advertised in news groups and amongst friends and relatives. This prototype was developed in order to assess the act of rating and its implications: to see how it influences users choices and appreciation. In order to reach this goal the integration of rating and interactive features (submission from users, freedom to manipulate data) was achieved by using MySQL database along with the scripting language PHP. The combination of the two allowed the creation of interactive web pages (HTML). The project was then evaluated through an online survey and data collected on the database. Ratings' weaknesses mentioned in previous research were confirmed. The distribution of mean rating tends to be higher in high ratings and users tend to rate what is already rated. On the positive side, it was found that compliance to the norm when rating does not work much in that specific environment (See Through) and that ratings can act as social facilitators. ContentsAuthorship declaration2Abstract3Contents4List of tables and figures6Acknowledgements7Acknowledgements7Introduction81Chapter 1: Ratings online and collaborative filtering91.1Introduction: Why use ratings?91.2Examples:91.2.1Le Monde91.2.2D (Cnet)101.2.3Action Forum101.2.4Ebay121.2.5Epinions121.3The context: Virtual community131.4Information filtering in a virtual environment141.5Related research projects151.5.1MovieLens151.5.2Group Lens161.5.3Answer Garden211.5.4Identity and reputation221.6Synthesis:262Chapter 2: The process of decision-making and different forms of social influence272.1Introduction272.2Decision-making: adaptations, heuristics and biases282.2.1Adaptive decision-making282.2.2Representativeness302.2.3Availability302.2.4Adjustment and anchoring312.3Group pressure and social influence312.3.1Social facilitation, social impairment and arousal312.3.2Social loafing322.3.3Conformity and compliance332.3.4Deindividuation352.3.5Altruism and helping behaviour352.4Conclusion363Chapter 3: Prototype design and research framework383.1Aims, objectives and the chosen method to reach them383.1.1Aims:383.1.2Objective383.1.3Method383.2The requirements393.2.1Specifications393.2.2Functional Requirements:393.2.3Non-Functional Requirements403.3Risks and potential benefits413.3.1Potential benefits:413.3.2Risks413.4The incentives423.5The design (according to the requirements)433.5.1The technology433.5.2Description of the database:463.5.3The website: Navigation map483.5.4The systems data flow493.5.5What should be considered for future enhancement depending on the first set of results:533.6Scenarios of use543.6.1Profiles543.6.2Synthetic sequences of events553.6.3The essay approach:553.7Synthesis564Chapter 4: Critical appraisal and conclusions584.1Concepts according to which See Through was designed584.2Launching the website594.3The online questionnaire: results and discussion604.4Feedback about the design and future enhancements664.5Synthesis665Appendices685.1Appendix 1: See Through screenshots685.2Appendix 2 : The PHP pages705.2.1Quest.php705.3Appendix 3: Website advertisement785.3.1Invitation to friends and relatives785.3.2Announcement on newsgroups795.4Appendix 4: See Through statistics collected805.5Appendix 5: Online questionnaire results825.6Appendix 6: The distribution of votes865.7Appendix 7: The initial project proposal885.8The work plan91References93Websites mentioned95List of tables and figuresFigure 1. The distribution of mean ratings on four websites (From Chen & Singh, 2001, p.158).24Figure 2. Distribution of mean rating for the questions61Figure 3. Distribution of mean rating for the sources of answer61Figure 4. The distribution of ratings for each question63Figure 5. The distribution of ratings for each source of answer64Figure 6: The page index.php68Figure 7: The page quest.php69Figure 8: The page sources.php69Figure 9: The administration area69Table 1: See Through statistics80Table 2: Online questionnaire results82Table 3: The distribution of ratings for each question86Table 4: The distribution of ratings for each source of answer87Figure 10. First Gantt Chart91Figure 11. Second Gantt Chart92Acknowledgements I would like to thank all the anonymous users who have gracefully accepted to answer the online evaluation of the prototype questions & answers database, Gregor Haddow for his priceless advice and technical help to build See Through (the prototype) and to discuss many usability aspects. Many thanks as well to Maxime Nokin and Fabrice Poulet for their technical support. I also wish to thank Ian Fraser, Tim Green and my father Jean-Michel Corre for their help in proof-reading my sometimes clumsy English. Thanks also to Colum Keating for the technical support and, last but not least, to my project supervisor Dr. Phil Turner for his support and useful suggestions. IntroductionThe advent of new technologies is a very trendy thing. They are sometimes more talked about than actually used. The simple reason to that is that the information age means information abundance or information overload. In this context, one could think that information filtering will become more and more important and crucial for future development of web design. Indeed, finding what one is looking for or reading new forms of information, not new in the contents but new in the shape, may significantly foster interest and gradually reshape the way we absorb information.Search engines are becoming more and more sophisticated and try to use at best criteria that were ignored before for sorting the results of a search. For example Google now asks willing users to install a Google toolbar on their browser to monitor their navigation in order to record links of interest between websites and to produce rankings for each site. The more links are found on the web leading to one website, the higher its ranking will be. The rankings, along with the word matching, are then used to sort the URL provided as results for a search.Another trend on the web in order to filter information is the use of human recommendation through simple ratings. Knowing what peers think of a 'product' before 'consuming' it, be it an article, a movie or a digital camera, can be very convincing. In the 'real world', it is very common to try to get some advice from friends, relatives or colleagues when making a choice. On the web, recommendations can be backed by a tremendous number of individuals if they are collected in an appropriate system. In this project we will examine the nature of ratings and attempt to state their strengths and weaknesses. In the first part, previous research in the area is summarised (chapter 1) and psychology concepts about decision making and group influence on judgement are investigated (chapter 2). The second part offers the description of the project's application, an online collaborative questions & answers database (chapter 3) and discuss the findings about users' behaviours toward ratings and information. The project's application is thus a website that uses a database and a scripting language to query and update the database dynamically.1 Chapter 1: Ratings online and collaborative filtering1.1 Introduction: Why use ratings?When making a choice in the absence of decisive first-hand knowledge, choosing as other like-minded, similarly-situated people have successfully chosen in the past is a good strategy. That is in effect using other people as filters and guides: filters to strain out potentially bad choices and guides to point out potentially good choices.Ratings and other sorts of evaluation and recommendation techniques on website can be seen as a communal history-of-user approach where everyone can contribute to a communal knowledge. Everyone can take advantage out of it by taking the bet that if a major part of other users rated an item (be it a product to buy, a movie to watch or an article to read) it is likely to please others.On the other hand, after having read an article, seen a movie or bought a product, one may be eager to “reward” its quality (and promote it) or to “punish” its mediocrity (and discard it from other users) with just a mouse click. 1.2 Examples:In order to discuss ratings it seems relevant to bear concrete cases in mind. The depth of the following descriptions depends on the interest of the website mentioned for the topic. 1.2.1 Le Monde (http:/www.lemonde.fr) a notorious French broadsheet. This example is mostly relevant because one of newspapers crucial qualities is the way information is presented. On a printed newspaper, the first page offers the headlines and articles that are “top of the iceberg”. The reader will browse information at his/her will either going straight to the page indicated on the front page, browsing every page, or checking a specific section.So far most news websites have reproduced this scheme. The home page corresponds more or less - to the printed front page, offering headlines, short articles and links to the full article. One can usually find a sections menu leading to topics and sub-home pages (politics, sport, international, etc.).At the end of every article on Le Mondes website, the reader is offered the opportunity to rate it simply by clicking a one, two or three star button. Once clicked, the buttons are replaced by an offer to consult the list of the eight most recommended articles.Whereas the first page will have the headlines of each section, supposed to compose together the most important news and most likely to interest readers, the list of the most recommended articles follow a less objective logic to offer a more advice-based approach. A comparison to real life situations could be cutting articles from a newspaper to send them to someone. This list of the most recommended articles can be seen as an alternative front page 1.2.2 D (Cnet)(), part of C - “The source for computing and technology”). The main point of this website is to offer freeware, shareware or trial versions of software all free to download. Software developers are invited to submit products they want to distribute. Each piece of software can be rated and reviewed with a simple form asking the user if s/he recommends this product - “thumbs up” or “thumbs down”, to rate its features, ease of use and stability on a five point scale (from bad to excellent), to give a one-line summary, detailed comments and sign with name or alias. Considering the risk presented when installing a piece of software (ranging from a mere loss of time if it is useless and disappointing to a crash of the machine or induction of a bug in the operating system) recommendations and testimony from other end-users can be highly appreciated. Comments and ratings will be reviewed by the site owners and, if approved according to guidelines, will appear where appropriate. There is no need to register to submit ratings and reviews.1.2.3 Action Forum (). This forum system was developed by the American advocacy community MoveOn.org that appeared during the Clinton impeachment hearings. MoveOn has the ambition to bring ordinary people back into politics: «When there is a disconnection between broad public opinion and legislative action, MoveOn builds electronic advocacy groups At MoveOn, every member has a voice in choosing our shared direction. Using our ActionForum software, you can propose issue priorities and strategies. Others will see and respond to your suggestions, and the most strongly supported ideas will rise to the top. We adopt the issues that rise to the top as our campaign priorities, on a cyclical basis. In the fall of 2000, for example, our members chose campaign finance reform and protection of the environment as our two top issues. Accordingly, these two issues are our major strategic priorities for the current congress. We'll also continue take the initiative to organize quick action on other timely issues such as the estate tax. » (MoveOn 1, 2002). These forums offer the user to rate each entry on a 1 to 5 star scale and to “agree” or “disagree” with it. A reader can thus