Shaping Consumers' Voice: Algorithmic Apparatus or Evaluation Culture?

In recent years, websites where ordinary users can rate and review products and services have become ubiquitous.  This crowdsourced and innovative form of evaluation broadly extends the area covered by existing professional valuation devices (technical benchmarks, travel and city guides, consumerist magazines): almost any good or service is today the subject of consumer rating.  

Related work and research goals
Existing research shows that online consumer reviews (OCR) affect the demand for products such as restaurants (Luca, 2011); and that they transform the economic activity of suppliers, who may use them as customer relation tools or as management tools (Scott and Orlikowski, 2012). Little empirical attention has been dedicated, though, to the reviewing activity of ordinary users, and to the nature of the valuation enacted by OCR devices - Pinch and Kessler (2011) focus on top Amazon contributors.
Understanding how “ordinary” customers evaluate products, the kind of criteria they rely on, is very important in order to characterize the shift produced by these new valuation devices, and their differences with expert and professional evaluation. What are the main consumer drivers for evaluating products? Does consumer evaluation favor certain goods or services? Ultimately, what kind of criteria and qualities are favored by customers?

Empirical material and case study
To address these questions, we focus on the restaurant industry, which offers a particularly interesting subject to complete this agenda. We combine three empirical materials dealing with online consumer and professional reviews in France:
1)    an online survey conducted on a representative sample of the French internet users, questioning their usage of online ratings;
2)    datasets from eight websites involving the evaluation of restaurants: customer reviews websites (Cityvox, Qype, TripAdvisor, LaFourchette, L’Internaute) as well as online versions of traditional gastronomic guides (Michelin, Bottin Gourmand, Gault & Millau). The online version of Michelin’s guide publishes both its own (professional) reviews and (ordinary) Internet users’ ratings and reviews.
3)    interviews with 33 intensive reviewers of restaurant reservation website LaFourchette (TheFork, bought by TripAdvisor in 2014). Interviews were combined with usage data provided by the website, and with extensive discussions with staff members.

Key findings

Online reviewing: two behavioral patterns
As expected, the intensity of participation is very heterogeneous among contributors: on most studied websites, more than half contributors left only one review, whereas 10% of contributors wrote half of the reviews. We observe that occasional and regular contributors follow distinct behavioral patterns. Occasional reviewers tend to give more extreme ratings (either bad or good); they also declare more to be motivated by expressing their feedback to the evaluated merchant/ producer. Meanwhile, active reviewers are more likely to have a balanced judgement: the global distribution of reviewing activity shows that they give mainly scores of 4 over 5. In expressing their motivations, they put forward a peer dynamic, oriented towards the “help of other consumers”; active contributors to LaFourchette underline that they participate, at their individual level, to a global contributive system from which everyone benefits.

Consumer reviews introduce new valuation criteria
The interviews with LaFourchette active contributors showed that users follow the criteria suggested by the website (food / setting / service), but most of them add a fourth quality, the warmth of the reception by the owner and staff, a quality that might specifically emerge from consumer evaluation. Moreover, most users do not consider the act of reviewing as an expression of gourmet taste (few of them describe meals in their reviews), but as an act of auditing the quality (freshness, etc.) of the restaurant, and the match between the experience and the promise made on the website. This coherent view of what a good review is, shared among almost all active users, can be described as an “evaluation culture” (McKenzie, 2011) built around the site.
The textual analysis of expert and lay reviews on the Michelin website confirms this trend, and demonstrates that the Michelin inspectors focus mainly on, say, what is in the plate, while online contributors underline the quality of service, and also take price as a modulator of their judgment. This importance of value-for-money leads consumers to give good ratings to restaurants which are considered by Michelin professional inspectors as average or low-end ones.

Conclusion
Our empirical material focused on restaurant lay evaluation allows us to provide a general understanding of OCR websites. Contrary to the idea that these websites operate the algorithmic synthesis of “unregulated opinions” (Orlikowski and Scott, 2014), we observe that active users develop an “evaluation culture” around the site, which frames valuation practices and procedures. This culture prevents extreme and too subjective judgements, and puts forward some specific qualities, distinct from those of professional judgement, such as the warmth of the reception. This culture is enacted by intensive contributors, not necessarily by occasional ones, who tend to articulate more extreme, and thus more heterogeneous, judgements. As they keep on surfing on and participating in OCR websites, contributors socialize themselves to its evaluation culture and learn how to produce a “good” amateur judgement.

References
Luca, M. (2011). “Reviews, Reputation and Revenue: The Case of Yelp.com”. Harvard Business School Working Paper, no. 12-016, September.
MacKenzie, D. (2011). “Evaluation Cultures? On Invoking ‘Culture’in the Analysis of Behaviour in Financial Markets”. Unpublished manuscript, University of Edinburgh.
Mellet K., Beauvisage T., Beuscart J.-S. and Trespeuch M. (2014). "A 'Democratization' of Markets? Online Consumer Reviews in the Restaurant Industry", Valuation Studies, 2014 2(1), 5-41.
Orlikowski W., Scott S., (2014), “What Happens When Evaluation Goes Online? Exploring Apparatuses of Valuation in the Travel Sector”, Organization Science, 25(3), 868-891.
Pinch, T., & Kesler, F. (2011). How aunt Ammy gets her free lunch: A study of the top-thousand customer reviewers at amazon. com. Managing Overflow in Affluent Societies.
Scott, S., Orlikowski W. (2012). “Reconfiguring Relations of Accountability: Materialization of Social Media in the Travel Sector.” Accounting, Organizations and Society 37 (1): 26–40.