Estimating trading risks through online feedback

Research by Chrysanthos Dellarocas

Online feedback mechanisms have become an important tool for electronic businesses and for consumers who use them to evaluate potential trading partners and gauge the relative risk of dealing with people whom they may never meet in person. So the reliability of online feedback is crucial. But getting a true picture from online feedback is not always as easy.

Because online feedback is voluntary, it is prone to reporting bias. People seem to report positive experiences far more often than negative ones, perhaps partly because they fear retaliation if they post a negative review. This is a plausible scenario on eBay, where each trader’s reports are immediately visible to the trading partner, so feedback from the trader who posts first impacts the feedback posted by his or her trading partner.

But people have more than two feedback options. They can post a positive review. They can post a negative review. Or they can also post nothing at all. Chrysanthos Dellarocas, associate professor of information systems, felt there was something important to be learned from the silences of online feedback. With co-author Charles Wood, University of Notre Dame, he developed a methodology that allows users of bidirectional feedback mechanisms to see a more reliable picture of what is happening in private transactions by taking into account the silences in online feedback.

The study looks at 51,062 rare coin auctions that took place between April and September 2002, on eBay. The auctions included items from more than 6,000 distinct sellers and more than 16,000 distinct buyers. The dataset included auction information, seller information and winning bidder information, including both buyer and seller feedback posted within 90 days of the auction.

Dellarocas and his co-author considered all possible combinations of each trader’s feedback behavior—positive, neutral, negative, and silence—as well as all possible temporal ordering of comments—buyer first or seller first. They developed a quantitative method that derives estimates of the distribution of the private transaction outcomes that produced the target sample of online feedback on the basis of the relative incidence of feedback behaviors. This is the first paper to provide concrete numerical estimates of the degree to which feedback bias is present on eBay.

The most detailed version of the model estimates that on average, eBay buyers walk away from a transaction satisfied 78.9 percent of the time, neutral or mildly dissatisfied 20.4 percent of the time, and very dissatisfied 0.7 percent of the time. The authors claim that this is a more realistic estimate of trader satisfaction rates than can be discerned by just reading the online feedback, which is overwhelmingly positive—99 percent of the feedback on eBay is positive.

This is a problem that holds true for any site that makes use of publicly posted feedback as a reputation mechanism. In any Web-based feedback system, whether rating plumbers or lawyers or doctors, voluntary feedback will be subject to reporting bias.

“We the consumers have to be very careful when we interpret online feedback,” says Dellarocas. “We only see the feedback that people are willing to report, and often there is a correlation between the willingness to report (or not report) outcomes and the outcomes themselves.”

A model such as this permits people to see through the bias to get a more reliable picture of the risks associated with trading on eBay. While the average consumer won’t be able to run Dellarocas’ model, a third party could build a meta-engine to mine this information from eBay—or any site that permits public reviews—and give consumers a more “objective” report on the probable reliability of any given trader. Dellarocas can imagine a kind of “Consumer Reports” for reputation mechanisms that uses his model to give consumers more reliable information than the mechanisms themselves, increasing the efficiency of electronic and traditional markets.

These methods could be useful for many industries. Imagine an online feedback mechanism that would let patients post reviews of their satisfaction with physicians. Current attempts to create this kind of reputation mechanism are hindered by patients’ fears of being sued by their physician, a fear that could potentially keep many people from reporting a bad experience. This feedback bias would result in an unrealistically high rate of positive reviews, much like the situation that now exists on eBay. But insurance companies could conceivably use this model to extract information from a patient’s choice to remain silent and provide their customers with a more reliable estimate of the rate of satisfaction associated with specific physicians.

“The Sound of Silence in Online Feedback: Estimating Trading Risks in the Presence of Reporting Bias” is forthcoming from Management Science.