Wisdom of the crowd corrected for bias gets accurate

Participants in the study were asked to estimate how many gumballs or beads filled a given jar. (Gumballs pictured: 659. Beads pictured: 27,852) Image courtesy Albert Kao and Andrew Berdahl.

In 1907, a statistician named Francis Galton recorded the entries from a weight-judging competition as people guessed the weight of an ox. Galton analyzed hundreds of estimates and found that while individual guesses varied wildly, the median of the entries was surprisingly accurate and within one percent of the ox’s real weight. When Galton published his results, he ushered the theory of collective intelligence, or the “wisdom of crowds,” into the public conscience.

Collective wisdom has its limits, though. In a new study published in the Journal of the Royal Society Interface, incoming Omidyar Fellow Albert Kao (Harvard University), Omidyar Fellow Andrew Berdahl, and their colleagues examined just how accurate our collective intelligence is and how individual bias and information sharing skew aggregate estimates. Using their findings, they developed a mathematical correction that takes into account bias and social information to generate an improved crowd estimate. In the study, their corrected measures were more accurate than the mean, median, and other traditional statistics.

“There is growing evidence that the wisdom of crowds can be really powerful,” Kao says. “A lot of studies show that you can calculate the average of estimates and that average can be surprisingly good.”

“However,” adds Berdahl, “there is a great deal of evidence that people have strong biases in estimation and decision tasks.”

The researchers recruited over 800 volunteers to participate in the study and asked each participant to guess the number of gumballs in a jar, which ranged over several orders of magnitude from 54 to more than 27,000. Additionally, they quantified how individuals incorporate social information into their own opinion. To do so, the researchers offered participants fake details about other people’s guesses and allowed them to change their estimate in light of that information.

Kao’s team found that while estimates varied considerably, they were highly predictable. People tended to guess numbers smaller than the actual value and guessed a wider range of numbers for larger jars. Social information also plays a role in collective wisdom. For example, the simulated social information revealed that peer advice more strongly influenced an individual if the knowledge suggested the actual number of items was higher than the guesser’s initial estimate. Smaller guesses, even if more accurate, appear to be more frequently discounted.

The findings define a clear set of rules for collective estimation, which in turn provide insight into why we observe the wisdom of crowds. This understanding can improve group wisdom in a variety of settings. Whether guessing the weight of a fat ox or predicting the outcome of future elections, Kao and his colleagues have helped the crowd grow smarter.

Learn more: New study improves ‘crowd wisdom’ estimates