Newsletter Sign-up

There is however a technique, well tested and documented, that provides a proven way to ferret the known out of the abyss of uncertainty. The surprising thing is that this technique has been around for 100 years, and yet very few risk managers seem to know about it.

This technique was first discovered in 1906 when the British intellectual, Francis Galton, attended a country fair in England. Galton was a scientist interested in the effects of breeding. It is important to note that he was also the sort of elitist who believed that crowds of common people were stupid, dangerous things.

Possibly to support his prejudice, he became interested in a contest in which fair goers paid a fee in order to guess the dressed weight of an ox. He decided to collect all of the guesses and analyze them. Given his dim view of the intelligence of the common man, he expected that the accuracy of their guesses would be awful.

What Galton discovered astounded him. The guesses of the 787 contestants averaged out to 1197 pounds. The actual dressed weight of the ox was 1198 pounds. The masses, when pooling their wisdom, were actually capable of guessing with great accuracy. This result flew in the face of all that he believed about the masses. Galton had no choice but to conclude, "The result seems more creditable to the trustworthiness of a democratic judgment than might have been expected."

What Galton had discovered with this little study was the Wisdom of the Crowds, or WOTC, effect. This outcome, which has been duplicated innumerable times in experiments as well as in real life, demonstrates that the average of many guesses tends to converge on reality. In other words, if one gathers a large number of guesses about some unknown, those guesses will come very close to the correct answer.

For example, if you ask 500 people to guess the number of jellybeans in a jar, the average of the guesses will typically be within about 1 percent to 3 percent of the actual number of jellybeans. The more guesses, and the more diverse the crowd, the more accurate the average estimate will be.

This effect is not only useful for county fairs or guessing jellybeans; it was proven to be effective in finding lost ships. In 1968, the U.S. submarine the Scorpion disappeared in the Atlantic. Very little was known about the location of the sub's final resting place. Because it was likely lost in a part of the sea that was several thousand feet deep, recovery seemed impossible.

However, a naval officer named John Craven knew about the WOTC effect and decided to apply it to this seemingly unsolvable problem. He gathered the guesses of many people who had experience with submarines and averaged their estimates. He took this result and began his search for the Scorpion near the average guess. Remarkably, this point turned out to be a mere 220 yards from the lost sub!

But how does this apply to guessing the future, one might ask. There are several examples of people using WOTC to derive very accurate predictions. My personal favorite, and the one that revolutionized the way I handle risk management, is that of the Space Shuttle Columbia. In 1993, Dr.Elisabeth Pate-Cornell of Stanford was asked to assess the risks of a space shuttle loss. It was a problem with no meaningful historical data and a mind-boggling list of potential risks.

Using an adaptation of the WOTC effect that is sometimes called expert elicitation, she determined that foam coming off of the front of the shuttle during re-entry into the earth's atmosphere could hit a wing and cause a catastrophic event. Recall that she made this prediction in 1993.

In 2003, foam came off of the space shuttle, hit the wing and caused the space shuttle to disintegrate 15 minutes prior to its scheduled landing time. It was a major catastrophe and one that drew a tremendous amount of scrutiny. An investigation into the crash was ordered by Congress. During this investigation, Dr. Pate-Cornell's prescient study was discovered. It was a complete bombshell. NASA had been told exactly what was going to happen and ignored the warning.The WOTC effect had been used in the form of expert elicitation to provide very accurate foresight.

Perhaps the most impressive evidence of the WOTC effect is the stock market. One of the reasons that the stock market is so incredibly difficult to beat is that it is so accurate. The stock market is essentially nothing but a real-time aggregation of the guesses of the traders. The markets seem so prescient that many have concluded that the efficient market hypothesis is indeed correct.

Take for example the Iowa Electronic Market, where one can buy and sell shares in political candidates. Many people try to predict presidential elections though polls (statistical sampling) or by consulting political experts, wonks and wags. These methods are consistently inaccurate.

However, the IEM has predicted presidential election results for the last 20 years with an accuracy rate of 1.37 percent. One could easily argue that the accuracy of the stock market is even greater than this. (Bubbles are an example of a different effect, which you can read about in Surowiecki's book--see below)

There are many more examples I could offer here in support of the validity of the WOTC. If you are interested in further research, "Wisdom of the Crowds" by James Surowiecki is an excellent read and details many of the nuances of the WOTC effect. It also explains how to avoid the things that cause the WOTC effect to go wrong (such as market bubbles).

You can also learn how to employ it on real-world risk management problems by attending Stanford University's Strategic Decision and Risk Management courses. (Kahneman and Tversky contributed extensively to the solutions being taught there.)

WOTC is a powerful tool and one that I now employ regularly within my risk management practice. It can be coupled with Bayesian analysis to create truly awesome accuracy in predicting future outcomes. There are, of course, pitfalls to employing it, most of which have to do with biases. But the behavioral psychologists have done a great job identifying these, and they can be overcome with a little study. (Again, Stanford has a great series of Internet courses on the subject.)

This is the type of solution that can crack many of the seemingly impossible challenges faced by risk managers. When one is faced with a new risk that has little or no historical data, one can gather guesses from a pool of experts and/or nonexperts, average the guesses and have good quantitative predictions.

This can work for outcomes such as potential loss amounts, profits and probabilities around each. It gives us a crystal ball that few others have. In areas where the actuarial sciences have dominated, such as insurance, creating a probability distribution without data is often considered to be impossible (even though actuaries have the Delphi method, which will be discussed in a later issue.) As a result, many risks are simply ignored. Unfortunately, ignoring risks does not make them go away.

Every single risk manager on the planet should know how to use the WOTC effect to create probability distributions. It is really not hyperbole to make such a statement; this is a powerful tool. It is also precisely the type of solution that will be showing up on www.riskreports.com very soon (with a little help from the RMR community).

BEAUMONT VANCE manages risk for Sun Microsystems Inc. This column was a complimentary excerpt from one of his latest "Risk Management Reports" newsletters, which he edits and publishes. For more information on how to subscribe to the full version of the newsletter, please visit www.riskreports.com/.

Click here to read Beaumont's latest Risk & Insurance® columns.

READ MORE: Features | Special Reports | Industry Risk Reports | Columnists | In-Depth Series

September 1, 2007

Copyright 2007© LRP Publications