more is not more

The CIA on investing

So with all their resources, and with all the time they put into valuing these companies, why aren't the "professionals" producing more accurate results? A 1973 report written by analyst Richards J. Heuer at the CIA (yes, that CIA) suggests one answer. In this study, several of the people who set the odds on horse races were tested to determine whether having more information resulted in their making better predictions on race winners. Given 88 pieces of data to choose from, the "handicappers," as they're called, were told to choose the five bits of information they considered most important (e.g., the horse's win/loss record, the jockey's record, the length of the race, and so on).They were then asked to place bets on a race based on their preferred data and to state how confident they were of their predictions.

In part two of the test, researchers doubled the amount of data given to the handicappers. They got their "preferred five" pieces of data, plus five more statistics that they considered of lesser importance. Bets were again placed. Confidence was re-measured. This test was repeated with 20 and then with 40 statistics to work from.

The researchers then analyzed the results and concluded that the handicappers' accuracy did not improve as they were given more and more data. In fact, several handicappers got worse the more data they were fed. But while the accuracy of their predictions didn't increase with the amount of information they had to work with -- their confidence in those predictions did. This despite the fact that, by their own admission, the extra data was not as useful to them as the original "preferred five" pieces of information.