Does Diversity Trump Individual Expertise?
The wisdom of crowds, the difference, and banned books.
Troy Tassier is a professor of economics at Fordham University and the author of The Rich Flee and the Poor Take the Bus: How Our Unequal Society Fails Us during Outbreaks.
Too many cooks spoil the broth.
Don’t put all your eggs in one basket.
These two proverbs suggest opposite advice. The first proverb suggests that only one expert should be in charge of making the soup. Diversity is bad. However, eggs spread over several baskets protect against a dropped basket breaking all of your eggs. Diversity is good.
In an econ 101 course you see similar contradictions about diversity. When speaking about comparative advantage we advocate for specialization in production. However, when speaking about investments we advocate splitting your portfolio into a wide range of stocks and bonds so that when one drops in value, hopefully another increases so that you smooth out your rate of return. In one context diversity is detrimental and in another it is beneficial.
James Surowiecki opens his book The Wisdom of Crowds: Why the Many Are Smarter than the Few and How Collective Wisdom Shapes Business, Economies, Societies, and Nations with British scientist Francis Galton heading to a small country fair in 1906. When he arrives at the fair, he notes a contest that piques his interest. An ox is placed on display for the crowd. Folks at the fair are asked to guess the weight of the ox and those who come the closest to the correct weight win prizes.
Galton pondered how close the fair goers would come to the correct answer. Some in the crowd may have been livestock experts. Most weren’t. All of these folks guessed a weight, expert or not. After the prizes had been distributed Galton asked the contest organizers for the list of the guesses. He imagined the non-experts would pull the average guess far away from the true weight. To his surprise this didn’t happen. The average guess was amazingly accurate. The true weight of the ox was 1,198 pounds. The average guess of 800 people was 1,197 pounds! The average missed the mark by only 1 pound!
Obviously, that is an amazingly accurate prediction. However, there is an even more subtle point here. Imagine the most expert ox farmer in all the lands. Could the expert be as accurate as the crowd of fair goers? Probably not. It would take an awful lot of luck for even a high level ox expert to beat this average guess.
Why is the average so close? One reason comes from really bad random guesses above and below the true weight cancelling each other out. This contains only part of the story. If we set those aside, let’s consider a few ideas for how you could calculate a reasonable guess. You might have a friend stand near the ox. If the ox appeared to be about 5 times larger than your 200 pound friend, you might guess the weight of the ox to be around 1,000 pounds. Or maybe you measure the height and width of the ox and create a guess of 1,300 pounds based on these dimensions. Some might know the average weight of a dairy cow and make a guess based on their perceived difference in the size of the two animals. Their guess might be 1,100 pounds. Each of these people bring a small piece of information and a different perspective that is being incorporated into the collective guess. When all of this information is aggregated together, the fair goers produced a surprisingly accurate estimate. This is where Surowieki’s wisdom of crowds originates. Again, this random group might be able to beat an expert who holds far more knowledge and experience than any one person in the crowd.
You can also think of these different pieces of information as a set of perspectives. People look at problems in different ways based on their areas of expertise and life experiences. Someone who plays a lot of billiards may be very adept at thinking about the world in terms of angles. A taxi cab driver may have an uncanny sense of direction even in an unfamiliar area. A professional musician may perceive noises in a busy city in a different manner than a non-musician. We all bring different perspectives with us into new and varied circumstances.
A few years after Surowieki wrote The Wisdom of Crowds, Scott Page* wrote another book, The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools, and Societies. He leaned into this idea of different perspectives improving predictions and helping groups to better solve problems. The key was having a diversity of perspectives within the group.
Similar to The Wisdom of Crowds Page showed how groups of diverse people sometimes can make better predictions and solve harder problems than a group of experts. Consider the well-known story of a truck stuck under a bridge. A group of engineers think through how to safely deconstruct a portion of the bridge or truck in order to let the truck pass under the bridge. In the story the engineers are debating several possibilities when a small child walks up and suggests that they simply let some air out of the truck tires to lower the height of the truck below the bridge! The child has a new perspective that allows it to see the world differently in that particular moment than the engineers. This new and different perspective solves the problem.
Like Surowiecki, Page argues that selecting a random group of “pretty good” people often beats a set of experts. Page’s book is based on a series of mathematics and economics research papers, many of them written with Lu Hong. No math appears in the book but the papers by Hong and Page show mathematically when the wisdom of the crowd works and why. Amazingly, it’s a pretty general result. What do you need? Mostly, you need people to be reasonably intelligent and you need the problem that they are solving to be sufficiently difficult. If all you are doing is counting beans, an expert bean counter isn’t going to lose to a random group of people. However, if the problem is difficult enough that no one person can solve it perfectly by themselves, on average, a random group of pretty good people can beat a group of experts.
Why does this occur? The experts tend to think about the problem in the same way. Their expertise overlaps with the other experts so they tend to create similar solutions to problems. Having a bunch of experts, all with the same training, is little better than having just one of them. However, the random crowd has the kid who sees the air in the tire and not the structure of the bridge or the mechanics of the truck. It is the unique perspectives that give the random group an advantage over the collection of experts.
We can gain insight from these observations in creating excellent organizations. The Difference discusses how organizations can leverage diversity within workers, students, problem solvers, and other groups of people to make firms, schools, and other organizations stronger, more resilient, and better overall. Diversity avoids group think and allows for a wider range of possible solutions when solving problems. Diversity creates disruptors that change industries and societies by introducing new ideas. Page provides an argument for the benefits of diversity but at the same time shows its limits. Importantly, Page argues that DEI initiatives can provide more than fairness and opportunity for marginalized groups. To be clear Page and I both would argue that fairness and opportunity are sufficient goals for a DEI initiative. However, DEI can give us even more. It can give us, as the subtitle of Page’s book states, better groups, firms, schools, and societies! If implemented correctly DEI doesn’t give us something suboptimal. Instead, it improves performance!
If one thinks about the current environment where DEI initiatives are being struck down in schools, government organizations, and firms, we should be reading books like these so that we better understand what DEI does for us. Books like The Wisdom of Crowds and The Difference are essential in this historical moment. However, I recently learned that The Difference was pulled from the Naval Academy cadet library along with 380 other books because they didn’t align with President Trump’s DEI ban. In removing The Difference from the library shelves, they have removed the diverse perspective of an original thinker that could improve the way the Navy works and thinks. It was pulled simply because it contains an analysis of that dangerous word, diversity. That one word got it pulled. That is a loss for them and a loss for our society overall.
*Full Disclosure: Scott Page was my doctoral thesis advisor and has been a great friend for over two decades.