Survivorship Bias - The Pitfalls Of Learning From The Lucky Winners

A Tweet about survivor bias led me to an old post on "You Are Not So Smart."  It's just good reading with lots of information.

It's a long, somewhat rambling, post.  But it's well told and brings lots of ideas together.  It's about how survivor bias leads us, often, to faulty conclusions.  The reason?  Because we only hear from the survivors and not those who failed, which often can be the vast majority. Starts with a long story about:

"The official name for the people inside the apartment was the Statistical Research Group, a cabal of geniuses assembled at the request of the White House and made up of people who would go on to compete for and win Nobel Prizes. The SRG was an extension of Columbia University, and they dealt mainly with statistical analysis. The Philadelphia Computing Section, another group made up entirely of women mathematicians, worked six days a week at the University of Pennsylvania on ballistics tables. Other groups with different specialties were tied to Harvard, Princeton, Brown and others, 11 in all, each a leaf at the end of a new branch of the government created to help defeat the Axis – the Department of War Math."

One of their more tangible problems was how to better protect WWII fighter planes in the Pacific.  The problem was the military saw the bullet holes only on the planes that survived and tried to protect the planes better in those places.  Mathematicians pointed out that those planes had made it back, so those hits were survivable.  They needed reinforcement in the places the surviving planes hadn't been hit.


It then goes on to give lots and lots of other examples of survivor bias.

• in employer surveys (people who quit don't get surveyed)
• old art and literature is better than modern stuff (bad stuff in the past disappeared and only the 'not bad' survived, while the current junk is still all around)
• 'how to succeed books' from the successful (you miss the stories of the many ways to fail, and success seems much easier than it really is)
• lottery stories

There's also an interesting section on luck and lucky people.

"Wiseman speculated that what we call luck is actually a pattern of behaviors that coincide with a style of understanding and interacting with the events and people you encounter throughout life. Unlucky people are narrowly focused, he observed. They crave security and tend to be more anxious, and instead of wading into the sea of random chance open to what may come, they remain fixated on controlling the situation, on seeking a specific goal. As a result, they miss out on the thousands of opportunities that may float by. Lucky people tend to constantly change routines and seek out new experiences. Wiseman saw that the people who considered themselves lucky, and who then did actually demonstrate luck was on their side over the course of a decade, tended to place themselves into situations where anything could happen more often and thus exposed themselves to more random chance than did unlucky people. The lucky try more things, and fail more often, but when they fail they shrug it off and try something else. Occasionally, things work out."

I'm not totally sure about this.  I'd guess there are lots of 'unlucky' folks who take crazy risks that get them into serious trouble.  But maybe Wiseman can explain how that still fits his model.

There's history in this post, plus some good science lessons.  Basically this seems like a specific kind of generalization based on a biased sample.

There was something familiar about the site and so I did check and found I had written about the book You are Not So Smart a few years ago.