My new favorite kind of books are ones that use statistical data and research studies to draw subversive conclusions about society and promote new ways of thinking about everyday things.
“Swimming pools are more dangerous than firearms”, “success as a Canadian hockey player is determined by your month of birth”, and “an increase in abortion causes a drop in crime”. These are just a few arguments made by the likes of Malcolm Gladwell and Steven D. Levitt.
Although I do enjoy these books, after reading many of them a strong pattern has emerged (outlined in the 3 steps below) and now they are starting to seem as formulaic as a Dan Brown novel.
1) Find a correlation: Use a research studies to find a statistical correlation (either positive or negative) between variable A and variable B.
2) Correlation analysis: Citing supporting evidence, make an assertion as to the nature of the correlation – either A causes B, B causes A, or a different variable causes both A and B.
3) Conclusion: Use the correlation to draw a broad conclusion.
The reason that arguments made in this fashion are so compelling is because they start with seemingly solid research and statistical facts. However, as these arguments travel through the process (from steps 1 to 3) the arguments usually tie in assumptions and the supporting evidence gets progressively weaker. Below are three common assumptions that tend to tacitly get mixed in.
A) The statistics cited are accurate and are still relevant today.
B) All evidence has been examined (it has not been cherry picked for supporting bits) and there is sufficient evidence (beyond an anecdote) to assert the nature of the correlation.
C) The populations (sample sizes) used in the base studies are large enough and sufficiently similar to the population referenced in the conclusion to allow statistical significance.
Keeping these assumptions in mind you may find that some seemingly solid, math-based arguments get reduced to an insignificant jumble of circumstantial and anecdotal evidence. It’s not to say that the arguments are any less entertaining or thought provoking – just less true. Overall, it’s important to remember this powerful and time tested rule of thumb: if it seems wrong – it probably is.