Since my son is taking longer and longer naps, I’ve had more time on the weekends to do things…like read the issues of Anesthesia and Analgesia that have been piling up over the last few months.
I came across a very interesting observational study by Robert S Greenberg, Melania Bembea and Eugeni Heitmiller entitled, Rainy Days for the Society for Pediatric Anesthesia.
They looked at all the national meetings held by the Society of Pediatric Anesthesia from 1987 to 2010, inclusive. In order to determine if it was more likely to be rainy on the first day of a national meeting, they looked at historical weather records:
“For example, if a meeting was held on October 9, 1987 in Atlanta, then data from October 9 for all years (1987-2010, inclusive) were evaluated for rainfall and included in the general calculations.”
They considered it to be “rainy” if < 0.01 inches of rain fell that day. What they found was startling. They didn’t just find an association between rain and the first day of a meeting, they found what looks like a really freakin’ strong association:
“Analysis…suggested a significant association between having the meeting and rain, with estimated odds ratio of rain comparing meeting and non-meeting days of 2.63 (P value 0.006, 95% confidence interval 1.32-5.22).”
They go on to mention that,
“SPA leadership has recognized this phenomenon and now offers a stylish umbrella with the SPA logo for purchase at the meetings. We believe further investigation is warranted…Until then, SPA leadership may use our rainmaking ability to benefit drought-stricken areas when choosing meeting venues.”
So, are we all chuckling?
Is this the April Fool’s issue of the journal?
I want everyone to stop chuckling. This report has some serious lessons to teach us about scientific research and observational studies in particular. In fact, A&A published this report specifically to remind its readers about the pitfalls of observational studies.
Firstly, most of us have an intuitive idea (or cognitive bias) that there is no association between weather and events. Although we may moan to the contrary, we don’t actually believe that it rains just because we are on vacation. Or that it rained on our wedding day for any reason other than chance and the fact that we booked our wedding during the “off-season” when the weather isn’t as nice in order to save a little money. Because we don’t really think there is a connection between rain and organized events like scientific meetings, our family vacation, or our wedding, we are quick to dismiss this report as either a freak of statistics or quackery or randomness.
Intuitive ideas, however, have also led some of us to believe that epidurals lead to caesarean sections, or slower labour, or cause problems with breastfeeding just because all of these associations have been found in retrospective observational studies.
I encourage everyone to read this paper and the accompanying editorial. The human brain excels at finding patterns even where there are none (does that cloud look like a unicorn? You bet it does!!!). Statistical methods are simply a mathematical way that we have developed to find “patterns” in “noise.” When we search for patterns in retrospective data, our brains, our wonderful pattern-finding brains, can lead us astray. Observational studies can be suggestive and interesting but even if the numbers are very large, and the statistical methods are very robust, they still suffer from known and unknown confounders and biases. They are often considered hypothesis-generating for that reason. Yet people without formal education in epidemiology or the interpretation of medical literature often cite observational studies as sources that prove causality.
That is simply wrong.
Remember that it is wrong. Remember the “rain-generating” power of the SPA.