Why observational studies need to be interpreted with extreme caution

Since my son is taking longer and longer naps, I’ve had more time on the weekends to do things…like read the issues of Anesthesia and Analgesia that have been piling up over the last few months.

I came across a very interesting observational study by Robert S Greenberg, Melania Bembea and Eugeni Heitmiller entitled, Rainy Days for the Society for Pediatric Anesthesia.

They looked at all the national meetings held by the Society of Pediatric Anesthesia from 1987 to 2010, inclusive.  In order to determine if it was more likely to be rainy on the first day of a national meeting, they looked at historical weather records:

“For example, if a meeting was held on October 9, 1987 in Atlanta, then data from October 9 for all years (1987-2010, inclusive) were evaluated for rainfall and included in the general calculations.”

They considered it to be “rainy” if < 0.01 inches of rain fell that day.  What they found was startling.  They didn’t just find an association between rain and the first day of a meeting, they found what looks like a really freakin’ strong association:

“Analysis…suggested a significant association between having the meeting and rain, with estimated odds ratio of rain comparing meeting and non-meeting days of 2.63 (P value 0.006, 95% confidence interval 1.32-5.22).”

They go on to mention that,

“SPA leadership has recognized this phenomenon and now offers a stylish umbrella with the SPA logo for purchase at the meetings.  We believe further investigation is warranted…Until then, SPA leadership may use our rainmaking ability to benefit drought-stricken areas when choosing meeting venues.”

So, are we all chuckling?

Is this the April Fool’s issue of the journal?

I want everyone to stop chuckling.  This report has some serious lessons to teach us about scientific research and observational studies in particular.  In fact, A&A published this report specifically to remind its readers about the pitfalls of observational studies.

Firstly, most of us have an intuitive idea (or cognitive bias) that there is no association between weather and events.  Although we may moan to the contrary, we don’t actually believe that it rains just because we are on vacation.  Or that it rained on our wedding day for any reason other than chance and the fact that we booked our wedding during the “off-season” when the weather isn’t as nice in order to save a little money.  Because we don’t really think there is a connection between rain and organized events like scientific meetings, our family vacation, or our wedding, we are quick to dismiss this report as either a freak of statistics or quackery or randomness.

Intuitive ideas, however, have also led some of us to believe that epidurals lead to caesarean sections, or slower labour, or cause problems with breastfeeding just because all of these associations have been found in retrospective observational studies.

I encourage everyone to read this paper and the accompanying editorial.  The human brain excels at finding patterns even where there are none (does that cloud look like a unicorn?  You bet it does!!!).  Statistical methods are simply a mathematical way that we have developed to find “patterns” in “noise.”  When we search for patterns in retrospective data, our brains, our wonderful pattern-finding brains, can lead us astray.  Observational studies can be suggestive and interesting but even if the numbers are very large, and the statistical methods are very robust, they still suffer from known and unknown confounders and biases.  They are often considered hypothesis-generating for that reason.  Yet people without formal education in epidemiology or the interpretation of medical literature often cite observational studies as sources that prove causality.

That is simply wrong.

Remember that it is wrong.  Remember the “rain-generating” power of the SPA.

Advertisements

6 comments

  1. Fabulously explained!

    1. Hey Squintmom! How is your shoulder?

      The Adequate Baby has hand foot and mouth…any advice?

      1. Shoulder’s ok…achy, but ok. The anesthesiologist was FABULOUS! I no longer have any fear of general.

        So sorry to hear about HFMD! W found nursing soothing, and cold drinks are also great. Basically imagine having a terrible case of strep throat, and what would feel good or soothing. Popsicles, milkshakes, etc. Nothing acidic. Also, don’t be afraid to ask your pedi for some magic mouthwash (lidocaine, maalox, and benadryl). It really helps. W was too young to use it as a wash, but we dripped small amounts in her mouth with a little sponge. Also, benadryl by itself is a bit numbing and also helps with sleep. Hang in there; iy’s a nasty virus and lasts 7-10 days.

  2. Laura · · Reply

    What a great post! I am pursuing a degree in nursing and then midwifery and read many, many medical journals. I really want the truth about issues relating to my field of medicine. What should I trust as reliable sources of information? (Good luck with your sick child – always a drag!)

    1. I think one of the things that is really difficult about trying to define an evidence-based practice is that, in a lot of cases, the evidence isn’t up to a great standard. A single study can rarely be considered definitive. Practitioners need to be familiar with the body of literature that pertains to them so that they can see how newly published pieces fit into the puzzle. It’s hardly a case of what’s “true” and more a case of what is “probably more true than not.”

      The best advice I can give you is to keep reading, but also to find a place to have meaningful discussions about studies published in your field (like a journal club) and to go to conferences to hear experts and participate in panel discussions…just keep talking and learning and questioning.

      It’s a process that never ends. If it wasn’t for all the advances in patient care and outcomes, I’d get nostalgic for the good old days when no one knew anything, there was no evidence and you just did what was “traditional.”

  3. Behavioral sciences are even more plagued with this problem. The field is overwhelmed with atheoretical research, poor methodology, and unfounded conclusions. While this study might not have much in the way of real consequences, other than fancy umbrellas, it is so scary to think that other similar studies lead to practice and policy change!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: