The problem with observational studies

May 9, 2012, 12:47 pm

An interesting recent Wall Street Journal article, “Analytical Trend Troubles Scientists“,  outlined some of the problems with observational studies. The author, Gautam Naik, notes that in observational studies “scientists often use fast computers, statistical software and large medical data sets to analyze information collected previously by others”. He points out that:

While the gold standard of medical research is the randomly controlled experimental study, scientists have recently rushed to pursue observation studies, which are much easier, cheaper and quicker to do. Costs for a typical controlled trial can stretch high into the millions; observational studies can be performed for tens of thousands of dollars.

These studies often involve methodological flaws and biases that can be difficult for readers or reviewers to detect, frequently detect associations that can be mistaken for cause-and-effect relationships, and are rarely corrected.As S. Stanley Young from the U.S. National Institute of Statistical Sciences points out in the article: “You can troll the data, slicing and dicing it any way you want . . . a great deal of irresponsible reporting of results is going on. Scientists and statisticians  Naik interviewed estimated that only 5-20% of observational studies can be replicated. This means not only that a great deal of misleading scientific nonsense is being published, but also that an enormous amount of work is being wasted trying to confirm shoddy work.

This article is a must-read. Unfortunately, the complete text is behind the Journal’s firewall, available only to subscribers.

Thanks to for bringing this article to my attention.

Comments are closed.