It May be Medicine, But Is It Science? - Why You Need to Know About John Ioannidis
In a number of pieces here on HealthCentral, I have raised the proposition that we have no valid scientific evidence base to support psychiatry’s claims for the efficacy of the meds they prescribe.
The argument is fairly simple: For long-term bipolar treatment, you can count the studies on one hand. These studies are so riddled with technical difficulties (such as nearly everyone dropping out) as to render them meaningless.
For mania, clinical trials typically go on for four weeks. Yet psychiatry uses the results to tell you that you will need to be on meds for the rest of your life, typically at knock-out doses.
For depression, clinical trials range from six to eight weeks. This seems to be the sweet spot for producing a measurable result - one that barely outperforms the placebo - before a host of things go wrong (such as the condition worsening or subjects dropping out).
As to that meds cocktail your doctor put you on, good luck finding any studies.
On and on it goes …
Now for the reassuring news: It’s not just psychiatry that is in the dark ages. If you are being treated for another illness, the scientific evidence is equally suspect.
The story begins in 2005 when a relatively obscure Greek medical researcher, John Ioannidis, published a piece in the journal PLoS Medicine. "Why Most Published Research Findings are False," read the provocative heading.
Today, Dr Ioannidis is world-renowned, a professor at Stanford, and a leader in the field of meta-research, basically the science of studying the studies.
In his PLoS paper, Dr Ioannidis calculated the type of things that can go wrong in research studies - researcher bias, small sample sizes, flawed study designs, attaching significance to insignificant results, and so on - devised a mathematical model, ran the numbers, and came to the astounding conclusion that “most claimed research findings are false.”
That same year, Dr Ioannidis also published a paper in JAMA. Here, he looked at 49 highly regarded medical research findings over the past 13 years. Forty-five of these studies claimed proof for effective interventions. Thirty-four of these studies had been retested by other researchers. Drumroll …
Four times out of ten, the subsequent study contradicted the original study results or found them to be greatly exaggerated.
It’s not that the original studies were wrong and the subsequent studies right, Dr Ioannidis points out. It’s that even in the most rigorous studies minor discrepancies have a way of creeping in and vastly distorting the result.
So, between two studies coming to contradictory conclusions, we have no objective way of showing which one is right and which one is wrong. This is Dr Ioannidis’ point. To show how absurd the situation can get, in a 2013 study, he randomly choose 50 ingredients from recipes in The Boston Cooking School Cook Book and went to work.
Forty of the ingredients, he found, had at least one cancer study associated with it. In an interview in MacLean’s magazine, he stated, “Most of the ingredients had results on both sides, positive and negative.”
“Coffee is good for you this week.” We’ve all rolled our eyes at news reporters citing these studies with straight faces. Next week, we know all too well, the situation will revert to coffee being bad for you. If you’re sensible, you pay no attention and pursue your guilty peasure in peace.
But what about for serious illnesses with our very lives riding on the outcome? For instance, in the past couple of years there has been a buzz about ketamine to treat bipolar depression. Wouldn’t we all love a magic bullet?
Don’t get excited, I said in an earlier blog post, or words to that effect. It was only a pilot study. Larger trials rarely pan out.
One of the enduring benefits of science is the scrutiny its practitioners routinely submit themselves to. If a finding cannot be convincingly replicated, it is discredited. Researchers go back to the drawing board and try a new approach. The field advances.
But as we have seen, even in the best of circumstances the process is deeply flawed. In our own lives, managing our own illness, we inevitably find ourselves guinea pigs in a chemistry experiment. It may be medicine, but can it truly be called science?
Further reading ...
For this piece, I am highly indebted to an article by David Freedman, Lies, Damned Lies, and Medical Science, appearing in the Nov 2010 Atlantic Magazine.