Reports on New Diabetes Drug and Treatment Studies: How To Detect Biases?
My local newspaper recently described a survey at a local high school asking graduates how well their high school math courses had prepared them for college work. Out of 75 questionnaires, 17 were returned. Of those, 7 said poorly, 7 said satisfactorily, 3 said well, and none said excellent.
(The choices aren't grammatically parallel, but what the heck; it wasn't an English survey.)
Anyway, the school principal reported that "10 out of 17 students said that the preparation was satisfactory or better." This is true.
The school board said that "14 out of 17 students felt less than well prepared." This is also true.
But each report gives a totally different picture of math preparation at the high school. They both ignore the fact that 77% of the questionnaires were not returned.
Obviously, readers of this blog don't give a hoot about the quality of math education at a tiny high school in Vermont. The point is that drug companies use the same type of statistical manipulation to report on the success of their drugs.
By merging results of studies into groups and then simply reporting on the groups, you can give a very misleading picture of what is really going on.
An example more relevant to diabetes is a study comparing colon cancer rates in people on a "prudent diet" with rates in people on a "Western diet." They lumped into the prudent-diet category all the things they thought were healthy and put everything else into the Western-diet category.
Hence the "prudent diet" included a lot of fruits and vegetables, including green leafy vegetables and cruciferous vegetables and legumes. Meats were fish and chicken. Drinks included tea and wine.
The "Western diet" included dairy, red meat, desserts, processed meats, refined carbohydrates, margarine, French fries, beer, eggs, snacks, and sweet drinks.
Not surprisingly the people on the "prudent diet" fared better. The conclusion blared by the newspapers? "Red meat causes colon cancer." No one tested whether red meat plus green leafy vegetables, cruciferous vegetables, and a few fruits would be healthy.
This means that when you read a news story about a new drug or a new approach to diabetes treatment, you need to look at the wording very carefully. Have the data been clumped into groups and the real numbers are not reported? Has the study excluded any of the results? If so, why? Do the variables in any one group include a lot of things but the press release is focusing only on the one that a particular group wants you to think is healthy (the study was probably funded by a company selling that food) or unhealthy (perhaps funded by a competitor)?
I'm not a statistician, and I can't analyze the statistics in the numerous studies of new diabetes treatments. There are many ways of presenting your data in the most favorable light, and I can't detect all the biases.
But we all need to be wary and scrutinize any results as carefully as we are able.