Last week we discussed the difficulty in interpreting data in research trials in general and applied the Washington Post article as a springboard to discuss the importance of soundly conducted evidenced based research. Now I would like to provide my approach to evaluate scientific literature.
My suggestions are as follows.
- Is the paper peer-reviewed? Peer review means that colleagues who have experience and expertise in the field evaluate the article. They will be very critical; looking very carefully at the data and analysis to determine if it makes sense. Most papers are reviewed by at least three reviewers to avoid bias. If the paper is not peer reviewed, less attention will be directed to the results. Many papers are not accepted to top journals for various reasons. However, that does not mean the research is defective; it may mean that the journal is not interested at the time in that particular topic, or that is not the direction of the particular journal. Therefore, many authors will submit to other journals after the first rejection. Keep in mind that a rejection from a peer-reviewed journal will often make the paper stronger in the end as most authors take the advice and criticism offered to make papers more robust. The authors will then read and move forward with the critique-often employing different or additional statistical methods to improve data analysis and therefore have more valid conclusions.
- Are there enough research subjects to draw a statistical conclusion? Four human patients who produce insulin after pancreatic islet cell transfer is very exciting; 4,000 patients who produce insulin is groundbreaking. Worthy of a Nobel Prize in Medicine.
- Does the Institutional Review Board (IRB) approve the study? Have the research subjects been given informed consent and treated ethically based on Worldwide Research Standards of Practice? Are the identities of the research subjects protected?
- Is the data analyzed with the appropriate statistical instrument? This is not always apparent, which is why it is important to have peers evaluate for the statistical tool’s validity or make different recommendations for data analysis.
- Is the study prospective or retrospective with blinding of the participants? A prospective “double-blind” study is much more valued in the scientific realm and provides the necessary results for generalization of information. A retrospective study also is very useful, but will provide different information. Many retrospective studies use chart review.
- Does the study prove causality or correlation? Just because something is associated with another does NOT mean that is caused by it. For example, people who live in areas with decreased sunlight (ie: Scandinavia) appear to have an increased incidence of type 1 diabetes. Does that mean decreased sunlight causes Type 1 diabetes? No. However, decreased sunlight does lead to decreased Vitamin D production. Does this mean low Vitamin D levels lead to Type 1 diabetes? Not true; however, low Vitamin D levels may be correlated or associated with Type 1 diabetes.
- Is the paper published in a reputable journal or website? Just because information is in print does not mean that it is true! It is very important to be critical of what you read. The information may be misleading or published to provoke you into buying a product that is unproven to be effective.
- Who is doing the research? Are they associated with a university, medical school, and/or pharmaceutical company? What is their track record?
- Do all the authors report affiliations with commercial entities and report how they are related (as consultants, researchers, speakers’ bureau, or stockholders)?
- Who is paying for the research study? Is the study funded by the NIH, FDA, university funds, philanthropic grants (Juvenile Research Foundation International, American Diabetes Association etc.)? When quoting a research study, I always note funding by a pharmaceutical company. That is not to say that the results are not valid; rather that the reviewer must be aware of the possible biases that may come into play.