Recently, Last Week Tonight’s John Oliver ran a special segment on the nature of scientific research and how it is reported in the media. Oliver discusses the importance of not just reading the headlines of a science story, or even the article/press release, but actually looking at the nuances of a study before taking the data to heart.
So what can someone do to get a basic understanding of the study design? Here are a few items to get an average person started. Scientific studies have a lot of nuances to them, and if you have questions you can reach out to a CWI faculty member, or even the authors of the study you are interested in to discuss the study design and the implications.
- Has the study been published in a journal or is the news report talking about a conference abstract?
Frequently when big science conferences happen, press releases have been generated to highlight a talk or poster at a given conference. If this is the case, you should proceed with caution. This data is an informal presentation of scientific data. It has not been vetted by the rest of the community. In fact, when researchers present at conferences, this is usually what they are after, informal feedback about their research that can inform the rest of the research, or at a minimum, their publication.
- Is this a meta-analysis or systematic review?
In short (and really simplified), a meta-analysis is when a researcher combines the data from a variety of studies to gain a higher number of data points to minimize errors. These studies are the best places to start when investigating a topic. By including the words Meta-Analysis + your search topic, in Google Scholar or PubMed you should be able to find any relevant peer reviewed publications. You can find out more about meta-analyses here.
Frequently, these meta-analyses are a part of something called a systematic review. A systematic review is when a scientist reviews the research within a given field and aims to remove any bias or errors that may have occurred within individual studies. You can find out more about systematic reviews here.
In general, the researchers will either indicate that their study is one or both of these types of studies either in the title of the paper or within the abstract. Here is an example that shows the type of study. What is nice about these studies is that even if you don’t totally understand the methodology used, the conclusions are clear, and you can have more confidence in the data based on the number of participants alone
- Finally, if it is an individual study, what are the research parameters?
Individual studies can be nebulous for those outside of specific field. Even I have trouble interpreting data that is within my field but outside my area of expertise. So, what can someone with little background in the field look for? Start with sample size and what makes up the sample population. Small sample sizes mean there is not nearly enough information for us to make a generalized statement about anything. One of the most notorious examples of small sample size is Andrew Wakefield’s paper that spurred the misconception that vaccines cause autism. Beyond all the conflicts of interest and other factors associated with this study, there are two sample issues: only 12 children were used, which is not nearly enough to draw conclusive evidence about any hypothesis, and all of them were referred to the researchers and were not a random sample of the population. Compare that to this meta-analysis with over 1 million children and you can see why meta-analyses have so much evidentiary power.
After looking at population size, next look for what system the research took place in. Research that is just in the beginning phase is frequently conducted in systems known as in vitro (in glass) or in silico (computer modeling). Generally, this means that this research is being conducted on cells, tissue or in a computer model. These studies are the farthest from human application. Next are animal models, these are closer to human application because they involve a fully functioning body, but are still not completely translatable to humans due to differences in physiology and anatomy.
Hopefully this is enough to get you started on reading scientific journals or at least interpreting the headlines you see in the news and on social media, but if you ever have any questions, the CWI Faculty team is here to help!
Scientific literacy is something discussed in most of our classes! Check out our upcoming courses here!
