A Rough Guide to Types of Scientific Evidence

Click to enlarge

Today’s graphic looks at science in general, rather than just chemistry. It’s in a similar vein to the Rough Guide to Spotting Bad Science posted last year, but this time looking at the hierarchy of different types of scientific evidence. You might think science is science, but some evidence is ranked higher in the scientific community than others, and having an awareness of this can help you sort the science from the pseudoscience when it comes to various internet claims.

This graphic was inspired by a couple of things this week. The first was the recent Gawker article on the Food Babe’s spurious chemical claims, which you may well have seen spreading like wildfire through social media earlier this week. The second was a Twitter exchange regarding the artificial sweetener aspartame, during which the user whom I was conversing with repeatedly linked me to poor quality ‘evidence’ in order to try and prove their point.

There’ll be more on aspartame in a future post, but what I took away from this was that some people out there aren’t sure what counts as robust scientific evidence. To an extent, you could argue that this is a fault in education. We spend a lot of time discussing the scientific method with classes – the concept of a fair test, and controlling variables, repeating experiments, that kind of thing – but we rarely talk about sources of scientific evidence, or the scientific process in fields like medicine or toxicology.

The reason for this is perhaps that, in parts, they’re more complicated concepts, but they’re ones that are of great benefit to understand, even for those who aren’t destined for a scientific career. The idea that sources of internet misinformation like the Food Babe might cease to exist with a better public understanding of scientific evidence is a bit of an idealistic one, but perhaps it might give those following cause to stop and question evidence provided, rather than merely accepting it at face value.

The first point to make about scientific evidence is that the types of evidence featured in the graphic aren’t always distinct, nor does the fact that a type of evidence is lower in the hierarchy mean that it should instantly be disregarded. In fact, some of them can be precursors to the more conclusive types of evidence. For example, medical trials of potential drugs will usually commence with animal or cell trials, before then progressing to randomised controlled trials. When enough randomised controlled trials have been carried out, a systematic review of these trials becomes possible. Thus, the amassing of scientific evidence is an ongoing process, often involving several of the types of evidence shown here.

Before discussing the two primary types of evidence, it’s worth discussing the outlier: Anecdotal evidence, or an expert’s opinion. An example of anecdotal evidence would be someone relating a tale of how they experienced a reaction after ingesting a particular type of food or medication. Whilst anecdotal evidence can act as a precursor to scientific investigation, in isolation it is often considered dubious.

Perhaps surprisingly, an expert opinion on a particular topic is considered to be at the same level as anecdotal evidence. Of course, if references to other, more rigorous scientific studies are provided as part of the opinion, it can help, but it’s still best to go to the source of the evidence in these cases. These opinions aren’t vetted, so whether published in a letters page, or merely online, they must be judged with this in mind. I could write that a particular chemical will turn you into a newt, but unless I can provide a link to scientifically rigorous evidence that backs up my opinion, it counts for next to nothing.

Generally, scientific studies often fall into two main categories: observational and experimental. Observational studies merely involve scientists observing an event that’s already occurring – for example, they may observe a group of subjects, some of whom smoke, and some of whom do not, to see which subjects develop lung cancer. However, they won’t intervene or have any effect themselves on these studies – they will merely gather results as the study progresses.

There are three main subcategories under the observational studies header. Case reports, used particularly in a health context, merely observe particular individuals and comment on their condition. Case series, often bundled in with case reports, only differ in that they include multiple subjects. These studies cannot prove that a certain treatment or exposure causes a certain effect, but they can reveal areas for potential further investigation.

A case control study, on the other hand, actually works backwards. To give an example from a medical context, it takes a group of patients, some with a particular symptom or disease, and some without. Researchers then work backwards and try to work out a common exposure or attribute that could have caused the symptom or disease. A cohort study, meanwhile, is essentially the lung cancer example previously provided, where two groups of subjects are observed over a period of time.

Types of observational studies not included in the graphic include cross-sectional studies, which involve analysis of data collected from a sample population at a specific point in time, and longitudinal studies, which involve repeated observation of the same variable over an extended period of time. Longitudinal studies are more commonly used in psychology and sociology than in the traditional sciences.

Experimental studies generally appear higher up the hierarchy than observational studies, with one exception: Animal or cell studies. These rank lower, at least when considering human effects, because they are both only models of what goes on in our body. In terms of drug trials, animals may respond to a drug differently than humans, and even in studies involving human cells in isolation, they may not act exactly as they do in the body. As such, these trials are usually precursors to studies in humans, where this is deemed appropriate. Of course, animal trials of cosmetics are now prohibited in many countries, so this evidence is not available in the case of new ingredients in personal care products.

Experimental studies involve scientists actually placing subjects into groups and carrying out experiments. In a typical randomised controlled trial, subjects are randomly placed in a group; in a simple medical study, one group will receive the treatment, whilst the other will receive a placebo that appears identical. In the best trials, neither the subjects nor the researchers know which group they have been assigned to, as this helps minimise any potential bias. A control group is important, but is sometimes not possible. For example, in trials of vaccines, it would not be ethical to withhold vaccines from those suffering from potentially deadly diseases, and instead give them a placebo.

At the peak of the scientific evidence hierarchy is the systematic review. These collate all high quality research on a topic, particularly that provided by randomised controlled trials where available. They can help mitigate bias, intended or otherwise, in independent studies, and give a more balanced picture.

Another type of experimental study not included in the graphic is the field study. These take place, not in a laboratory, but in the natural setting of the subject. In some areas of research, for example research into pesticide use, they are sometimes seen as having higher validity than controlled trials, though their conclusions can also be uncertain due to the lack of control over some other factors.

In conclusion, hopefully this graphic helps make clear the different forms of scientific evidence out there, and what each involves. Being aware of each of them is a useful tool in interpreting scientific evidence, and making sense of claims in the media and elsewhere. If you haven’t already, you should also check out the Rough Guide to Spotting Bad Science, which is on a similar theme, and which has also been spruced up for the occasion to match the style of this graphic.

Meanwhile, if you’ve any suggestions for improvements to the graphic, I’d be happy to entertain them – feel free to drop them in the comments below! Thanks to the those who provided feedback on this graphic prior to posting: @simbelsim, @MrsDrSarah, @aHardNightsDay, @futscinow, @Durant_Scholz, and @jolayfield.

Support Compound Interest on Patreon for post previews and more!

DOWNLOAD

SUBSCRIBE

The graphic in this article is licensed under a  Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License. Want to share it elsewhere? See the site’s content usage guidelines.