A brief detour from chemistry, branching out into science in general today. This graphic looks at the different factors that can contribute towards ‘bad’ science – it was inspired by the research I carried out for the recent aluminium chlorohydrate graphic, where many articles linked the compound to causing breast cancer, referencing scientific research which drew questionable conclusions from their results.
The vast majority of people will get their science news from online news site articles, and rarely delve into the research that the article is based on. Personally, I think it’s therefore important that people are capable of spotting bad scientific methods, or realising when articles are being economical with the conclusions drawn from research, and that’s what this graphic aims to do. Note that this is not a comprehensive overview, nor is it implied that the presence of one of the points noted automatically means that the research should be disregarded. This is merely intended to provide a rough guide to things to be alert to when either reading science articles or evaluating research.
EDIT: Updated to version 2!
EDIT 2 (April 2015): Update to version 3, taking into account a range of feedback and also sprucing up the design a little.
Support Compound Interest on Patreon for post previews and more!
The graphic in this article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License. Want to share it elsewhere? See the site’s content usage guidelines.
You can download the previous version as a PDF here.
Purchase in A2, A1 & A0 poster sizes here.
The second version of the graphic is also available in: Portuguese (courtesy of Marco Filipe); Russian (courtesy of orgchem.by); and Spanish, (courtesy of Carolina Jiménez at nuakas.com).
222 Comments
Alex
I think #5 should be replaced with nearly the opposite. Scientists recognize that any conclusions they make are tentative, and treat them as such. Theories are the best possible explanation based on available evidence and if they are impervious to being overturned by contrary evidence, they are not scientific. Certainty only exists in math and philosophy.
Science finds evidence that supports. Pseudoscience finds “facts” that “prove.”
The primary scientific literature and good pop science writing should not use words like “certain,” “absolute,” “definite,” “right,” “correct,” “always,” and their opposites when referring to observational or experimental science.
Compound Interest
A fair comment – I guess I was trying to get more at the over-use of these. I think I’ll leave it as is for now (I literally just spent a few hours tweaking it), because I think it’s also important that people realise exactly what you say – that research is just theory, even when it’s not ‘bad’ science, and that they should be open to questioning it, rather than just taking it at face value.
Jane
I agree, Alex, and it’s something I push frequently when people claim science has having “the truth”. Science determines the most likely probability, no more than that.
Byron Sharp
Great. But #11 is a bit weak “potentially replicable” is one thing, having been replicated by independent research is another. Then ideally it should be tested over a wide range of conditions to check for generalisability and see if the result is highly specific/fragile.
Compound Interest
Thanks for the pointers – I’ll rephrase that one in the next version to improve it.
Mike
Along the same lines, I was definitely looking for some mention of peer review here.
Mike
I guess that more or less goes in #12… maybe I’m putting too much emphasis on the phrase “peer review”, since the process of peer review has been more or less described in #12.
Compound Interest
Tweaked the description for #11 in this current version now anyway – hopefully a little clearer!
David
Point 6 – the issue of small sample sizes is relevant to all trials, not just humans. This is going up on the undergraduate common room wall..
Compound Interest
Good point – I’m planning to update it a little following people’s feedback this evening, so I’d hold fire on printing it until then!
jin
You might want to raise some stipulations in regards to sample size. Plenty of studies have a few subjects, but have high levels of control by observing them multiple times or creating highly controlled experimental environments. For example, how many different types of balls do you have to roll down a plane before you are convinced you have a relationship between angle and speed? Many behavioral studies in both humans and animals also utilize single-subject design (the name just represent the experimental design, more than 1 subject is often used). Using within-subject designs often allows for greater levels of control, with each subject serving as their own control.
Compound Interest
I think I’m going to add some qualifiers underneath the graphic in the post. This was only intended to be a ‘rough’ guide, which got a lot more attention than expected 🙂 Your points are worth including, and since there are probably several other points where exceptions can be made, it makes sense to include more detail outside the graphic rather than trying (and probably failing) to squeeze it all in!
Mike Casey
Good work, the graphic includes many useful tips for sorting the wheat from the chaff.
One other guideline that might be worth a mention is that ‘extraordinary claims require extraordinary evidence’. Claims for breakthroughs, especially ones that contradict well-established ideas, should be treated with great caution unless the supporting evidence is extremely strong. Though there are notable exceptions, most such claims turn out to be unfounded, e.g. ‘cold fusion’, or the recent claim that cells can be converted into stem cells by treatment with acid.
Compound Interest
Cheers! I’ll try and squeeze that one in somewhere – not sure there’s space for an extra point, but maybe I can fit it into one of the subsections – which do you think would fit it best?
Mike Casey
One possibility would be to merge it into Point 11, by making the point that extraordinary claims should be regarded with particular skepticism until tested and reproduced by others.
Another option might be to merge Points 8 and 9, both of which are concerned with the need for good control experiments, and add the tip about the need for extra caution when evaluating extraordinary claims as a separate point.
These are just some very quick thoughts, the dilemma may get worse if other good guidelines are suggested. But do please persevere, this is really useful work.
Compound Interest
Thanks, I think merging it with 11 seems like it’ll be the easiest to fit in concisely. Working on version 2 at the moment!
G.Harrington
As a general science geek (albeit, an armchair one), I find myself often trying to explain precisely these key points in arguments on the value of numerous claims by sensationalists and info-spammers. This relates the important points succinctly; I don’t have to be a professional researcher to find value in it. Thank you very much for taking the time – it is appreciated. Expect many, many popular networking shares!
Compound Interest
Thanks! Whilst the idea was to try and get it out there and persuade more people to consider these points when appraising science in the media, it’s certainly taken off a little more than I expected it to 🙂
Rodrigo de la Jara
#6: The language implies that larger sample sizes will reduce bias. This is a common misconception. If a sampling method is biased, larger sample sizes will not fix the bias unless the sample size gets so large as to approach the population size.
Compound Interest
Thanks for the pointer – that wasn’t the intention when writing it, but clearly in trying to cram it into the shortest sentence I could fit, it could come across that way. Since non-representative sampling is mentioned separately, I didn’t mention it as part of that point. Any particular suggestions you’d make for how to reword it so it’s a little clearer?
Rodrigo de la Jara
To keep things simple, I’d just remove “when applied to the whole population”.
Ramon Casha
In many cases these could be bad reporting of good sciene.
I’ve seen countless cases where properly done, carefully worded papers are spun into something much more exciting but factually wrong.
Then of course the internet is the ultimate “telephone game”, where the original paper is turned into an article for the science fan by a magazine, which gets picked up by a newspaper and reworded for the layman, which in turn gets picked up by my-favourite-conspiracy-theory.com and turned into something bizarre which somehow drags in chemtrails, the Illuminati and Area 51.
Oliver H
That falls a bit for #1 itself.
Yes, blinding makes for better science, but no blinding alone doesn’t make for bad science. There are plenty of constellations in which blinding simply isn’t possible – whether it is that I compare an intravenous drug to an orally taken one, or I compare the use of two different surgical instruments or laboratory analyzers, there are plenty of constellations in which blinding is simply not possible. Sometimes, blinding of the patient and the doctor is impossible, sometimes blinding of the treating physician isn’t possible. That’s simply a fact of life.
A smaller issue is the sample size. Stating that smaller sample size lowers the confidence in the results is waffling a bit. It increases variance and partially as a consequence increases the probability to commit type 2 errors. If I nonetheless see a difference it might very much speak for one existing, if I can see it despite the amount of noise. In any case, small sample sizes should only then be reason for suspicion when larger sample sizes would have been feasible and/or there was no pressing need to make the study to begin with. When dealing with orphan diseases, you’re pretty much stuck with small sample sizes. That can hardly mean that trying to find a cure or at least a remedy against the worst symptoms is “bad science”. It’s just difficult to do reliably.
Good science can do a lot of things, but it can’t shape reality to fit what’s desirable. In general, these are good points, but it misses out on the issue of feasibility. The main point should be to understand that bias can be introduced and what bias can be expected, so that if all else fails, efforts can be made to control for it.
Compound Interest
Thanks for taking the time to comment!
The points you make are all inarguable. The goal with this graphic was to create a succinct list of points with a short, readable summary for each – in the process, it was necessary to slightly simplify one or two of the more complicated points, in order to fit them into the available space. Particularly, on points such as blinding, to put in a disclaimer that it’s not relevant in all trials, whilst absolutely correct, would have diminished the amount of text available for explaining it, which I decided to prioritise.
I agree it’s an unhappy compromise, but if you’ve got any suggestions for modifications of the text within the small scope available, it’d be great to hear them! I’m currently making a few changes based on feedback so far, which’ll hopefully address a few issues that have been brought up.
Brant Cebulla
Well stated. On top of some feasibility issues, it should also be noted that we need observational studies and small interventional trials before we can move forward with large well-designed trials. Calling these types of studies “bad science” lacks perspective.
In medicine, we usually move from observational studies (#3, #7) -> open-label trials (#8, #9) -> small RCTs (#6, #10) -> large RCTs. The observational studies, open-label trials and small RCTs are necessary components to the progression of discovering something valuable in medicine. It doesn’t mean that these studies are bad science; it’s just that they are not the highest of quality of evidence.
Perhaps a more apt title would be, “Lower quality science.”
Compound Interest
Hi Brant, thanks for the comment. I completely agree that small trials are absolutely a part of science, and with this graphic I’m not trying to say they should be disregarded. As the title, ‘A Rough Guide’, suggests, the aim here was merely to flag up a few points that *may* contribute towards the science or conclusions of a study being unreliable. Too many people will read an article online and it won’t even occur to them to look past the substance of the article into how reliable the research it was based on is. I suppose to an extent they are the target audience of this graphic, though admittedly they’re probably the harder demographic to hit.
I’d say it’s perfectly acceptable for a study to hit one of the criteria in the graphic, say a small sample size, or no blinding, but for its conclusions to still be completely valid. In reality, it’s not something we can put a catch-all ’12 Tell-Tale Signs of Bad Science’ tag on – people need to consider each of these points on a case-by-case basis, and ultimately decide whether the conclusions of the study are realistic in light of this.
Brant Cebulla
I have friends — smart people but not versed in evidence-based medicine — who often understand a few of these points you list and overuse them to analyze studies. The “correlation is not causation” argument is classic; people get very giddy to share this concept; it’s cute. However, dismissing studies alltogether because they have inherent limitations — e.g. an observational study — is almost as blind sighted as not understanding potential limitations in the first place. So in addition to teaching about bad science, I’d love to see an infographic on the totem poll of evidence: http://www.cebm.net/?o=1025
And goes for teaching about medicine in general. I’d love to see more focus on teaching about evidence-based medicine rather than teaching about bad science (of course, they mostly go hand in hand).
Cheers, Brant
Compound Interest
Something focusing more on the processes involved in research could be a good idea for a future post. It’d be great to give people a clearer idea of the steps involved in the research process, particularly in the field of medicine.
ohminus
You can only teach the willing, though, Brant. Given that I once had a laboratory specialist (you should think these would be the number crunchers among physicians) tell me that he didn’t want to hear a maths lecture when I tried to explain statistical constraints on measurement precision, I sometimes have my doubts. And I’ve likewise seen it too often that industry had to stage statistics workshops to enter a meaningful discussion of the pros and cons of certain studies because the necessary knowledge simply wasn’t there in parts of their customer base…
Al_de_Baran
“The ‘correlation is not causation’ argument is classic; people get very giddy to share this concept; it’s cute.”
Almost as cute as your condescension and your straw man?
Few intelligent individuals, whether “versed” in the mysteries of the scientific priesthood or not, “dismiss” studies because they are muddy with respect to causation. Such commentators merely indicate that fact, as well as the fact that some studies (and reports thereof) do a better job of clarifying the relationship between correlation and causation than others.
Oh, and when urging us to learn about “evidence-based medicine”, don’t forget the evidence-based medicine that brought us such wonders as Vioxx.
ohminus
And after the large RCTs we nowadays go back to observational trials again to observe real-world use…. both in terms of adverse effects as well adherence and ACTUAL clinical and economic benefit.
RCTs are the equivalent of a laboratory experiment: They are great to isolate certain factors and test for them – but we should take great care not to jump to the conclusion that this result is actually relevant in a real-world context.
It’s actually ironic: years ago, payers were all demanding more RCTs, more RCTs, more RCTs – now they are all too often complaining that the results observed in RCTs don’t manifest in actual use.
uptodate
On point 9 – some modification might be in order to say that in certain circumstances such as in vaccine testing a controlled randomized double blind study against a placebo would be unethical (antivax folks love to pull that one as a way to “prove” vaccines have not been tested adequately for safety). We wouldn’t do that with car seats but other ways of testing efficacy proves them to be a valuable tool in helping to prevent serious injury in car crashes for example. The skeptical OB did a good blog post on this point.
Compound Interest
Thanks, I’ll try and squeeze that in somewhere in the updated version.
uptodate
Sure I just know it’s graphs like these that can be grabbed by the antivax crowd and yes, cherry picked in such a way as to totally distort and supposedly support their POV. Maybe a simple “when appropriate ethically as well as feasibly double blind studies…” Others may have a better way to say it but that is just one option. You could even put a * to point to more detail about that particular concern. Just a thought and thanks!
Compound Interest
Thanks for the rewording suggestion – managed to fit it in quite comfortably! The updated version will be up shortly.
uptodate
Awesome. Looking forward to sharing it with my group of vax advocates.
Marco Filipe
It’s a great guide, especially for general use of people not trained in science. I was wondering if it’s ok to translate it to portuguese, I’m part of a skepticism/science non-profit organization and this is the kind of subject that we usually have to deal with. It would be great if it could be shared with non-english speakers. We will give you full credit and direct people to here, of course.
Compound Interest
Hi,
Thanks for the comment – if you’d like to translate the text to Portuguese, with a link back to the site, that’d be great! All I’d say is to hold fire until this evening, as I’m currently editing a few points based on feedback that’s been submitted, and then I’ll update the image/PDF.
Marco Filipe
Ok, I’ll wait for the new version. Thanks!
Compound Interest
Updated now!
ABA Prof
Important to note that Single Subject Designs do not use random assignment, because we are interested in within-subjects variability. Most people who use single subject designs are interested in individuals on the tails of a normal distribution. We do “cherry pick,” so to speak – but we also include a detailed subject description and are cautious about over-stating the generality of our findings. So a well-designed, replicated Single Subject Design would not fit into Numbers #7, #8, or #9. (Participants serve as their own control by observing their performance over time.) I know that this is a general knowledge poster, but just wanted to point out those details for follow up.
Compound Interest
Thanks for the tip – I think it’s probably getting to the stage where I need to start including a long list of disclaimers in the accompanying post, as they’re not all going to fit onto the graphic itself 🙂
Tara
This is fantastic, and I’m looking forward to seeing the revised version with others’ suggestions incorporated. While you’re updating, please correct #2: “misinterpreted” is misspelled.
Compound Interest
Damn, just changed this to the newer version before noticing that. As soon as I get time I’ll fix it, cheers for the spot!
Dave Wasser
Thanks for creating this graphic. It’s very good.
suzi meyer
Here are some of my comments. Note that I am a social scientist doing human subject experiments and this list may not have been intended for this kind of research. However, it seemed to be positioned very generally so here’s my take:
#1 not a fan of sensationalizing but it’s not necessarily a sign of bad science. unfortunately, this is also a direct outcome of the “publish or perish” system.
#5 with the exception of math and its applications, science will always use ‘may’, ‘could’, ‘might’. it’s a stochastic world and that’s why we need/use statistics.
#7 representativeness with regard to variables of interest is what matters. otherwise we’d need clones for replicates.
#8 last sentence: there’s no way you can control “all” variables in human subject research. you control what you can and the rest will be included as noise. you just try to prevent systematic biases.
#9 i’d add “and/or random assignment”.
#10 may be too hard to detect, especially due to file-drawer factor.
#12 disagree with this one. no need for scientific elitism. science, nature, etc. have their own issues and biases. there are a ton of other quality journals that don’t have the same ‘wow’ factor. also citation counts may not be the best indicator of quality.
These may not be relevant to the original intention behind the list. If so, it would be useful if you could either specify the target audience/discipline or clarify some of the assumptions underlying the items you listed. Thanks for receiving comments!
Compound Interest
Hi Suzi, thanks for your comment.
Obviously, the graphic was entitled ‘A Rough Guide to Spotting Bad Science’, with emphasis on the ‘rough’! Your comments seem to pertain to the original version of the post, rather than the updated version that’s here on the site, so you’ll find that several of your points may already have been addressed. In addition, I’ve also responded to some of the other comments, some of which made similar suggestions.
As far as the intent of the graphic goes, it is meant to be general, though since this is obviously a chemistry-related site, it was originally made with the natural sciences in mind. The idea was to point out some of the factors that may contribute towards ‘bad’ science, and I’ve tried to make it clear that the presence of some of them does not automatically denote that the research should be written off. It’s very much directed at a general audience, rather than for academics – I’m sure there are plenty more detailed articles out there for those seriously critiquing research methods, and this isn’t intended to be a comprehensive analysis. Obviously, dependent on the branch of science, some of these points may be less relevant, but it would take a little longer for me to make one for each branch of science 🙂
Hopefully this answers a few of your enquiries, and the latest version of the graphic addresses a few of your concerns.
suzi meyer
Thank you and I apologize. I saw the updated version after having posted my comments. Yes you have addressed several of those comments already 🙂 Thank you!
Compound Interest
No problem 🙂 Glad it’s corrected some of the ambiguities!
Richard L Harnack
Good job. This applies fairly much to any intellectual endeavor. I will recommend it to my history students.
unbound55
Sadly, the majority of dietary scientific studies fall victim to 2, 3, 4, 9, 10 and 11, which is why you keep reading conflicting studies all the time. So while a lot of science is done well in many disciplines, not all disciplines follow scientific principles equally.
Corry lazarowitz
I rarely comment directly, but this is something that bothers me a lot! So, here are 2 more I typically find in junk “science” . Unpublished data. Often just the results of statistical analysis is given without any access to the data, which leads to my next one, “models” used without explanation. Without the data, ad the math for the model, no one can even attempt to replicate the results, and it’s difficult to see how they manipulated the numbers to get their sensational headlines. Plos one has even now started requiring all data to be public!
R.C.
TOP!
Robert Mckeown
Wonderful … printed out, up on my wall now. Thanks
Compound Interest
No problem!
うさじま うさこ
Hello.
I think this graphic is great!
Would you me allow to translate it to Japanese?
I’m running a blog about bad science and anti-vaccination rumors in Japanese, and trying to introduce helpful information for spotting bad science by translating it from English to Japanese.
I’d like to share this great graphic with Japanese people.
Compound Interest
Sure! If you like, translate the text then send it over, and I can plug it into the original graphic to preserve the formatting and font. Drop a message to my email through the about page and we can discuss it further!
うさじま うさこ
Thank you!
It will be a great help for Japanese people.
Well, I post my translation on my blog and have some discussion with my friends to refine sentences.
Then I will send it to you.
Andi Ramsay
The first thing to grab my attention is the incorrect pluralisation of “3. Conflict of Interests”, this should be “3. Conflicts of Interest”. Please make this right! 🙂
Everything else is great, so helpful to the masses, let’s hope they read it!
Compound Interest
Hi Andi, thanks for the comment! Regarding ‘Conflict of Interests’, this is in fact grammatically correct – I even went and double checked to be sure when making the graphic, and it’s used on the European Research Council’s site. Obviously, ‘conflicts of interest’ is correct too, but I’m going to stick with the one already on the graphic for the sake of convenience 🙂
OlinR
I would be a little careful about “speculative language.” There are times a conclusion is tentative, because it is necessarily tentative. Medicine, in particular, often requires multiple studies to make a real conclusion. It’s more a problem when the speculation or hedging is taken as gospel truth that is the problem.
Enku Ide
as a social scientist, I’d say this is applicable to “normal science” and positivist science, but much less so to post-positivist science that many of us engage in.
ohminus
Could you explain that? What is “normal science” vs post-positivist science for you? As I understand it, post-positivism embraces a probabilistic concept of knowledge, which is very much supported by these points, especially those who touch on statistical aspects such as sample size
Andrew Turrisi
Perhaps in human medicine, where trials are expensive, retrospective research compares treatments but two other glaring issues arise here: SELECTION. why did each group get assigned to treatment A or B; and FOLLOW UP: when one looks for late effects of a new treatment, they may not appear for MANY years, even decades. Reporting events at 3 years when the worst effect may occur at 10 or 15 years was just reported in the Journal of Clinical Oncology with a shorter less expensive treatment looking just a little worse at three years, when the peak effect, urinary retention, urethral fibrosis, may not emerge for 5 or 10 years.
Lionel GUERIN
To be scientifically correct, critics of “bad science theories” should apply the principles they recommend to bring proof of what they denigrate.
Misinformation, ridicule and censorship are not scientific arguments.
Nancy darby
I’m not sure this was meant for the average Joe or Jane, and my own interpretation, but this graphic helps clarify a lot of what I find grossly flawed with the teacher evaluation systems that are running rampant through the public school systems today. A little attention to the scientific method would be appreciated. With your permission, I will post in my middle school Health class for our future thoughtful citizens to ponder.
Compound Interest
Absolutely, feel free to use it in your classes – that’s the idea 🙂
Benny Sanders
Funny thing… as I was discussing vaccinations with someone, I came upon this piece. This in a nut shell, describes Jenner’s research on Smallpox and the vaccine… almost exactly. Amazing. Oh, and the current state of some ‘clinical trials’, especially the ‘cherry-picked’ results.
Robert H. Woodman
Andy, thanks for this. I just found it recently, and I shared a link to this page on Google+ with the Science on Google+ community.
Compound Interest
Thanks for sharing! It’s great that, even a year on, this post is still proving popular and generating discussion. Whether people agree with every individual point or not is largely immaterial, but it’s good that it’s getting people to consider and debate what constitutes ‘bad science’.
Harnley Barn
This article was written on the money. The net is a spawning ground for bad science. Lets put it this way. Science is tadpoles, frogs, misquitos, asthma, molecular science, cures for disease, inventions, space travel. If it says “Gay men make more money, that is just pure
lies.” That isn’t science. Good job
Gabriel Rosa
This is an updated link for the Portuguese language version: http://bit.ly/1b6eXtu
nico_la
Any chance you might offer this as a full-sized poster? I’d like to put it up in my office :D.
fustbariclation
Missing:
– Original data not available
– Negative results not reported
– No independent confirmation in a peer-reviewed journal
– Use of arbitrary frequentist stats, rather than Bayesian stats
– (closely related) Abuse of p-values
Andrew Beale
I would like to share this with some Mozambican colleagues – is there a Portuguese version available or, could I translate it?
Compound Interest
Hi Andrew, sorry for the delay getting back to you – there is a translation of an older version of this graphic here: http://comcept.org/wp-content/uploads/2015/05/Um-Guia-Geral-Para-Detectar-M%C3%A1-Ci%C3%AAncia.pdf
PharmaceuticsPharmacy Practice
A must have in your research Lab. I just printed it out for our research students
Thanks a lot Compound Interest!
Kate Kate
Another reason why access to all original research should be free… The saddest part about pay walls is that the journals don’t put the profit back into science funding, they’re just paying for some fat guy to sit on a beach.
Comments are closed.