"Where there is data, the best kind of study to base behavior change on is a systematic review of randomized, controlled trials.

The second best is a randomized controlled trial.

The third is an observational study." — Kay Dickersin, Director of the Center for Clinical Trials and Evidence-Based Healthcare and Professor of Epidemiology at Brown University Medical School

Even the best-designed study in the world can occasionally produce impressive looking results by chance. This is why scientists talk about “replicated” research. Until someone else has repeated the results, new research is usually seen as provisional. The more studies there are that have similar results, and the more research there is in an area supporting these outcomes, the more likely it is that the effect is real. If there are dozens of studies supporting a particular conclusion, that’s usually a sound basis to change your activities to reflect that risk or benefit.

“The way science goes back and forth, one study never settles things, says George Gray, Acting Director of the Harvard Center for Risk Analysis and co-author of “Risk: A Practical Guide for Deciding What’s Really Safe and What’s Really Dangerous in the World Around You” (Houghton Mifflin, 2002). “People can get too excited and change their behavior because this week oat bran cereal isn’t good for you any more. And then they start to think that science is either incompetent or [they] stop paying attention to anything and put ‘don’t smoke’ in the same category as ‘should I worry about acrylamide in french fries?’”

While there are hundreds of studies repeatedly and consistently showing the health risks of smoking, the risk from French fries (outside of that from excess fat and calories) has not been demonstrated. Look at the body of evidence, advises Gray: “Something real and strong enough to make me change my behavior will be seen time after time, like smoking and lung cancer.”

meta-analysis
Researchers have devised methods for looking at the body of literature on a subject and mathematically summing up what it means. This is known as meta-analysis or systematic review. It can be useful to sum up large bodies of evidence, although one should note that, sometimes, meta-analysis can be controversial. Some researchers find the process flawed when evidence is pooled from numerous small studies, analyzed as though it was really one large study, and then comes up with an effect that was too small to be seen in most of the original data. Because this effect may be an artifact of the way the data was re-analyzed, more research would be needed to support behavior change in such cases.

converging data
Similarly, if there are different types of research that support a particular conclusion, that makes it even more likely to be sound. For example, when observational and experimental studies show the same thing, and when it is seen in both humans and various animal species, it’s much more likely to be correct.

risk home

STATS home