HOME > STATS > ARTICLES

Hyping Echinacea with Bad Numbers
Julie J. Rehmeyer, December 28, 2007
The press mangles the statistics on an unpersuasive Echinacea study.

To read the headlines, June 25 was a banner day for cold sufferers:

“Echinacea, the North American flower widely used to protect against colds, actually works - and works well - a scientific review found,” read the Bloomberg News. “The plant, also called the purple cornflower, cut the chances of getting a cold by nearly two- thirds compared with a placebo.”

But before you rush out to buy some Echinacea, it pays to double-check those statistics. First of all, the study – “Evaluation of echinacea for the prevention and treatment of the common cold: a meta-analysis” (Shah, S.A. et al., Lancet Infectious Diseases 7 (July), 2007) – didn’t show that Echinacea cut the chance of getting a cold by nearly two-thirds. The accurate number is less than one third. Unfortunately, every major news organization that gave figures about the study’s claims for Echinacea’s effectiveness made the same mistake, including the New York Times and the Los Angeles Times.

Furthermore, this “scientific review” is nothing more than a rehash of old data, sliced and diced to provide a different and less-than-convincing result. That’s not to say that the study is without value, but it’s far from a definitive proof of Echinacea’s efficacy – as even the author of the study admits.

“We’re not saying without doubt that Echinacea works,” says Dr Craig I. Coleman of the University of Connecticut School of Pharmacy, an author of the study. “Ultimately, what we’re trying to suggest is that more studies should be done.”

That’s a bit of a let down from the definitive statement that Echinacea “actually works – and works well.” In fact, the best study to date has found that Echinacea does not have a statistically significant effect, and this new study does little to revise that perspective.

The problem with the media’s reports boils down to statistical illiteracy: Few reporters who covered the story understood the difference in significance between a  top quality, randomized, double-blinded, controlled study and a questionable meta-analysis. And apparently none of them had the basic statistical knowledge necessary to report the claims of Echinacea’s effect accurately.

Making a Hash of the Data
So what is a meta-analysis, and why do researchers do them?

Think of a meta-analysis as a powerful but tricky microscope. If used correctly, it reveals details you can’t see otherwise. But if used incorrectly, it can distort the picture so much that it becomes unrecognizable.

Chance is the fundamental problem that obscures medical researchers’ ability to see the effects of a treatment. For example, antihistamines may work great for my allergies but leave my friend just as allergic – and sleepy to boot. If someone studying an antihistamine’s effectiveness against allergies happens to get a group of folks mostly like me, the antihistamine will look like it works terrifically, much better than it actually does. But if the researcher gets patients who are mostly like my friend, the drug will look like a dud.

One way of protecting against this problem is to study lots of patients. After all, if a researcher studies ten thousand people, it’s pretty unlikely that the great majority of them will all respond unusually strongly or unusually weakly to the treatment.

Small studies can still show definitively that a treatment works if its effect turns out to be really big. Suppose, for example, that a study of a cancer drug contained only twenty patients but their tumors all vanished after receiving the drug. In that case, the researcher would be pretty sure the drug was doing something. But the smaller the number of patients, the bigger the effect has to be for the researcher to be sufficiently confident.

This is why statisticians developed the notion of “statistical significance.” For any size population, statisticians figured out how big an effect you need to see in order to be 95% sure that you’re not just seeing random variations. An effect that is smaller than that is said to be statistically insignificant. So in a small study, a treatment may seem to have a sizable positive effect but still be statistically insignificant. Then the researcher is left wondering whether the effect was real but the study was too small to detect it with confidence, or whether the treatment just didn’t work.

That’s just the situation that meta-analyses can sometimes help with. If several small studies have been conducted on a single treatment, a researcher can combine the data from all the studies together and analyze them as if they were from a single large study. Then the researcher can see small effects that were invisible in the individual studies.

It’s a great idea, and if done carefully, it can be revealing. But meta-analyses can fall into lots of traps. Combining studies is only legitimate if they’re really studying the same thing, but most of the time, studies have important differences in design. Furthermore, if the original studies are poorly designed, the meta-analysis will be lousy too. Another problem is “publication bias:” If a treatment has a small effect, some studies will probably, by chance, show a negative effect. But studies that get negative results very rarely get published, so the researcher doing the meta-analysis will get an artificially positive collection of studies.

The result of all of these things is that the quality of the meta-analysis depends largely on the judgment of the researcher selecting the studies. Different meta-analyses that include different studies can come to strikingly different conclusions.

As a result of that, most doctors and researchers view meta-analyses with a fair bit of skepticism. “One good experiment – controlled, randomized, double-blind, with a reasonable number of subjects – beats a meta-analysis of any number of observational studies,” says Philip Stark, a statistician at the University of California, Berkeley.

The recent Echinacea meta-analysis has been criticized for all the weaknesses meta-analyses so often have. The original studies vary widely. Some administered the cold virus to the participants and some just observed whether the participants got colds on their own. Some used one species of the plant and some another. Some studied Echinacea mixed with other products like vitamin C or propolis. And, the critics say, some were well done and some were simply badly designed.

Back in 2005, Ronald B. Turner of the University of Virginia School of Medicine and his colleagues performed the most careful study of Echinacea to date. They created their own tincture of Echinacea so that they could carefully control the potency. Then they divided 437 participants at random into two groups and gave half of them Echinacea and half a placebo for a week. Neither the participants nor the nurses administering the treatments knew which was which. Then the nurses inoculated them with a cold virus. The patients stayed in hotel rooms for the next five days, and the nurses monitored their symptoms.

To everyone’s disappointment, the effect of the Echinacea was not statistically significant. And this study met all of Stark’s criteria and more – it was randomized, double-blind, and placebo-controlled.

Even as carefully as Turner had designed and performed his study, there were still criticisms of it. Turner used Echinacea angustifolia, and some say that a different species, Echinacea purpurea, is more effective. Furthermore, some argued that Turner should have used a higher dosage of Echinacea than he did. So despite the disappointing results of Turner’s study, it’s not unimaginable that some form of Echinacea, in some dosage, is effective.

But it hasn’t been proven. And Coleman’s meta-analysis isn’t enough to outweigh Turner’s careful study. At most, it suggests exactly what Coleman says it suggests: that more studies should be done.

Looking at the Numbers
In general, if a meta-analysis shows a surprising result, something is fishy. And if it shows that a treatment has a surprisingly, large effect, something is even fishier. Remember that a meta-analysis is like a microscope: it helps to show small effects that can’t be seen by smaller studies. As the press reported it, Echinacea reduced colds by 58 percent – an enormous amount. If Echinacea were really that effective, previous studies would have clearly shown it.

That should have been enough to set off some alarm bells. And an inspection of the numbers shows that the press simply didn’t know enough statistics to read the study correctly.

Computing how much Echinacea reduced the chance of getting a cold is pretty straightforward from the data in the study. The participants who received Echinacea got a cold about 65 percent of the time, whereas the participants who got a placebo instead got a cold about 45 percent of the time. So those who took Echinacea got about 30 percent fewer colds ((65-45)/65), not 58 percent fewer colds.

So why did all the news reports say 58 percent? Because the study stated that Echinacea reduced the odds of getting a cold by 58 percent. In regular speech, we use the words “odds” to mean the same thing as the word “chance,” but in statistics, they’re different.

The odds of something happening are defined as the chance of it happening divided by the chance of it not happening. Gamblers tend to talk about odds, but not many of the rest of us do. Instead, we keep it simpler and just talk about the chance that something happens.

The study was absolutely right that the odds were 58 percent lower for the Echinacea users (though one might suspect that the researchers chose to report the odds rather than the probability because of the more dramatic percentage). Among the participants who took Echinacea, about 45 percent got a cold and 55 percent didn’t. So the odds of getting a cold were 45/55, or .81. Among those who received a placebo, 65 percent got a cold and 35 percent didn’t, so the odds were 65/35, or 1.88.

The reduction in odds, then, was (1.88 - .81)/1.88, or 58 percent. But that 58 percent does not mean that the Echinacea-users got 58 percent fewer colds. In fact, it doesn’t mean much of anything that an ordinary person can relate to.

I think I’m getting the sniffles…
What does all this mean for the regular person who is trying to decide what to do at the first sign of a cold?

First, science hasn’t proven that Echinacea works. Turner’s study shows pretty definitively that 900 milligrams a day of Echinacea angustifolia doesn’t help significantly. The most positive studies have been done on the leaves and flowers of Echinacea purpurea, but none of these studies are of the size and quality of Turner’s study.

So the evidence for it is pretty weak, though it’s also true that even placebos, which have no active ingredient, often make people feel better. If you’ve been taking Echinacea and you feel like it’s helped you, you may be right even if it isn’t having a direct biological impact.

The Risks from Echinacea
Echinacea’s questionable benefit has to be weighed against its risks. The safety of Echinacea hasn’t been extensively studied, though it seems to be well tolerated by most people. Even so, some people definitely shouldn’t take it, like those with auto-immune disorders like asthma or lupus. Echinacea also may well interact negatively with many common prescription drugs, like statins, antidepressants, and protease inhibitors for HIV. Consulting with your doctor before taking it would be wise. And, if you do decide to take it, you should only do so for short periods, either at the first sign of a cold or for a few days before a situation where exposure to a virus is likely, such as air travel.

You might also want to consider that native Echinacea species are dwindling because of habitat reduction and over-harvesting. But, in any case, you shouldn’t take it on the basis of false claims in over-hyped news reports.

Reporting on Studies of Herbal Medicine
Coleman’s team performed their meta-analysis as the first step to convince funders to support a study of Echinacea in cardiac patients. For that, all they needed to show was that there’s some reason to believe that Echinacea might work. And they did do that – it might. Whether the chance is great enough to justify spending money on the further research they have in mind is for their potential funders to decide.

But the media wasn’t savvy enough to realize that this was all the study implied, and so we ended up with news stories reporting it as a vindication for Echinacea. To their credit, a few of the reports did include quotations from skeptical outsiders and pointed out explicitly that the meta-analysis used no new data (though all of them still fell into the statistical trap about odds).

Despite this effort at balance, the news stories contributed to a kind of ping-pong effect that only serves to erode public trust in science. After all, if I like taking Echinacea (or any other treatment) and I read a news story about a study showing that it doesn’t work, why should I pay any attention? Within a year or two, I’ll almost certainly read another story vindicating the herb.

The press needs to approach studies of herbal medicines with special caution because of the difficulties in performing rigorous analysis of their efficacy. Most plants, like Echinacea, come in a number of closely related species with slightly different properties. Depending on growing conditions, they contain different compounds in different strengths. There is no regulation governing manufacturers, so pills often don’t contain what they say they do, and researchers who use those pills can go awry as a result. Most studies are poorly funded and done on a small number of patients. When the studies are bad, as they so often are, they simply shouldn’t get covered.

These issues are especially important because many people have an enormous desire for herbal medicines to work. Herbal treatments often address ailments we have no other particularly effective way to deal with, and they seem traditional, natural and romantic at a time when big pharmaceutical companies and traditional medicine are viewed with a great deal of suspicion. The result is that news stories about herbal medicines, especially if they are positive, play to an audience eager for validation.

The press is guaranteed to receive a steady supply of studies claiming to show that these treatments are effective, because it is impossible to prove that any herbal treatment is completely hopeless. There always might be some other species or some other dosage or some other combination of herbs which might be effective. And given the emotional and commercial desire for success, researchers will continue to try to show effectiveness and, occasionally, will appear to succeed. But only some of those efforts are based on research that is high enough in quality to merit public attention; the press needs to do more – much more – to distinguish such research from the rest.

View the Technorati Link Cosmos for this entry

comment
email this page to a friend