STATS ARTICLES 2003
One AIDS Vaccine, Two Interpretations
March 07, 2003
Statistical confusion rather than news "bias" at fault
USA Today frames the issue of clashing AIDS headlines - USA Today said "Vaccine for AIDS Shows Promise," while The New York Times covered the same story as "Large Trial Finds AIDS Vaccine Fails To Stop Infection," - as one of news bias. The paper's founder, Al Neuharth, wrote that the more positive spin in his paper reflects its tendency to emphasize good news and the Times' skew to the negative.
But the story of the latest AIDS vaccine research is really one about how to understand statistics and scientific research, and in this, USA Today's optimism doesn't really fit the facts.
The trial studied a vaccine manufactured by a company called VaxGen. The research found that, when looking at the entire sample of 5,400 high-risk patients (mostly gay men from the U.S., the Netherlands, Puerto Rico and Canada), those who received active vaccine were no less likely to become HIV-infected than those who received a placebo. In fact, the people who received active vaccine appeared to be slightly more at risk of becoming ill.
However, a subset of non-white participants may have experienced some protection. In this group, 9.9% of those on placebo became infected, while only 3.7% of those who received the real vaccine did.
Sounds promising, right? Those numbers seem to add up to some 70% protection, and for people of African descent in the sample, the number was even higher at 80%. With an epidemic raging out of control in Africa, that should be cause for celebration.
Unfortunately, to determine what the research really means, it's important to look at the actual numbers in a statistical context. Pulitzer-prize winning Newsday AIDS reporter Laurie Garrett detailed them this way:
"Among the 314 black volunteers, VaxGen reported 13 infections: 4 among the 203 volunteers who received the vaccine, and 9 in the placebo group of 111. Of the Asian volunteers, two of the 20 who got the placebo became HIV infected, as did two of the 53 who got the vaccine, said Dr. Michael Para of Ohio State School of Medicine, who led the statistical analysis conducted for VaxGen."
Statistical analysis works by comparing the number of actual events to those which could be predicted to occur merely by chance. The smaller the numbers involved, the more likely it is that chance alone could explain any differences between groups. The numbers here are so small as to be statistically suspect.
Further, the more subset analyses you do, the more likely you are to find the effect that you are looking for. Researchers call this "data dredging." While it can increase the odds of finding a difference that means something, it also increases the odds of finding a chance effect that only looks real.
When one set of numbers is this confusing, context helps. Many AIDS researchers have long been skeptical about the approach used to make the VaxGen vaccine because it did not protect monkeys from infection with the simian form of HIV. It also does not produce the kind of immune response believed to be necessary to control infection in humans. Although vaccine research has in the past produced some surprises where field results are different to those seen in the lab, the U.S. government pulled out of funding for a large scale trial of this vaccine in 1994.