STATS ARTICLES 2008
Worst survey of the year
By Rebecca Goldin, Ph.D
First, Do the Data No Harm
A survey conducted by the Physicians' Foundation finds that almost half of all practicing physicians in the US plan to cut back or quit practicing medicine. According to the survey’s press release, “49 percent, or more than 150,000 practicing doctors– say that over the next three years they plan to reduce the number of patients they see or stop practicing entirely."– begging the question of whether doctors closer to retirement, or unhappy doctors, are more likely to fill out a questionnaire to begin with.
But without a representative sample of doctors, the survey only tells us something about the 4 percent who completed the survey, not everyone. The statistics professor who advised the group that there was a one percent margin of sampling error should consider renaming his department ‘bullistics.’
Reporters covering the story seemed oblivious to the margin of chutzpa. USA Today, in a story headlined “Primary care doctors in short supply,” went so far as to tell readers that “the margin of error is slightly less than 1 percentage point, according to the foundation.”
Perhaps, because the survey tapped into the general well of complaint physicians have about American health care, its lack of statistical rigor went unnoticed, even by doctors.
Ted Epperly, president of the American Academy of Family Physicians, told USA Today that the survey findings were “probably pretty close to accurate. There is a lot of distress out there among my colleagues.”
Writing in the New York Times, Dr. Pauline W. Chen M.D. noted that the “survey indicates that the primary care crisis may not be looming on the horizon; it may already be at our back door.”
Such is the power of confirmation bias. It may well be true that many primary care doctors are distressed about the state of the profession and of health care; but citing this survey in evidence is not a scientific diagnosis.