STATS ARTICLES 2004
Pollster-Pundits Lead Media Astray On Wisconsin Primary
February 23, 2004
Ana Marie Cox
Edwards defeats polls (but not Kerry).
For interested voters, last week's most compelling political drama played out in real time on Tuesday night, as returns from the Wisconsin primary indicated that Senator John Edwards was not only doing much better than earlier polls indicated, he might even win. His surprise close second (with 34 percent to Senator John Kerry’s 40 percent) made for exciting television, provided Edwards with a burst of favorable media coverage, and made it possible for voters and pundits alike to consider his winning the Democratic nomination a real possibility.
For pundits and pollsters, the drama of Tuesday night is still being played out – on political chat shows, in op-ed columns, and on internet bulletin boards. The leading character s are not the pols, messrs Edwards and Kerry, but rather the polls that seem to have failed so miserably in predicting the outcome of the race.
Predictably, this failure has prompted criticism in the polling community – not of the polling methodology, but of the media's reliance on polls. Had less been made of Kerry's commanding lead, the thinking goes, then Edward's second place finish would not have been so dramatic and thus not have created the momentum and positive coverage that it did. Kerry's weaker-than-expected win would also not seem to be as, well, weak.
This argument was laid out by the pollsters interviewed by Jim Rutenberg in a New York Times article last Thursday. Geoffrey Garin, a Democratic pollster, said "Virtually every news outlet, based on public opinion polling, created the expectation and standard that John Kerry would have a big victory… So anything that was short of a big victory almost inherently becomes a disappointment"
On the face of it, this seems to be a rather self-serving argument on the part of pollsters. "It's not our fault," the reasoning goes, "we just conduct the polls – we don’t tell people about them." To gauge how plausible this claim may be, let's look at the polls themselves. Was there any indication that journalists should not have taken them at face value or should have drawn back from promoting their results?
An American Research Group (ARG) poll released Feb. 12, and conducted on Feb. 11 and 12, showed Kerry with an apparently insurmountable lead: Kerry 53, Edwards 16, Dean 11.
A Zogby poll released on Feb. 15 showed an almost as dramatic lead: Kerry 47, Edwards 20, Dean 23. Zogby conducted this poll Feb 13-15.
Should journalists have picked up on the ever-so-slight increase in support for Edwards as an early indication of his later surge?
Polls conducted by different companies rarely show the exact same numbers for each candidate, due to the use of different polling methods. In this case, Zogby polled "likely primary voters" and ARG simply "registered voters." Thus, the more engaged attitude of likely voters may have accounted for the stronger support of the lesser-known candidates. Even if you decided to compare the two polls, factoring their margin of sampling error – both at +/- 4 points – makes Edwards’s increase in support disappear.
Just about the only weak spot in the polls is one that went largely unreported until after the vote, when it then became the dominant explanation for Edwards’ victory: the percentage of voters who had not yet made up their minds. (See Tim Rutten's analysis in the Los Angeles Times.)
In each poll prior to the primary, the percentage of undecided voters was unusually large for residents of a state whose primary is only days away: ARG found 16 percent of voters undecided and the Zogby poll found 14 percent. But these undecided voters don’t show up in Zogby's poll.
In fact, the Zogby poll's slightly better numbers for Dean and Edwards, in addition to relating to a more engaged voter pool, may well stem from the Zogby practice of "pushing leaners." That is, a respondent that says he is undecided about who he will support is asked which candidate he feels he will most likely support (who he's "leaning toward"). This eliminates "undecided" as a category in the final polling figures, but at the expense of creating a poll that can appear to show a more dramatic race than is really the case.
A sharp reporter should always note the "softness" of polls that include such leaners in their tabulations; indeed, to the extent that the media is to blame for plumping the prospect of a big win for Kerry or missing early evidence for Edwards's surprise finish, it is simply that reporters did their job as they usually do: They gave just the bare poll numbers; they supplied no indication of how methodology may influence the polls' outcome; they did not give proper prominence to margin of error; and they failed to place the poll within the context of polls that had come before it. But before we lay blame for inaccurate poll coverage solely at the feet of the media, it's worth looking a little more closely at the pollsters.
Pollsters typically present their poll results in press releases that do little to educate journalists as to the meaning of the numbers they contain. In Wisconsin, the pollsters trumpeted the apparent Kerry landslide with the same confident aplomb that voters would see in any newspaper: "Kerry On Pace To Carry Wisconsin, New Reuters/MSNBC/Zogby Poll Reveals"! Or, according to AMG: "Kerry in Commanding Lead Among Democratic Primary Voters in Wisconsin." These are hardly the messages of institutions that wish their results to be treated with caution.
What’s more, both AMG and Zogby are more than just dispassionate polling organizations. John Zogby is a prominent pundit in his own right, called upon to speculate about the meaning of his polls on MSNBC and on the Sean Hannity show. ARG's Dick Bennett is a less visible but still vocal pundit as well, providing newspapers and radio shows with poll numbers and predictions – something you’d think that Bennett, the man who called New Hampshire’s1996 primary for Bob Dole, would stay away from.
And pollster-pundits have the same pressure on them that journalists do: They must make the numbers exciting, which more often than not means making the numbers fit into the dominant story line of the day. "Again, it's all about electability," was John Zogby's sound bite prior to the Wisconsin primary, echoing the conventional wisdom. Dick Bennett chimed in with, "What voters are looking for is who can beat George Bush."
It's not that these sentiments are wrong; it's that these sentiments are hard to derive from poll numbers.
Pollster-pundits are unlikely to talk about what poll findings don’t tell us, such as the strength of Kerry's support. Even more disturbing, the pollster-pundits lend legitimacy to the approach your everyday pundits and journalists take to polls: If Zogby and Bennett don't talk about margins of error or methodology, why should anyone else?