HOME > STATS > ARTICLES

"The question in Washington," according to Brian Williams on the January 30 NBC Nightly News, "was this: did the Bush administration... try to cook the books on the topic of global warming?" The evidence, according to Andrea Mitchell, was "a new study revealing unprecedented and widespread interference with scientific reports," which was released in conjunction with congressional hearings led by Representative Henry Waxman (D–CA.).

The study received widespread major media coverage. On ABC World News Tonight, Jake Tapper reported, "scientists say their work on global warming has been watered down and twisted by (the) White House...." 1

Next morning the New York Times reported, "Almost 60% of the scientists... said they had personally experienced" incidents such as being pressured to change the wording of their reports, having their work altered by others, and being asked to provide misleading information to the public.

The Chicago Tribune added, "43% perceived or personally experienced changes or edits to their research that changed the meaning of scientific findings."

The study tough talk also stimulated editorials, such as the Minneapolis Star Tribune’s "Bush's blatant abuse of climate scientists."

Science may sometimes be a messy business, but it’s a business that scientists ought to be able to freely pursue – otherwise, what value has it? If there are problems in a particular piece of research, the free market of ideas will ensure that competitors, if not colleagues, will catch them eventually.

Surveying can be a messy business too; but in the case of the report by the Union for Concerned Scientists, the question is whether the media should have been quicker to spot that this was a survey with problems.  And the source of these problems lay in the decisions the researchers made regarding their sample of scientists, their phrasing of questions, and their presentation of results. Together these issues call into question the validity of the findings as reported in the media.

The report was based on a mail survey of government scientists dealing with climate change. The heart of the survey asked scientists about a dozen "types of activities" that they "perceive in others and/or personally experienced, including pressure to eliminate words like "climate change" and "global warning," and requests by officials to provide "incomplete, inaccurate, or misleading information to the public."

The most obvious problem with the study is its low response rate. Only one out of six scientists (17%) who were contacted returned the survey form. This means we don’t know anything about the views and experiences of the other 83%.

Any professional survey researcher (including those employed in the survey units of major media organizations) will tell you that a 17% response rate is inadequate to draw conclusions about the group being surveyed, in the absence of other information demonstrating that the sample is representative. As Harris Poll chairman Humphrey Taylor recently noted,

"A low response rate does more damage in rendering a survey's results questionable than a small sample, because there may be no valid way scientifically of inferring the characteristics of the population represented by the non respondents…. Many of the leading peer-reviewed academic journals will not accept papers based on surveys for publication unless they have a response rate of 50% or more."

In this case there is also an obvious danger of selection bias. The scientists who took the time and effort to fill out and return the questionnaire might be precisely those most upset about perceived interference. We simply don’t know if this made the problem appear greater than it actually is. In addition, no federal directory of climate scientists exists. Therefore UCS had to create their mailing list by searching through government websites for staff titles, checking professional activities, and soliciting expert opinion. To illustrate the difficulties of accurately identifying a population in this manner, 20 percent of those who returned the questionnaire possessed only a bachelor's or master's degree in their field.

There is also a serious problem with the questionnaire itself. Scientists were asked not only about their own experiences but about their perception of others. For example, the Los Angeles Times reported that "nearly half of all respondents perceived or personally experienced pressure to eliminate" certain words. The actual wording was, "I have perceived and others and/or personally experienced the following types of activities affecting climate science:" This was followed by a list of activities that the scientists marked as "perceived" "experience," or " neither."

This wording creates a statistical phenomenon that artificially inflates the impression of a hostile work environment. Consider an agency that contains 10 scientists. One tells the other nine that he has encountered interference. When they are surveyed, all ten report that they have "perceived in others and/or personally experienced" interference.

So one act of interference is counted as ten acts that are "perceived" or experienced"; ten percent of the scientists have been interfered with, but 100 percent report "perceiving in others and/or personally" experiencing interference. If the agency contains 100 scientists, the interference experienced by one becomes "perceived" interference by the other 99, and so forth.

Further, there is a great deal of social scientific evidence that such perceptions are often incorrect. The tendency to believe that others will be influenced by forces to which we ourselves are immune (e.g., by misleading advertising or partisan rhetoric) is so common that sociologists have a name for it -- the "third person effect."

In addition, people’s perceptions of their own situations may be colored by their prior beliefs and emotions. In fact, out of the 12 "activities" listed, nearly half are actually mental states or inferences: "self-induced pressure to change research," "fear of retaliation" for expressing concerns either inside or outside one’s agency (listed as two separate activities), "implicit expectation by officials for scientists to provide" misleading information. (Of course, the researchers also assume that the scientists were always right and their superiors always wrong in urging them to eliminate certain words, requesting that they insert opposing views, making changes and edits etc.)

Finally, the researchers assume that all these responses refer to officials’ efforts to alter certain kinds of findings about global warming. But that is not specified in the questionnaire. For example, a scientist could encounter "new or unusual administrative requirements or procedures that impair climate related work," without the requirements having anything to do with the debate over global warming.

Not surprisingly, such complications confused many of the generalist reporters who covered the story. For example, NBC’s Andrea Mitchell was flat wrong in saying, "A survey of more than 300 scientists in seven agencies studying climate change found nearly half were personally pressured to eliminate the words ‘climate change’ or ‘global warming.’"

Apart from the number of scientists and agencies being wrong, only 21% reported experiencing such pressure personally. The remaining 33 percent "perceived" it "in others." Many other reports such as the LA Times story noted above) included the phrase "real or perceived" without noting any of the problems this pairing concealed.

The largest number of all appeared in the New York Times report that, "Almost 60% of the scientists... said they had personally experienced" some type of interference. How did the number of personal experiences grow so large this time?

Because it represents the proportion of scientists who reported that at some time in the past five years they had experienced any one of the dozen conditions that were listed, including self-induced pressure, the fear (but not the fact) of retaliation, the perception that officials had inappropriate "implicit expectations," disagreeing with editorial changes in their reports, etc.

This may represent the "unprecedented and widespread interference" that Andrea Mitchell reported, but it also may include the typical disagreements and personality conflicts found in any office populated by highly skilled knowledge workers.

Notably, none of the news reports referenced some countervailing findings of the study. For example, only one in five scientists (21%) disagreed with the statement, "US federal climate research is independent and impartial." And only 8% disagreed that, "my agency’s leadership aspires to and expects a high level of integrity and professionalism."

Finally, among the dozen "types of activities" listed, there was a straightforward measure of overt behavior rather than mental states, imputed motivations, or disputable perceptions of right and wrong: "Situations in which scientists have actively objected to, resigned from, or removed themselves from a project because of pressure to change scientific findings." Only 6% of the scientists who were surveyed reported experiencing such a situation in the past five years. Of course, that's 6% too many. But it's a far cry from the 60% reported in the Times.

1. After this piece was posted, ABC correspondent Jake Tapper informed us that ABC News also found the study too flawed to draw statistical inferences from it, and that he intended his comments to refer specifically to those scientists who testified before the committee. We regret any misleading impression this may have caused.