The publication of the paper that I co-authored, Quantifying the consensus on anthropogenic global warming in the scientific literature, in May 2013 caused quite a splash. The paper received a lot of positive coverage, including Tweets from Barack Obama, Al Gore and Elon Musk. (They didn’t always get the details quite right: our survey was of the literature, not of scientists’ opinions and we had nothing to say about how dangerous climate change would be.) The paper has been downloaded, as of June 2nd 2014, 183,335 times, which is a record for any Institute of Physics paper. The editorial board of the journal, Environmental Research Letters, awarded the paper the “Best article of 2013” prize. The research for the paper was done by a team of unpaid non-specialist volunteers of students and industrial and academic scientists, along with other enthusiasts and the funding for its publication was raised by donations from Skeptical Science readers. No taxpayers were harmed during the making of this article.
Not everybody was pleased to see our work in print. In particular, many people who are unconvinced of the urgency to act on mitigating climate change have been claiming that our study is flawed and biased. Even somebody like economist Richard Tol—who accepts that global warming is real and caused by humans and who acknowledges that there is an overwhelming scientific consensus—has made extraordinary efforts over the past year to get a critique published. He finally succeeded, with the paper Quantifying the consensus on anthropogenic global warming in the literature: A re-analysis published in the journal Energy Policy,. The journal editors somehow came to believe that this article fell within its scope of being “an international peer-reviewed journal addressing the policy implications of energy supply and use from their economic, social, planning and environmental aspects“.
Energy Policy was good enough to give us a chance to respond, which we did with this short paper, which has now (June 21, 2014) been published. We were only allowed 1000 words, however, so we wrote a much longer article, published on the Skeptical Science website, detailing the 24 errors that Professor Tol made.
Click on the graphic to view the document
Somebody (I have no idea who this person is) anonymously uploaded a short paper that harshly criticized part of Tol’s analysis. Tol quickly replied. However, the easiest criticism of Tol’s estimation of the consensus at 91% is that this requires that there are 300 more abstracts in our sample that we somehow missed. I examined a big chunk of the 12,000 abstracts myself and I know that those 300 AGW rejection abstracts do not exist. It would be easy for anyone to prove me wrong, just point to them, the data we used are all freely available.
On Consensus Denial
Why do people like Richard Tol try so hard to dispute a paper whose central conclusion they broadly accept? Why do denialist pundits keep claiming that our consensus paper is flawed when their champion himself admits that we were not so much wrong as merely stating the obvious :
Click on comment for link to original
Writing on a different subject altogether—the denial of evidence of growing inequality—Paul Krugman noted:
By the way, I’m not accusing [Financial Times journalist] Mr. Giles of being a hired gun for the plutocracy, although there are some self-proclaimed experts who fit that description. And nobody’s work should be considered above criticism. But on politically charged issues, critics of the consensus need to be self-aware; they need to ask whether they’re really seeking intellectual honesty, or are effectively acting as concern trolls, professional debunkers of liberal pieties. (Strange to say, there are no trolls on the right debunking conservative pieties. Funny how that works.)
I don’t know what motivates Professor Tol, but claims that he is acting out of concern for propriety and professional standards ring hollow when you examine his own track record. For example, read the sorry history of the Tol-Ackerman dispute and the extraordinary open letter to Tol written by a long list of distinguished economists, or the recent article by Andrew Gelman and the discussion that follows.
Consensus Matters
A recent report by the Yale Project on Climate Change Communication contains the following graphics:
When Professor Tol claims that: “The consensus is of course in the high nineties. No one ever said it was not.” he is seemingly unaware that in fact only 12% of Americans agree with him. Of course, as researchers like Dan Kahan have demonstrated, there are huge cultural influences determining what Americans allow themselves to believe and it is not sufficient to keep repeating facts to change people’s minds. But not all of the “consensus gap” is a result of cultural bias.
Based on research by John Cook.
Even those on the extreme left of the political spectrum think that there is less scientific agreement than there actually is. If we can help close the blue part of the consensus gap, we will have made some progess, at least.
One last little thing
Professor Tol’s paper makes a direct reference in a footnote to something that I wrote in a private discussion.
Putting aside, for now, the shabby ethics of citing stolen material, this interpretation of what I said is false. The quote was part of a discussion that a couple of us working on the project were having about our impressions that some of the abstracts in the sample were repeated. We were not sure if those impressions were real or whether they arose from having read hundreds of abstracts over several days and not remembering each one exactly. We had no way of checking the database for ourselves and we alerted John Cook so that he could investigate further. The notion that I was concerned about the effect of “fatigue ” on the quality of my work is quite false and this is obvious to anyone who reads the quote in context.
I will be asking the editor of Energy Policy to correct this error and to remove any references to stolen private discussions in Professor Tol’s paper. It is surprising and disappointing that the reviewers and editorial staff of the journal did not spot this.
Other blogposts and news items (will be updated):
Skeptical Science Richard Tol accidentally confirms the 97% global warming consensus
Greg Laden The Consensus on Climate Change
Eli Rabett In the Spirit of Deep Climate
Collin Maessen Richard Tol Versus Richard Tol On The 97% Scientific Consensus
Collin Maessen Richard Tol’s 97% Scientific Consensus Gremlins
Dana Nuccitelli (Guardian) The Wall Street Journal denies the 97% scientific consensus on human-caused global warming
Dana Nuccitelli (Guardian) Republican witness admits the expert consensus on human-caused global warming is real
Graham Readfearn (DeSmogBlog) Richard Tol’s Attack On 97 Percent Climate Change Consensus Study Has ‘Critical Errors’
University of Queensland Global Change Institute UQ researcher responds to criticism of research
Graham Wayne Arson Attack on Ivory Towers
Graham Wayne Climate change consensus: the percentage game
John Cook (Skeptical Science) Resources and links documenting Tol’s 24 errors
Richard Tol (Guardian) The claim of a 97% consensus on global warming does not stand up
Richard Tol’s blog Occasional thoughts on all sorts (several articles)
Hot Whopper BUSTED: How Ridiculous Richard Tol makes myriad bloopers and a big fool of himself and proves the 97% consensus
Eli Rabett Model Making, Mathturbation, and Bullshit Tests
Critical Angle 
“The paper received a lot of positive coverage, including Tweets from Barack Obama, Al Gore and Elon Musk. (They didn’t always get the details quite right: our survey was of the literature, not of scientists’ opinions and we had nothing to say about how dangerous climate change would be.)”
True, but I see it a bit differently. I know there is a broad consensus among researchers and others related to climate change. I know this because I can read it, see it, hear it, and I’m always reading, looking, listening. I also know that the dissent is not part of the science at all. I don’t know where you get this crazy 3% figure from, it is very hard to find! The consensus is broad and deep.
And, it is clear that climate change is a great concern, probably the number one policy concern today.
So, I see your paper not as defining consensus, because it was already there and clear. Rather, I see your paper as a single (well done and thorough) sampling of one aspect of the consensus. The null hypothesis is that there is consensus (because it is so abundantly obvious). If that is not true, there should be much less consensus among the peer reviewed papers. Your paper tested that hypothesis. The overall model stands: Climate change is recognized for what it is by scientists and others, and in the literature, and it is serious.
Pingback: Richard Tol's 97% Scientific Consensus Gremlins » Real Sceptic
Pingback: Resources and links documenting Tol's 24 errors
Pingback: Resources and links documenting Tol's 24 errors | Gaia Gazette
Pingback: Resources and links documenting Tol's 24 errors
Pingback: Consensus, Criticism, Communication | Critical Angle
Pingback: Consensus, Criticism, Communication | Planet3.0
Barack Obama reveals himself to be about as scientifically illiterate as I would expect*—however, one of the mistakes he made is understandable: he may have thought your study was about scientists’ opinions because:
1. ‘consensus’ means ‘majority opinion’ in English.
2. the claim that 97% of [climate] scientists agree, as opposed to their PAPERS, has been made repeatedly, and is even trotted out in this very blog post more than once.
*Yeah, I know he doesn’t write his own Tweets, but that’s a violation of Twitter’s very clear rules, which say that we’re perfectly correct to blame him for all activity on his account. (Apparently his Harvard Law degree didn’t enable him to parse simplified legal boilerplate any more accurately than he can parse a “science” paper.)
Thanks, Brad
It’s true that in formal usage we should be careful to distinguish between the broad opinions of climate scientists and the specific opinions that they express in their publications. However, I’m not aware of any cases where scientists believe one thing while expressing a different opinion in the literature. Therefore, mixing up the two categories of opinion when writing informally on blogs and Tweets isn’t particularly misleading, in my opinion.
I think that the more significant error in Obama’s Tweet was referring to “dangerous” climate change, which, I guess, probably less than 97% of scientists would agree with.
I am sure that Twitter is well aware that staffers write Tweets for busy celebrities. What their rules seem to be concerned with is impersonation and I’m sure that the Tweets on the @BarackObama account are endorsed by his administration and are compatible with Twitter’s rules.
https://support.twitter.com/articles/18311-the-twitter-rules
Thanks for a measured yet non-evasive response, Andy.
I completely agree with you that opinions expressed in the scientific literature can safely (and ethically) be assumed to be those of the scientists themselves, until/unless we have some evidence to the contrary.
(Mind you, in the specific case of climate change, we *do* know a number of cases of scientists revealing in the non-scientific press that they are afraid to voice the full horrors of their expectations regarding impacts in the scientific literature, because what they allege to be a relentless campaign of death threats by deniers has made them even more ‘reticent’ than ‘scientific reticence’ usually demands. So even if we do not associate ‘self-censorship’ with science in general, climate scientists in particular have claimed to practice it.)
On the other hand, even if scientific papers reflect their authors’ views with perfect candor, the the views expressed in x% of sampled scientific papers can NOT be assumed to be the views of x% of scientists unless the papers were sampled randomly by author from the set of all scientists, or to put it another way, unless every scientist had the same probability of being the author of a sampled paper.
Yet we know this kind of “one-scientist-one-vote” condition can NOT be assumed to hold in the present topic.
For instance, if we only count papers about a particular phenomenon, e.g. climate change—as the authors of consensus analyses invariably and necessarily do—then we are more likely to sample a paper written by a scientist who not only views the phenomenon as real but important enough to write papers about. Common sense suffices to tell us that any scientist who did not “believe in” climate change (assuming such disbelief were scientifically possible) is very unlikely to have written as many papers in the “solutions [to climate change]” category as a scientist who DID “believe in” not only its existence but its seriousness as a problem.
In fact, a scientist who did not “believe in” both the existence of AND the serious danger posed by the well-established phenomenon of climate change is far less likely to even be working in the field of climate science. Like all academic professions, climate science is self-selecting.
While I suspect you may be right about this:
“I’m sure that the Tweets on the @BarackObama account are endorsed by his administration”
… I worry that once we start admitting the false “…and dangerous” meme was propagated with White House approval, we risk legitimising ideations about a conspiracy of “alarmism” that goes all the way up to the Oval Office.
Finally, I think you are to be applauded for being one of the rare commentators and bloggers who actually grasps that a ‘consensus’ is exactly what every single dictionary defines it to be: a majority (or nearly unanimous) opinion; and that a ‘scientific consensus’ is simply the majority (or nearly-unanimous) opinion of relevant scientists. I assume you grasp this semantic fact, seeing as your first paragraph in reply to me uses phrases like “the broad opinions of climate scientists,” “the specific opinions that they express in their publications,” “expressing a different opinion in the literature” and “the two categories of opinion.”
However, opinion is not evidence, and cannot ever be passed off as a form of evidence, in science.
This is an axiom of science. Half of the non-scientific public may be scientifically-illterate enough to mistake scientific opinion for scientific evidence, but it does not follow that it is morally or scientifically acceptable for those of us who ought to know better to exploit their misunderstanding by publicly citing [what we know is] non-evidence in an effort to convince them of [what we think is] the truth, deliberately elevating non-evidence to the role of pseudo-evidence.
Consensus is opinion, not evidence, and therefore does NOT—contrary to the title of your post—matter. If we had taken 5 minutes in the last 25 years to explain to the masses this fundamental aspect of *how science works*, they wouldn’t be deluded by slick “science” communicators into falling for the idea that scientific opinion is even worthy of their attention.
Evidence is the ONLY thing a real science communicator ever communicates about.
Because evidence is the only thing that “matters.”
By the way Andy, Twitter doesn’t have (as far as I know) a different set of rules for “celebrities.”
Unless I’ve missed something, all the following words (from https://twitter.com/tos) were addressed not only to Joe Blow and Andy Skuce but to Barack Obama:
“You are responsible for safeguarding the password that you use to access the Services and for any activities or actions under your password….
You are responsible for your use of the Services, for any Content you provide, and for any consequences thereof, including the use of your Content by other users and our third party partners…
All Content, whether publicly posted or privately transmitted, is the sole responsibility of the person who originated such Content…”
They even summarised it in non-legalese, just for the benefit of people whose Harvard Law degree didn’t cover how contracts work:
“What you say on Twitter may be viewed all around the world instantly. You are what you Tweet!“
Brad
I would stress that “evidence” is not always clear cut. There are always uncertainties, noise, the possibility of mistakes made in observations and calibrations, and the possibility of bias in what evidence was acquired and what was not. Expert judgement is therefore indispensable in the interpretation of evidence and its significance. Opinions of experts matter and when those opinions are rooted—as they are for most scientists—in considered interpretation of evidence, then the consensus of those opinions carries some weight.
Having said that, our paper was descriptive, doing our best to classify what scientists wrote in the literature. The results are not and cannot be prescriptive, telling anyone, scientists or laymen, what they should believe. In practice, though, everyone relies on rules-of-thumb in forming their own opinions and one such rule is following what most experts say in a subject where you don’t have the time or expertise to read examine all of the evidence for yourself. Of course, that’s not a certain path to knowing the truth.
As I said, I wouldn’t expect that Obama would deny responsibility for the Tweets his staff make on his account. I’m not sure what point you are trying to make here. I do not claim that Twitter has different rules for celebrities, just that they are surely aware that some high-profile people don’t type all their own Tweets.
By the way, for full disclosure, I don’t follow @BarackObama but I do follow @TheTweetOfGod and I have no doubt that He composes all of His own Tweets, being omnipotent and all.
Thanks for another thoughtful reply Andy.
You say,
“The results are not and cannot be prescriptive, telling anyone, scientists or laymen, what they should believe.”
‘A consensus exists’ is certainly descriptive.
‘Consensus matters’ is prescriptive, however. It tells people what they’ve been told repeatedly throughout the climate conversation, although no scientist had ever said anything like it prior to that: ‘you should take into account the high percentage of scientists who think such-and-such is true.’
You say,
‘ In practice, though, everyone relies on rules-of-thumb in forming their own opinions and one such rule is following what most experts say in a subject where you don’t have the time or expertise to read examine all of the evidence for yourself. ‘
But here’s the odd thing: outside the climate change issue, I’ve never in my LIFE believed a scientific proposition on the grounds that it was the majority opinion of scientists. Nor, I suspect, have you. And this is easily proven by the fact that we’ve never KNOWN what percentage of scientists believed or disbelieved any scientific proposition. Nobody before Oreskes ever measured, or investigated, a scientific consensus about anything.
(Or if they did they managed to confine their discussions to the sociology journals, where such topics belonged.)
How ever did we manage without it? How did modern science last 300 years without ever resorting to opinion polls?
And what changed?
I have often listened to my doctor when he told me what the consensus among medical specialists was for a certain course of treatment. I’m glad also that engineers, for example, meet and agree upon safety standards and construction codes. I don’t feel compelled to follow an expert consensus, but it’s usually the safer bet.
The reason that people like Oreskes and Cook undertook consensus studies was because climate contrarians often claimed that no such consensus existed or was crumbling. Anyone who had attended a large gathering of climate scientists knew that this was not true. Yet the public still fails to appreciate this, as the public opinion survey cited in the original post shows.
Most scientists already know what the consensus position is within their speciality. They don’t need a survey or a poll, they make their own assessment of it every time they read the literature or meet with their colleagues. When they write a textbook or lecture students, they generally outline what the consensus position is.
Some climate contrarians seem to manage to hold in one head the contradictory ideas that consensus doesn’t matter, while making rather extraordinary efforts to debunk existing consensus studies.
Our paper has now been downloaded more that 250,000 times. Clearly, many people think the subject matters, whether they agree with us or not.
Andy,
“I have often listened to my doctor when he told me what the consensus among medical specialists was for a certain course of treatment.”
1. doctors aren’t scientists
2. even though they’re not scientists, they’re trained and legally and ethically required to use EBM, Evidence Based Medicine, when forming treatment recommendations
3. in EBM, the majority opinion among specialists is treated as meaningless, non-evidentiary noise (as it should be)
4. I therefore question your recall of what your doctors have advised you. I’d be very surprised if they knew, cared or told you what 9 out of 10 doctors thought.
5. I suspect what they actually told you was what the EVIDENCE showed, which in practice typically comes down to: what did the latest Cochrane systematic review find when it meta-analysed the evidence for and against a given medical intervention?
“I’m glad also that engineers, for example, meet and agree upon safety standards and construction codes.”
Agreed, but they’re not scientists and those aren’t scientific questions.
“I don’t feel compelled to follow an expert consensus, but it’s usually the safer bet.”
Not in science though; not as far as anyone can tell. We’ve only tried it once, on climate change, but so far I wouldn’t say it’s been a resounding triumph, would you?
Will consensus one day be a topic of concern in any field of science (other than climate science)?
Who knows. Maybe scientific epistemology has had it all wrong for the last 300 years and we really should treat human opinion as having some information value in science. But so far the world has seen no signs that such a shift would be anything other than retrograde, suicidal folly on the part of the scientific world—which is presumably why ALL the non-climate sciences are perfectly content to keep honoring the centuries-old ban on using consensus as evidence. How’s it working for climate science, do you think?
“The reason that people like Oreskes andCook undertook consensus studies was because climate contrarians often claimed that no such consensus exists or was crumbling.”
THAT’S the reason? (Do you actually believe this, Andy?)
Wait—so they WEREN’T interested in measuring the consensus until they heard (unspecified) contrarians denying there was one? And it was this anonymous gossip, denying an otherwise-uninteresting fact, that compelled them to write 5 or 6 papers repeatedly affirming that otherwise-uninteresting fact?
Really?!
Call me skeptical but that excuse rings a bit hollow, especially after the 3rd or 4th consensus study.
Who exactly ARE these contrarians who were denying the particular consensus Oreskes studied, and which contrarians, exactly, were STILL denying the consensus in question in 2013, compelling Cook to re-debunk said denial?
If I didn’t know better, Andy, I might assume these supposed “denials” were just a convenient meme used to justify a cottage industry of otherwise indefensibly pointless papers.
If someone is asserting within the literature that “no such consensus exists”, then sure, what the heck, write a formal response addressing the actual consensus they’re denying in the actual words that they’re using.
But writing paper after paper just to respond to anecdotal, non-literary claims that never even seem to rise to the level of quotations?
That’s bordering on pathologically insecure.
It’s abnormal academic practice at the very least.
“Some climate contrarians seem to manage to hold in one head the contradictory ideas that consensus doesn’t matter, while making rather extraordinary efforts to debunk existing consensus studies.”
But where’s the contradiction, Andy?
The irrelevance of consensus doesn’t make the quality of studies irrelevant. Critics have every right to maintain—simultaneously—that a researcher is wasting energy studying something meaningless AND that he’s doing it wrong. I’m not talking about anyone in particular here, but as a general proposition there’s no incoherence, no logical problem here as far as I can tell.
Besides, as you should know by now, “contrarians” are the kind of people who are perfectly liable to get p___ed off about shoddy methodology in academic studies whose outcome they couldn’t care less about.
Nine times out of ten “contrarians” are just standing up for methodological standards.
Since I’m a “contrarian” myself, I may as well come straight out and say it: we seem to be the last people on earth who grasp the concept that science is a METHOD.
For example, our entire beef with MBH98 was methodological. We just rolled our eyes in pity, disgust and boredom every time some scientifically illiterate apologist said, “but it was confirmed by subsequent studies!”
Are we ALONE in understanding epistemology? The concept of *science*? The need for rules?
“Our paper has now been downloaded more that 250,000 times. Clearly, many people think the subject matters, whether they agree with us or not.”
But YOU said it matters, Andy. You can’t justify that by saying, “well, 250,000 people on the Internet agree.”
I suspect the reason MOST of them downloaded it is that, after a generation of climate miseducation and dumbing-down of science, they’re now brainwashed into thinking consensus really does matter—and they honestly believe they can find out the truth about the climate by reading a survey of opinion.
God help America.
OK, Brad. I don’t think this discussion going anywhere useful now.
Thanks for your earlier comments.
Pingback: Richard Tol's 97% Scientific Consensus Gremlins - Real Skeptic