http://climatesci.org/2008/02/22/is-there-agreement-amongst-climate-scientists-on-the-ipcc-ar4-wg1/
Roger Pielke Sr has written up our experience of trying to get a poll published.
Thus should have been my job but, 1; Roger has a much larger audience than I, and b; he is the senior member of the team, even if I was the researcher, and c; I suddenly got busy and left him and James hanging on a limb. For this, I owe them my sincere and public apologies; I hate failing to keep a promise.
As Roger’s comments link is closed, if anyone wishes to comment on the article, or on our experience of the publications concerned, please feel free. Just don’t expect me to publish them if you are a total wingnut, or abusive.
If you are a scientist involved in a climate-related discipline, or a meteorologist, you may also wish to express your own opinion, as I only sent out 1807 original enquiries, and you may not have been on the list.
If you are just another ordinary Joe (like most of us are), and have an opinion, feel free…
I won’t comment on any of this yet; let’s see what comes in and I’ll try to respond to that.
Be free.
Old man in a cave 
25 comments
Comments feed for this article
February 24, 2008 at 11:56 pm
William Connolley
Some comments: “there is not a universal agreement among climate scientists about climate science as represented in the IPCC’s WG1” is a strawman really. Can you imagine a situation in which it would be otherwise?
“It is important to recognize that we are not presenting the results as representing anything other than the views of those who responded…” this is weaselly. Its a get-out-of-jail-free card anytime anyone challenges your method, but its clearly not true: you *are* saying this survey means something.
There is a curious asymmetry between 4 and 6. 4 is about attribution. 6 is about consequences. I could, for example, assert that the IPCC should be 100% certain about attribution, and yet be convinced that no action is needed. Which should I tick?
These are quibbles, though. I can’t see why this wasn’t published. But what did you expect with James “Pariah” Annan as a co-author?
February 25, 2008 at 8:28 am
John Mashey
Interesting work, and within the always-tricky design of polls, especially with:
a) Multiple statements / choice
b) And allowing people to mark several choices
the results don’t seem too surprising to me: something that looks grossly like a normal distribution [although with a *nonparametric* scale, that doesn’t mean much of course, so it was wise not to get too deep into “expected that a normal distribution….”].
However…. there are problems with the presentation of the data, which make it harder to understand what’s going on. Tufte would be down on you, and I’m sorry to say I must be also. There’s nothing that can’t be fixed, but to be honest, if I’d been a referee, I would have had to say “Reject” or maybe “Accept, with major changes”.
1) It says there were 140 responses. But later, “one respondent suggested … two could find….”, etc.
At this point, I have no idea of the actual number of responses graphed. I don’t know whether the odd cases were ignored, or subtracted out, or whatever, because you didn’t give me the raw data.
2) “The exact response rates are given in Figure 1.”
Well actually, they aren’t. Those are response percentages, which are numbers computed from a base that might be 140, or might be something else, but in any case, add up to 99%, not 100%.
3) Each red number gives the sum of percentages of the bar underneath plus the bars on both sides, i.e., the percentage that included the central bar alone, or with one of the others. If this is explained, I missed it, and it’s confusing, because they are a kind of percentage, but as a group add up to more than 100%.
4) Then, the discussion sometimes slices the data in different ways, with numbers that don’t correspond directly to numbers on the chart.
“in total, 18% responded that … probably overstates.” The 18 corresponds to 3+10+5, and the 18 shown on the chart, that is, it includes 3.5, 4, and 4.5.
Then:
“A further 17% expressed … underestimates or seriously underestimates…”
There’s no 17% on the chart: from the right, it must be: 9+1+7, i.e., 7, 6.5, and 6, but it omits the 13 who are at 5.5. Maybe there’s a plausible reason for that, but it needs to be articulated. I.e., you counted the 4.5 piece on the one side, but you didn’t count the 5.5 piece on the other side.
“The remaining 65%…” is weird, because 5% of that has already been counted once.
” The largest group of respondents (45-50%)” …
then 15-20% is given for both understate and overstate, and I just don’t understand where those come from.
All of a sudden, we have % ranges, after using specific single numbers. You could say:
47% simply concur.
30% have some to substantial concern that IPCC understates.
18% have concern that the IPCC overstates [or think CO2 overestimated]
4% think it’s essentially natural.
99% total
OR you could say:
65% include concurrence
17% think understate
13% think overstate
4% think natural
99% total
OR, allowing overlap, you could say:
65% include concurrence
30% include understate
18% include overstate
4% natural
117%, i.e., 17% overlap
[And whichever you want, show the chosen groups with bars, or shadings, or something.]
5) BUT, why does this have to be *so* hard.
a) Give a simple table of the 140 responses (or 140+ones that didn’t get counted), i.e., the raw data of counts.
X no answer, or all 4. etc.
140 responses, shown in histogram
b) Do a straightforward histogram of the counts. If you want percentages, especially of groups, add a group together and THEN show percentage, avoiding rounding problems. if you want to have a histogram with both count & percentage scales, that’s useful.
c) All these words with different slices and dices and overlaps that don’t directly correspond to obvious things on the chart, just get confusing.
c) If you want to make overlapping groupings, put them in and label them, i.e., like horizontal bars. That might get messy.
6) I’d be tempted to put table 1 before Figure 1, as then I can just see what people were being asked. Also, describe the interim responses there. Then show me the Figure, after I can know what it means.
February 25, 2008 at 1:54 pm
Cam
Its an interesting piece. Just a raised eyebrow at this statement from R.P. Senior’s introduction to it:
“From this experience, it is clear that the AGU EOS and Nature Precedings Editors are using their positions to suppress evidence that there is more diversity of views on climate, and the human role in altering climate, than is represented in the narrowly focused 2007 IPCC report.”
Having been on the receiving end of journal rejections, I sympathise, but isn’t it a stretch to claim a deliberate suppression of evidence in this case?
February 25, 2008 at 7:58 pm
fergusbrown
Thanks to all of you for your engagement with this. One at a time:
William: thanks for the input. I know you have some quibbles with elements of the poll, which is fine. Some of your feedback to the wiki debate was very interesting, since it dealt with many of the problems all pollsters seem to face.
That early sentence you criticise as a strawman went in to get rid of the rather more provocative ‘the science is more or less settled’, which we agreed was too likely to provoke silly feedback. As always, finding the exact combination of words to express something meaningful and yet avoid exaggeration or hyperbole is the difficulty; we did the best we could with the time and resources available. The underlying point is relevant, though.
The ‘get out of jail card’ is really a recognition that, in terms of pure opinion polling methodology, we didn’t have enough to justify a ‘strong’ claim. We expected an editor to baulk at claims which were unjustifiable given the available evidence, so we went with the ‘softer’ option. My personal feeling is that the survey does strongly suggest that the range and extent of disagreement with the IPCC, as well as the consistently strong support for it, are both potentially interesting and perhaps surprising, therefore worthy of further research. For the record, I went into the research hoping to get a strongly supportive result; Roger probably felt the opposite; James expressed surprise at the number of people who appear not to be comfortable with the WG1.
I didn’t spot the 4-6 asymmetry before; thanks for pointing it out. Like all of these things, its easy to see a better phrasing or way of doing things after the event…
As for ‘Jonah’ Annan; he brought some critically important rigour into the process and his unbiased eye was an important part of the mix. next time I’ll ask Hansen…
February 25, 2008 at 8:04 pm
fergusbrown
John:
Can’t really argue with any of that. Just think word count; we had to seriously cut down the explanations and commentaries to fit into a suitable size for submission, so lots of detail was missed out. Perhaps it’s better to consider this as a snapshot, or preliminary, as we suggest in the paper. My first try was 13 pages long before I realised that no-one would publish it in that form.
February 25, 2008 at 8:12 pm
fergusbrown
Cam:
Roger has consistently expressed this view in his blog, and it is hard to argue that it lacks evidence other than this one example. It is, however, his opinion and interpretation of the editor’s actions. I
t just seemed very odd that an erstwhile two-week process took four months, then we were told it ‘wasn’t suitable’; Fred offered no explanation why it took EOS four months to tell us something they should have been able to decide on in about two hours. Likewise, Nature precedings was recommended to us by a member of the Nature editorial staff, so we not unreasonably imagined that, since one of their own staff suggested that it was appropriate, we would be okay. Do you know how few pieces are actually rejected by NP? At least they only took about ten minutes to reach their decision.
If it were simply a matter of Roger claiming this, one might be more willing to be sceptical, but word-of-mouth reaction has generally been one of surprise, not dissimilar to James’; it seems quite a few scientists find the suggested level of disagreement a news item. What does this imply?
February 25, 2008 at 9:08 pm
Climate Science: Roger Pielke Sr. Research Group News » Follow Up By Fergus Brown To “Is There Agreement Amongst Climate Scientists on the IPCC AR4 WG1?”
[…] Fergus Brown has weblogged on the article that was presented on Climate Science on February 22, 2008 entitled Is There Agreement Amongst Climate Scientists on the IPCC AR4 WG1?. His weblog invites constructive comments, and Climate Science encourages its readers to respond there. It will be very valuable to build on this important contribution that was led by Fergus. […]
February 25, 2008 at 9:13 pm
Roger A. Pielke Sr.
Fergus – Thank you again for your leadership on this! Regardless of whether readers agree with me or not on the role of the AGU and Nature in suppressing this preliminary poll, the availability of our article should encourage open minded researchers to build on and improve our survey, as well as to adopt the most rigorous scientific polling approach in follow on studies.
February 26, 2008 at 12:55 am
Steve Bloom
Fergus, as I pointed out here when I first saw it, the survey was poorly designed in some pretty obvious ways. You could have gotten help on that, e.g. from RP Jr. or someone else in his shop. Assistance from a David Jepson is mentioned, but based on the results it’s unclear that he has the appropriate expertise.
There are various problems with the questions, including the one William mentioned, but the fundamental flaw in the poll is that the lack of sampling control ensured that with such a small response (less than 10% of the scientists who were solicited) you would be unable to support *any* conclusions, let alone one that a “significant” number of the scientists contacted think the WG1 report is too strong or too weak. Didn’t it occur to you that people with critical views would be more likely to respond, and that you had to figure out a way to control for that for the survey to have any value?
I agree with William’s point about the asymmetry of the questions, which IIRC is a point I raised before.
BTW, were I in charge of Nature Proceedings it would have taken me even less than ten minutes to establish that this survey was not “quality” as required by their guidelines.
February 26, 2008 at 12:56 am
John Mashey
Fergus: as Mark Twain is reputed to have written “I’m sorry this letter is so long, I didn’t have time to make it shorter.”
The goal of my comments was to be able to reorganize it so it made its points in a shorter, clearer form. It was just *harder to read* than it needed to be, and used more words, but it’s fixable. I’ve been through this many a time, where I’ve written something, put it aside, and come back to it later, and was able to cut it down and say it better. I don’t know enough about polling design to have any useful comments there.
I have no opinion about “suppression”, but like I said, if I’d been an editor [which I do every once in a while, like right now], *I* wouldn’t have accepted it as is, because really, the description of the results was simply more confusing than it needed to be.
February 26, 2008 at 2:19 am
Aaron Lewis
All the poll says is that, some “scientists” have not done their homework.
I have hated science by vote since some cheerleaders in a high school science class tried to do physics by vote. Perhaps the physics proposed in WG1 is wrong, in which case a small group of very bright minds should be set to building better models and making better measurements. This is happening even now. However, the science will move forward by offering better models and data, not by taking a poll on the correctness of the current models and data. Current models and data are not perfect, but only those that offer better, have legitimacy to criticize the current models and data. (My view is that the WG1 significantly understates the impacts and effects of global warming. Would YOU like to be a Polar
Moreover, I do not care how many dislike a theory, I care about why they dislike a theory. Have they really found a problem with the science, or does it simply conflict with something that their mother told them years ago?
All of which is not to say that the poll is not legitimate science. It is – in the field of media and communications, history of science, or models of cognition. The methodology should have been tailored to these fields and submitted to a journal in one of these fields. This is exactly the kind of study that a young Karl Rove would be interested in. It is the kind of thing that the tobacco companies needed to know as they spread discord in the popular media over what the science on tobacco smoke really was. It the kind of thing that the API needs to know, in order to convince the public that “many” scientists do not accept the theory of global warming.
February 26, 2008 at 6:44 pm
fergusbrown
Steve: Hi again, and thanks for coming back to this.
There are always things wrong with polls; the important question here is, have we tried in a fair and balanced way to solicit the honest personal opinions of a range of people engaged in disciplines related to climate science? And to what extent do the replies we received accurately reflect the range of opinions in this community?
I simply did not have the resources available to generate a control sample and then follow it through, using the recommended methodology for statistically valid polling results. There is no such thing as a database of climate scientists, unless you consider the 3500+ names I collected to be such a thing. The next best thing available was to collect a large number of names of people, as described in the supplement, check their credentials and send them an enquiry. In the absence of a known ‘community’, best practice suggests a controlled number of known members as a sample. In this sense, had we received 200, or 400 responses, we could have justified attributing statistical significance to the results, even if the margin of error was going to be quite high.
We fell short, so we have presented the survey as a ‘work in progress’. in the hopes of stimulating further response, or perhaps more research on the matter by others.
As far as response bias is concerned, you might take a look at the curve of responses; it does strongly resemble a standard distribution; does this not suggest that the bias in the extremes was counterbalanced by an inclination by supporters of the IPCC to respond supportively? You might also note that there is very little evidence of ‘hard denialism’, so if the responses were ‘biased’, this could tell us something important in itself about what the true opinion of scientists might be, and it doesn’t favour the extreme elements of the blogosphere.
I’m sorry that you don’t think the work is up to standard; you are entitled to that opinion. I would ask, though, that you look at the results and what they might suggest, rather than focus on the imperfections.
February 26, 2008 at 6:50 pm
fergusbrown
John: my understanding is that editors or referees are expected to offer relevant comment on rejection, rather than simply reject. Perhaps I am deluded. What was most disappointing was the apparent impossibility of getting a response, even a polite and short one, from anyone. If a business behaved in this way to me, I would not use them or buy their product, because I would presume that they were unprofessional.
Several people, here and previously, have offered sensible and constructive criticism of the paper, based on actually having read it. Is it too much to expect a scientific journal or portal to do the same?
Best,
F
February 26, 2008 at 6:59 pm
fergusbrown
Aaron: thanks for your comments. i was fine with what you were saying until the last paragraph. You seem to have got it into your head that I was hoping to stir up trouble, by questioning the validity of the IPCC. You have visited often enough to know that this is not me: I am a supporter of the IPCC, broadly, and a strong advocate of action on climate now.
One interesting side point; the comments that have been made about this poll by people (not you, necessarily) who already hold strong opinions at the ‘ends’ of the spectrum, are equally critical irrespective of their ‘colour’; both ‘denialists’ and ‘environuts’ seem to dislike the result, because in both cases, it can be read to undermine their desired position. I have also read comments from both ‘ends’ which cite the survey as evidence in support of their claims.
What this tells me is that people’s response to the results are determined much more by their predisposition/prejudice, and much less by the content.
As I pointed out before; this was an honest attempt to solicit the honest personal opinions of a range of people. As such, the results are what they are.
February 26, 2008 at 7:01 pm
fergusbrown
Comment received via email (with permission):
Dear folks,
I am not a candidate to take part in your survey. I found it well worded, and the results interesting. While the initial response suggesets option 5 is chosen with much greater frequency than the other choices, it was also informative to see that about a third of the respondents chose other options, and that these were evenly split between the options 1-4, and 6-7.
The results were reported clearly, and the statistical lack of rigor also reported well, and upfront.
I would be interested in a similar set of questions posed to the same group concerning the uncertainties in projecting future climates. This survey seemed to emphasize a perception of what is occuring at the present, and causation there of.
Sincerely,
Richard “Heatwave” Berler, CBM
Chief Meteorologist
KGNS TV
Laredo, TX
February 26, 2008 at 11:03 pm
Atmoz
Fergus,
I’m unsure if the version posted by Dr. Pielke was the version submitted to EOS, but I counted ~2000 word equivalent [~1600 words + 1 figure (400 word equivalent)]. The maximum length stated on the EOS website for a forum piece is 1500 words. Could this be the reason for rejection?
I also agree with Steve Bloom when he writes, “Didn’t it occur to you that people with critical views would be more likely to respond, and that you had to figure out a way to control for that for the survey to have any value?” You reply with “As far as response bias is concerned, you might take a look at the curve of responses; it does strongly resemble a standard distribution; does this not suggest that the bias in the extremes was counterbalanced by an inclination by supporters of the IPCC to respond supportively?”
The presence of a almost-Gaussian shape does not suggest that the poll was uninfluenced by bias for the extremes. The bias that Steve is talking about would result in a widening of the distribution; it would not change the shape. This bias of the extremes would cause the poll to show more disagreement on both ends than in actuality.
February 26, 2008 at 11:57 pm
John Mashey
Fergus: yes, I usually expect some feedback, even if it’s “Not interesting for this audience”, and certainly, when I referee, I try to give constructive feedback if there’s any hope. [Sometimes there is no hope.]
One never knows with referees, if it gets that far. Long ago and far away, Brian Kernighan & I wrote a paper for a conference, and got 4 referees’ reports:
1 & 2: accept, great!
3: reject, have heard it before, ho-hum.
4: reject, irrelevant to this conference
Hence: reject.
It turned out that exactly our take on this topic was discussed seriously at the conference, and an editor later asked Brian for a paper, and it got published there, and then later yet, IEEE Computer asked us to update it for them.
One more time: one has to be very careful about talking about Gaussian-ness of this sort of non-parametric data. Yes, it looks ~Gaussian, but the scale is *not* a measured scale, but a set of (more-or-less) ordered choices of multiple statements describing positions. However, if a survey asked the same people something that generated a parametric answer, like “How serious do you think AGW is, from 0 (no problem) to 5 (some problem) to 10 (really serious)?” you might well find that this more like a lognormal or at least heavily skewed, or might be multimodal.
The paper *didn’t* compute mean, std.dev, skew and kurtosis … thank goodness, because that would have been a clear REJECT.
February 28, 2008 at 12:15 am
William Connolley
You’ve made Nature, thouh: http://blogs.nature.com/climatefeedback/2008/02/climate_consensus_is_opinion_e.html
February 28, 2008 at 3:00 am
Michael Tobis
I once wrote a paper that got three reviews, one accept with revisions, one reject on the grounds that it’s obvious and one reject on the grounds that it’s obviously wrong.
The review process is flawed, but that doesn’t necessarily indicate which flaw got you. Perhaps it really isn’t a suitable piece for EOS. Maybe they didn’t think a scientific newspaper is the right place for a poll of any sort.
I find the results somewhat interesting but unsurprising, and I find Pielke Sr.’s interpretation of them peculiar.
February 28, 2008 at 8:45 pm
fergusbrown
Another comment from a contributor via email. No names, by request:
Dear Fergus
Thank you for the opportunity to contribute to the “follow on polling to the preliminary polling” for your recent paper. As you noted in your paper, it is hard to find a collection of statements with which I fully agree. I have chosen response 3, as it is closest to what I think.
However, I would rewrite it as follows:
3. There are changes in the atmosphere, including added CO2 from human activities, but we do not know what net effect these changes will have on climate. It is possible that the net effect will be minor, and/or benign. Currently observed temperature variations may be within natural limits, and could be due to forces other than changes in atmospheric components (either natural and/or anthropogenic). Emphasis on CO2 diverts attention away from other, important research on climate variability and change.
Note: I am uncomfortable ascribing political motives to those with whom I disagree (“The ‘scares’ are exaggerations with a political motive”). I’m sure many supporters of the IPCC view are just as sincere in their views as I am in mine. And of course, political influences are likely to motivate some supporters of both sides of the AGW debate.
RESPONSE: 3
COUNTRY: Australia
SCIENTIFIC DISCIPLINE: Agricultural Science
ROLE IN 2007 IPCC REPORT: None.
There were additional interesting suggestions about the IPCC, which I may post another time.
March 3, 2008 at 1:09 pm
Nexus 6
Hi Fergus,
I’m putting together a (rather long) post on your poll which I’ll probably get up in the next day or two. From my limited knowledge, there are plenty of ‘easy’ improvements to make (like using a larger sample to reduce the margin of error). Even reporting the margin of error with 95% confidence intervals would be a start. Also, in my view the ‘answers’ are far too wordy and ambiguous, and likely to introduce further error on top of the MOE. Add interviewer error and non-response error and you are getting a serious blowout, which would make it very difficult for a reputable journal to publish your article – there’s a decent chance it’s way, way off the mark.
I would have rejected it in its current form, but having no feedback is a pain in the arse.
I’ll report back when my post is up.
March 11, 2008 at 5:49 am
Peter Houlihan
From your rejected paper”
“It is important to recognize that we are not presenting the results as representing anything other than the views of those who responded as we have no way to assess the relationship of the responders with the total relevant population.”
Maybe that’s why it was rejected? Perhaps the journals wanted to maintain their standards and it had nothing to do with censorship. Do a legitimate poll next time.
January 24, 2009 at 10:02 pm
Gaz
Fergus,
Your unnamed Aussie Ag Science correspondent indirectly suggests a wider issue with the poll, beyond the methodological issues raised here.
The poll would have had a valuable additional dimension had the reasons for disagremeent with the IPCC orthodoxy been drawn out.
For example, your correspondent says:
“Currently observed temperature variations may be within natural limits, and could be due to forces other than changes in atmospheric components (either natural and/or anthropogenic).”
That some currently active climate scientists diverge from the norm is well-known but I, along with many more average Joes, would have been intrigued to find out the basis for their views.
I don’t think it would be too hard to add an “If so, what is the main reason..” type of follow-up to the question. You would be getting down to some pretty small samples by then, of course, but even an impression would be of some interest.
August 3, 2009 at 6:31 pm
Extenze
Hi, I can’t understand how to add your site in my rss reader. Can you Help me, please 🙂
September 9, 2009 at 9:18 pm
Does Extenze Work
I like one reader above would find the discussion even more intriguing if there were more data on the why’s of particular choices. Polling for beliefs doesn’t really reflect on accuracy of propositions though, right?