by Judith Curry
There is an unfortunate knowledge monopoly in climate science and policy – the IPCC and UNFCCC. As a result there is insufficient intellectual and political diversity in assessments about climate change. To break this monopoly, we need identify new frameworks for encouraging, publishing and publicizing independent and interdisciplinary ideas and assessments.
The publication of Jim Hansen’s new paper, discussed previously [here], has raised a host of issues that are apart from the actual content of his paper. Some of these are discussed in Revkin’s two posts:
- Whiplash warning: when climate science is publicized before peer review and publication
- A rocky first review for a climate paper warning of a stormy coastal crisis
Here is my take on why we need to rethink how we deal with consequential policy relevant science and its publication and publicity.
Diversity versus knowledge monopolies
What do I mean by a ‘knowledge monopoly’? This term was coined by Richard Tol (in the context of climate, anyways), see this previous post IPCC as a knowledge monopoly. Andy Revkin’s recent posts on the controversy surrounding Jim Hansen’s paper included this quote from Richard Alley that succinctly describes the climate knowledge monopoly:
For those focused primarily on the broader implications of the science, the logical path is to start with the authoritative assessments from the US National Academy of Sciences, the Royal Society, the I.P.C.C., etc. Science by its very nature celebrates attempts to overthrow established results, which unavoidably makes our work look “noisy” and confusing to a non-specialist, obscuring for those outside a field what is actually well-founded inside the field. To overcome this difficulty, governments and the broader society have established assessment mechanisms in which the full range of scientists, volunteering in the public eye for the public good, provide up-to-date information on what is solid, what speculative, and what silly. The assessment results are inefficient at generating headlines, he-said/she-saids, and “clicks” on web pages, but the assessment results are far better than the latest press releases at generating reliable, useful, policy-relevant understanding.
(Full disclosure: I have been [I.P.C.C.] or am [N.A.S., R.S.] a member of the groups I’m praising here.)
Well, I find that these government sanctioned assessments don’t adequately account for the broad range of relevant publications and don’t include minority perspectives. This concern has been raised multiple times by me, by John Christy in his push for a team B or red team, and by the existence of the NIPCC report (not to mention many others).
These negotiated government sanctioned assessments don’t adequately account for the very substantial disagreement about climate change that arises from:
- Insufficient observational evidence
- Disagreement about the value of different classes of evidence (e.g. models)
- Disagreement about the appropriate logical framework for linking and assessing the evidence
- Assessments of areas of ambiguity and ignorance
- Belief polarization as a result of politicization of the science
All this leaves multiple ways to interpret and reason about the available evidence.
A previous post Importance of intellectual and political diversity highlighted the problems with lack of diversity in climate science and assessments.
John Christy’s ‘red team’ idea is a good one, but then who sanctions/selects the red team? A previous post Institutionalizing dissent discusses these issues.
This post is targeted at the individual scientist, or science team, that wants to publish dissenting assessments of climate science or produce meta-analyses of aspects of climate science or major interdisciplinary research, outside of the establishment government sanctioned organizations (e.g. IPCC, WCRP, etc). In other words, non-institutionalized dissent. Dissenting scientists can write a book, or take to the blogosphere, or hook up with an advocacy group/think tank and publish reports. You can judge for yourself to what extent writing books or blogging or think tank reports influence the scientific and public debate on climate change. Academic research scientists don’t get much credit for publishing in such venues and – the mainstream media is focused on ‘peer reviewed’. But what are the options for a scientist that wants to publish such analyses in the peer reviewed literature? There aren’t many.
Where to publish a lengthy interdisciplinary, comprehensive analysis?
Nearly all journals have page limits for their articles – for Science and Nature, the page limits are quite stringent. Then general work around is to carve things up into multiple papers – this strategy works well for scientists in that their merit is often assessed by number of papers and citations (which is inflated by writing a large number of shorter papers). But these short papers don’t really meet the needs of writing an assessment or meta-analysis or comprehensive interdisciplinary study, which necessitates a more lengthy paper.
So what is the motivation for a scientist or science team to write a lengthy paper on an assessment or meta-analysis or comprehensive interdisciplinary study or new pathbreaking research? Perhaps because they think that the establishment assessments got it wrong, or the scope of their analysis requires a lengthy paper. Senior scientists have the luxury of not worrying about their paper count, and can focus on more comprehensive publications.
Reviews of Geophysics allows for lengthy papers, but the papers are invited and it is intended for reviews (including synthesis and assessment), not for edge-pushing new research. Dissenters would be be unlikely to be invited to write such a review.
Some online journals are allowing for more lengthy papers, since they are unbound by the limits of paper publication. One example is the open “discussion” journals, including Atmospheric Chemistry and Physics Discussions (ACPD), which is where Jim Hansen submitted his paper.
Policy relevant science and the need for information yesterday
Scientists wishing to publish an important paper on a topic of policy relevance desire a high impact journal with a quick turnaround on publication, and would most likely seek to publish the paper in Science, Nature, or PNAS.
Nature, Science and PNAS have stringent word count limits on articles (say 2000 words for most contributions). These venues obviously don’t work for a lengthy paper such as Hansen’s.
There is a more insidious problem with publishing policy-relevant papers in these journals, particularly Science. You may recall my recent post The beyond-two-degree inferno, I raised a concern about Chief Editor Marcia McNutt’s advocacy editorial in that it would introduce a bias in the selection of papers sent out for review.
Sure enough, the next week I received an email from an economist I had been communicating with about his draft paper, that is highly policy relevant in context of the forthcoming Paris meeting. The paper not only addresses a very important topic and has an important result, but is solidly done and its methodology and results are not likely to be controversial. The paper was submitted to Science. The author received the following response:
“Thank you for submitting your manuscript XXX to Science. Because your manuscript was not given a high priority rating during the initial screening process, we have decided not to proceed to in-depth review. We are therefore notifying you so that you can seek publication elsewhere. We now receive many more interesting papers than we can publish. We therefore send for in-depth review only those papers most likely to be ultimately published in Science. Papers are selected on the basis of discipline, novelty, and general significance, in addition to the usual criteria for publication in specialized journals. Therefore, our decision is not necessarily a reflection of the quality of your research but rather of our stringent space limitations”.
In other words, maverick papers that do not support the policy preferences of the Chief Editor will not be published. Once this paper is eventually published (hopefully sooner rather than later), I will do a blog post on it.
Discussion journals, with instant ‘publication’ of the discussion paper, can provide a venue for getting policy-relevant papers rapidly into the public arena.
How to peer review complex, integrative policy relevant papers?
Lets say an author has managed to overcome the hurdles of identifying a journal that will publish a lengthy assessment or interdisciplinary paper, has a reasonable impact factor, and has a fast turn around for publication. How can this paper most effectively (and fairly) be peer reviewed? Too often, reviewers cheer for their ‘home team’, even if not explicitly a pal review.
Normally, an editor sends a paper out to three reviewers (unless the author is a ‘skeptic’; their papers usually receive the ‘courtesy’ of additional reviews). For a complex interdisciplinary paper such as Hansen’s, three reviews aren’t sufficient – there are too many subtopics associated with discrete expertise.
In the case of policy relevant papers, I have an interesting example to relate. Recall the Webster et al. paper on hurricanes and global warming [link], that created quite a media storm given its publication shortly after Hurricane Katrina. Well, 6 months later we published a follow up paper [link], that strengthened the link between % cat 4-5 and SST [link] . Because of the intense media interest surrounding the first paper and the continued fascination with the topic of hurricanes and global warming, in advance of the embargo journalists sent the paper to over a hundred scientists, statisticians and mathematicians, conducting a far more rigorous peer review than the journal did. The involvement of statisticians and mathematicians was important given the novel analysis technique used in the paper. We saw a version of this for the Hansen et al. paper, with numerous scientists commenting on the paper after a journalist provided them with a copy of the paper and solicited their opinions. The media plays a very important role in the peer review of headline worthy research papers. And also follow on analyses by blogs. For science of high consequence, rapid peer review is crowd sourced by the media and blogs.
But again, the other more insidious issue is reflected in this tweet regarding Hansen’s paper:
When climate peer review is a closed shop, what’s a radical to do ?
Bottom line: on a comprehensive, multidisciplinary, controversial paper, traditional anonymous peer review (say with 3-5 reviewers) isn’t very useful. The spate of dubious papers published in Nature Climate Change, many of which don’t survive their press release before being debunked, is a case in point. Crowd sourced reviews and Discussion papers seem much better to me, although scientists wanting to enforce the knowledge monopoly don’t seem to like this.
Advantages of a discussion journal
A discussion journal such as ACPD allows for lengthy papers, rapid publication, and extensive reviews by peers and the public.
From Revkin’s post: Ken Caldeira is not a fan of the discussion journals that publish both a discussion paper and the final archived/accepted version of the paper:
The continued circulation of the non-peer-reviewed draft can act as a kind of pollution of the scientific literature, as it is often unclear to the uninitiated what it means to be published in an EGU “Discussions” journal. By publishing papers that are not peer reviewed, EGU journals such as ACPD are contributing to the noise of science, when the role of the editorial process should be to help readers find the rare nuggets of important high quality signal amid the abundance of excess noise.
I really disagree with this. We need more diversity of published science, not less. Let the assessment process (with the diversity caveats mentioned above) sort out what is noise and what is not. That ‘noise’ is the potentially important findings that establishment gatekeepers would prefer to see ignored since they don’t align with the prevailing paradigm. Caldeira’s perspective acts to reinforce knowledge monopolies.
It will be very interesting to see how the review process plays out for Hansen’s paper. I have experience with two controversial papers submitted to APCD – one paper on which I was first author (link ) and the other one which I was a reviewer (Makarieva et al, discussed here at CE). Both were very controversial as evidenced by the reviews (although neither was associated with a controversy that was relevant to the media); after about 2 years, both papers were finally accepted for publication in the archival journal APC. APC does not move quickly to make editorial decisions.
Publicizing new ideas outside of the knowledge monopolies
If a scientist has survived the process so far, and has a policy relevant paper that they would like to publicize, the usual process is for the university (or govt lab), funding agency or the journal to issue a press release. But what about retired or independent scientists? And for scientists whose universities won’t issue a press release? E.g., Georgia Tech declined to issue a press release on Lewis and Curry; the paper was publicized on my blog and by the GWPF. In Hansen’s case, presumably NASA or Columbia could have issued the press release. But probably not including Hansen’s most alarming statements and policy prescriptions.
So, is issuing a press release and making a big media push for his discussion paper prior to peer review ‘kosher’ in academic circles? (Recall, Richard Muller also did this with the Berkeley Earth papers). Revkin’s post includes this comment:
The result here, just two days after posting, reinforces the danger in making too much of a brave new narrative before that open process has taken place. That point was reinforced in a comment sent Friday by Bárbara Ferreira, communications manager for the European Geosciences Union, endorsing what others have said about the dangers of publicizing research before it has been peer reviewed: “Our policy at the European Geosciences Union is to not advertise research submitted to our journals before the paper has been accepted and published in its final, peer-reviewed form – which, in this case, wouldn’t be for at least another three months, possibly more.”
But what about policy relevant research, where the need for information is yesterday? Especially when there is a looming policy decision deadline (December in Paris).
The IPCC AR5 was published 2 years ago. Several subfields have undertaken mini assessment updates. Apart from wanting the most up to date information, usefully informing policy makers requires that they have input from multiple perspectives. Revkin makes the following point:
But in the public sphere, with consequential science, the result can be whiplash, at best, and confusion and disengagement at worst.
Whiplash is what you get in controversial and highly uncertain science, if you fail to provide adequate context and address uncertainties. Apart from this, the way to get around the whiplash problem is to get rid of the mindset and decision analytic framework whereby policies are based on a most likely outcome (with an uncertainty range), determined from a negotiated consensus about a highly uncertain topic. My numerous posts on decision making under uncertainty [link] speak to this issue.
And finally there is the issue of Hansen’s use of a professional marketing/lobbying firm to handle the press release and media strategy. I can understand that universities or funding agencies are not the right venue for this kind of a media strategy. Another option is think tanks/advocacy groups (e.g. GWPF, Cato, Environmental Defense, etc.), but they may not want to promote reports that do not originate from their organization, although GWPF seems open to this sort of thing. This issue needs some thought, and hopefully some non partisan organizations with expertise and experience will be identified to support this kind of communication.
The art of integration and assessment
Individual research papers push forward the knowledge frontier; it is for this kind of research that scientists get ‘credit’ and recognition. Synthesis, integration and assessment require different skills than frontier research. Scientists who have some broader perspectives may be invited to write News and Views for Nature (which is not regarded as a peer reviewed publication, so a scientist doesn’t get much ‘credit’).
More comprehensive synthesis and integration efforts are needed in climate science, with different perspectives (e.g. logical frameworks, weighting of evidence, assessments of unknowns). When searching for quotes for my No consensus paper, I stumble on this statement by Galileo, which is very apt:
“In questions of science, the authority of a thousand is not worth the humble reasoning of a single individual.”
The reasoning of an individual about a complex topic is very valuable, in the sense of providing a complete (and hopefully consistent) logic. This is different from, say, the IPCC assessment, which is largely a dump (or consilience) of information, with inconsistencies between working groups and chapters, that is cemented by expert judgment.
Who are the individuals that are inclined to conduct such syntheses and integrations, particularly in context of policy? In climate science, there are increasing numbers of hybrid voices, scientists that are engaging or speaking out about policy (in an advocacy role or not). Revkin’s post has a relevant statement:
Hansen has become a hybrid voice. That’s always complicated. In a Facebook discussion of my Thursday article, Steven A. Leibo, a professor of international history and politics at The Sage Colleges in upstate New York, put it this way:
It strikes me that Jim Hansen is playing a somewhat different role these days, rather like that of Paul Krugman — the fully qualified Ph.D. scholar who does not have a peer reviewed process for every commentary he writes, and of course this study is in so many ways going to go through a fast, crowd-sourced peer review fully in time to be ready as we move closer to Paris.
I have found blogging to be an excellent vehicle for broadening my scope and promoting efforts at synthesis and integration. The art of writing effective Congressional testimony is an example of synthesis and integration. This is a valuable skill that should be nurtured in scientists, especially senior scientists – instead we are implicitly told (in terms of salary increases, etc) that such activities that take away from discovery research aren’t really worthwhile, although lip service is made in support of outreach activities.
JC conclusions
We need more maverick climate scientists that devote time to looking at the big picture in an integrative way. And that is why I applaud Jim Hansen for what he has done, in spite of not finding much in his paper to be very convincing. We need to figure out ways to nurture and reward this kind of scholarship, suitable publication venues, and accepted and effective straggles for publicizing this.
Journalists have been well trained to be dubious of non-peer reviewed papers. However, on a comprehensive, multidisciplinary, controversial paper, traditional anonymous peer review (say with 3-5 reviewers) isn’t very useful. Crowd sourced reviews (coordinated by journalists and bloggers) and Discussion papers seem to me to be a much better way to approach this.
Filed under: Communication, Sociology of science