by Judith Curry
How valid conclusions often lay hidden within research reports, masked by plausible but unjustified conclusions reached in those reports. And how the IPCC institutionalizes such masking errors in climate science.
In the previous post, we discussed the motivated biases of individual climate researchers, stimulated by the paper by Lee Jussim, Joe Duarte and others entitled Interpretations and methods: Towards a more self-correcting social psychology
The Jussim et al. paper provides additional insights that are relevant to the motivated biases in climate change, which become particularly serious and problematic once these biases are institutionalized. Here are additional excerpts from Jussim et al. for the topic I would like to discuss in this post:
<begin quotes>
“In this paper, we consider how valid conclusions often lay hidden within research reports, masked by plausible but unjustified conclusions reached in those reports. These conclusions do not necessarily involve the use of questionable research practices. Invalid conclusions may be reached based, not on failing to report dropped conditions, failed studies, or nonsignificant analyses, but on selective interpretations of data that highlight researchers’ preferred conclusions while masking more valid ones.”
JC comment: This is basically the problem that I have with the IPCC assessment reports. Deep in the chapters, there is much good information that is reliable, although the reports relatively ignore some topics. The problem is with the conclusions that are reached (particularly in the Summary for Policy Makers), and inflated levels of confidence that are ascribed to these conclusions.
“We characterize situations in which the data justify a different conclusion than reached in a published report as situations in which that different conclusion is “masked.” Masked phenomena may constitute alternative explanations for a pattern of results, reasons to believe the published interpretations are true but exaggerated, or reasons to believe the published interpretation is simply incorrect. These conclusions are typically masked because the original report does not even consider or acknowledge them, and because the data that are presented usually create the superficial appearance of support for the presented conclusions.”
JC comment: I touched on a related issue in my paper Reasoning About Climate Uncertainty, in context of the ‘framing error.’ In my recent reports and congressional testimony, I typically quote the IPCC chapters extensively, in context of different arguments or confidence levels than those reached by the IPCC.
“Phenomena may often be masked because researchers failed to include procedures that could reveal them. Researchers’ data may be clean (obtained without any questionable practices) and analyses performed statistically appropriately, and their conclusion may still be wrong.”
JC comment: The whole IPCC effort, under the mandate from the UNFCCC, has been framed in terms of assessing ‘dangerous anthropogenic interference with the climate system.’ This results in systematic masking of natural internal variability and ignoring other processes (e.g. solar indirect effects).
“Confirmation bias leaves other, often more viable, interpretations masked by virtue of being neither tested nor mentioned in the scientific articles. Confirmation biases (seeking information interpretable as evidence of motivational influences on perception, but not seeking to disconfirm such influences) has led to many unjustified or inadequately justified conclusions.”
JC comment: Bernie Lewin’s book on the IPCC outlines the politics surrounding the manufacture of the IPCC consensus during the period of Second Assessment Report. Failure to identify a ‘discernible’ anthropogenic warming and dangers associated with the warming not only would have reduced the influence of the IPCC in climate policy deliberations, but in the early years would have justified its disbanding by the UNEP.
“Scientists should not be in the business of simply ignoring literature that they do not like because it contests their view. Nonetheless, our view is that overlooking a large body of research that appears to directly conflict with one’s conclusions is a problematic practice whenever it occurs. And the solution is simple — cite it, grapple with it, and, if one is claiming one effect is stronger than another, report effect sizes for both.” “We recognize that it is not possible for every researcher to be aware of every study that has ever been published in their field.” “[But] true sciences do not act as if data that conflicts with a preferred narrative simply do not exist.”
JC comment: Without endorsing the NIPCC in any way [see this previous post], if you compare the bibliography/references for the IPCC versus the NIPCC assessment reports for comparable topics, you will find relatively little overlap in the list of papers in the bibliography. I understand that that some selection is appropriate, but the published literature is sufficiently broad to support multiple, different arguments and narratives.
Another point. The scientist activists (Lewandowsky, Hayhoe et al) think that this paper debunks all of the skeptical papers (the so called 3%) Those 3% of scientific papers that deny climate change? A review found them all flawed. I have no idea how this twaddle gets published.
Each week (or every 2-3 weeks), most of the science papers (top third) in my Week in Review posts fit in with a narrative (at least in my own head) that challenges some aspect of the so-called ‘consensus.’ I wish I had time to write all this up.
“The incentives that reward the telling of compelling narratives in scholarship encourage cherrypicking. To some extent, the practice of cherrypicking presents a classic social dilemma: whereas it is in most individual scientists’ self-interest to tell compelling stories (facilitated by cherrypicking), it is clearly not in the interest of the scientific field as it undermines the field’s validity and credibility.”
JC comment: Scientists claim to have a privileged ‘seat at the table’ owing to their alleged higher levels of rationality that is associated with the scientific method. Cherry picking and other shenanigans negate that privilege, and activist scientists become no better than lobbyists.
“We anticipate several rewards for telling far less compelling narratives based on messy and contradictory data. First, we maintain our own scientific integrity. Second, we maintain the integrity of our field. Third, acknowledgement of conflicting results and messy data provides an opportunity for theoretical advance and new empirical research to resolve those conflicts, either by showing that one set of results are irreplicable, or by identifying conditions under which both sets of conflicting findings can be consistently obtained. Thus, the more traditional rewards may then become available to the researcher capable of resolving such conflicts.”
JC comment: Brilliantly stated. But all of this seems irrelevant to activist scientists who think they are saving the planet, or are addicted to careerist and financial rewards for sounding the alarm and ignoring (or worse) any science or scientist that doesn’t fit the narrative.
“We are merely arguing for a process that acknowledges and wrestles with data that does not comport with one’s preferred narrative.”
“The nature of scientific progress is that we will get many things wrong. A healthy science, however, will: 1. keep such errors to a minimum; and 2. quickly self-correct when errors have been made. Limiting its practices that camouflage true phenomena in the name of promoting Wow Effects and preferred narratives is one way to accomplish these goals.”
“These errors occurred, not because questionable research practices, but because of unjustified interpretations.”
JC summary: The masking bias and cherry picking isn’t research misconduct, but it misleads the science and policy makers alike. The IPCC, through its focus on ‘dangerous anthropogenic interference with the climate system’ has institutionalized the masking bias in climate science, with trickle down effects to the national funding agencies and what gets funded, how grant proposals and journal publications are framed, and what gets taught in university classrooms.
Viewpoint diversity and identification of alternative explanations and hypotheses
“When in doubt, we can seek out colleagues with very different views than our own. We do not have to agree with them or be persuaded by their arguments. But those who disagree with us will probably have very different blind spots than we have, and will usually be quite happy to point ours out. Of course, just because a colleague claims we have missed something important does not mean we actually have. The point is to reveal masked findings, studies, and explanations that our own blind spots have led us to miss. Once unmasked, nothing prevents us from critically evaluating them, too — we still can conclude that they are not as important as our critics presume. But at least we will have an opportunity to address them, rather than marching on as if they did not exist..”
“Ideally, when alternative explanations exist for a phenomenon, researchers will develop methodologies that pit alternative hypotheses against one another. The point is not to demonstrate that one is “true” and the other “false.” Indeed, some influential social psychological scholarship has advanced the position that most hypotheses are true under some conditions. If one subscribes to this view, it is downright silly to try to “disprove” any theory. Even if one holds this view, our perspective is that it is still invaluable to pit alternative perspectives against one an- other in particular research contexts. If everything is true under some conditions, then any particular hypothesis is probably not true under all conditions. To find out which conditions Hypothesis X accounts for all or most of the data, and under which conditions the alternative, Hypothesis Y does, we need to test both.
“This is why adversarial collaborations have considerable potential to advance the field. No matter how prone we are to confirmation bias, and how difficult it may be to be completely objective in our interpretations, we often have colleagues who are ready, willing, and able to tell us how wrong we are. To address these issues, then, we suggest social psychol- ogists play to one of their strengths. The field has long embraced diversity, in part on the grounds that people from diverse backgrounds bring different experiences to bear on psychological problems. In short, diverse people have diverse ideas, thereby enriching the “marketplace” of ideas.”
“Such collaborations are probably quite difficult, because those on opposing “sides” of some debate — whether theoretical or political — often hold considerable hostility for one another. Nonetheless, one of the few known solutions to confirmation bias is to adopt an alternative desired conclusion. It may not be easy, but our prediction is that it will usually be worth it.”
“And what about failed attempts at such collaborations? One criticism often leveled at adversarial collaborations is that they often do not work, because the adversaries are sufficiently hostile to one another, or one another’s views, that they cannot work together. This, we would argue, is a testament to just how powerful researcher confirmation biases can be. The lie is put to the ideal image of objective scientists reaching conclusions entirely on the basis of logic, method, statistics, and data by all such failures. Both sides may be equally culpable, or, perhaps, one side is biased and the other is not. Regardless, such failures are a strong signal that something other than the objective and dispassion- ate pursuit of truth is going on.”
“A recent article in astronomy (Loeb, 2014) has made important points about diversity of ideas.. Loeb (2014) highlighted example after example where prestigious astronomers “believed” something to be true on the basis of little or no evidence, obstructed the ability of younger scientists and others with new ideas to make progress on that problem because the alternatives were perceived as outlandish. In each case, many years later, it was ultimately discovered that the “outlandish” claim turned out to be true. In our terminology, unjustified but confidently-held conclusions masked the evidence, and sometimes even the search for evidence, of more valid ones. “Uniformity of opinions is sterile; the co-existence of multiple ideas cultivates com- petition and progress.”
“Of course, it is difficult to know in advance which exploratory path will bear fruit, and the back yard of astronomy is full of novel ideas that were proven wrong. But to make the discovery process more efficient … funding agencies should dedicate a fixed fraction of their resources (say 10–20%) to risky explorations. This can be regarded as affirmative action to promote a diversity of ideas….”
JC comment: Jussim et al.’s suggestion of collaborating teams of scientists with different perspectives simply won’t work for climate science. Activist scientists won’t debate climate scientists with different perspectives and even block them on twitter. If a non-activist scientist engages too enthusiastically with someone such as moi, Christy, Pielkes and the like, the activist scientists ‘pressure’ them into submission.
Gatekeeping and masking by journals
There have been many many instances of this recounted over the years, both in climate science and more broadly in academia.
I have a new anecdote to report that is relevant to this post, from reviews on a paper I recently submitted to a journal. Without getting overly specific about the paper in question or any specific criticisms (I am still trying to figure out what to do with this paper), a few statements from reviewers and editor give the game away:
“Overall, there is the danger that the paper is used by unscrupulous people to create confusion or to discredit climate science. Hence, I suggest that the author reconsiders the essence of its contribution to the scientific debate on climate science.”
A further gem from Reviewer #1:
“Finally, it includes some errors such as: “Known neglecteds in 21st century global climate change scenarios include: solar variability and solar indirect effects, volcanic eruptions, natural internal variability of the large-scale ocean circulations, geothermal heat sources and other geologic processes.”: this statement provided without justification and obviously wrong since this is evaluated in e.g. CMIP5 model experiments.”
Bazinga! Masking is fully successful; climate scientists now think that all this natural variability stuff (including geologic processes, volcanic eruptions and solar variability) are evaluated in the CMIP5 model experiments. (JC’s head explodes).
Based on these brilliant and hard hitting reviews, the Editor concludes:
“We regret that we cannot accept your manuscript for publication and will not consider it further.”
Even assuming that there are severe flaws in the paper (there aren’t; at least nothing approaching the misconceptions of the reviewers), explicitly stating that they would not even consider a resubmission is something I have never seen from an editor.
I need to find a journal with triple blind review system (which includes the editor).
Steve Koonin and red teaming
Hence the idea of ‘red teams’ and other similar methods to bring alternative perspectives into more prominence. See these previous posts for background.
Steve Koonin has been the most visible and prominent proponent of red teaming. Koonin recently gave a presentation on this at Purdue University.
Gavin Schmidt wrote a post at RealClimate on Koonin’s presentation . To put it politely, Gavin’s post is highly dismissive of what Koonin has to say.
Since the Steve Koonin that I know is very intelligent and insightful, I suspected that Gavin’s post was way off the mark. Then I spotted a post by Koonin at WUWT that clarifies what he actually said in his presentation, and how this was misrepresented and misinterpreted by Gavin.
I think Koonin’s overall concern with the climate science assessments relates to this idea of ‘masking bias.’ Framing the issue in this way may help with the justification of the various red teaming efforts under consideration in the U.S.