by Judith Curry
Insights into the motivated reasoning of climate scientists, including my own efforts to sort out my own biases and motivated reasoning following publication of the Webster et al. (2005) paper
A recent twitter thread by Moshe Hoffman (h/t Larry Kummer) reminded me of a very insightful paper by Lee Jussim, Joe Duarte and others entitled Interpretations and methods: Towards a more self-correcting social psychology
Apart from the rather innocuous title, the paper provides massively important insights into scientific research in general, with substantial implications for climate science.
The Jussim et al. paper is the motivation for this blog post that addresses the motivated reasoning of individual climate scientists. And also for my next post that will address the broader ‘masking’ biases in climate science.
<begin quote>
“Getting it right” is the sine qua non of science. Science can tolerate individual mistakes and flawed theories, but only if it has reliable mechanisms for efficient self-correction. Unfortunately, science is not always self-correcting. Indeed, a series of threats to the integrity of scientific research has recently come to the fore across the sciences, including questionable research practices, failures to replicate, publication biases, and political biases.
Motivated reasoning refers to biased information processing that is driven by goals unrelated to accurate belief formation. A specific type of motivated reasoning, confirmation bias, occurs when people seek out and evaluate information in ways that confirm their pre-existing views while downplaying, ignoring, or discrediting information of equal or greater quality that opposes their views. People intensely scrutinize counter-attitudinal evidence while easily accepting information supporting their views. People generate convincing arguments to justify their automatic evaluations, producing an illusion of objectivity.
Scientists are not immune to confirmation biases and motivated reasoning. Values influence each phase of the research process, including how people interpret research findings. Reviewers’ theoretical and ideological views can influence their evaluation of research reports, leading them to judge studies that oppose their beliefs more critically than studies supporting their views. Consequently, they are then less likely to recommend publication of studies with undesired findings or funding for studies based on undesirable theories or hypotheses.
There are powerful incentives to present a strong, compelling story when describing their research. Most of us are motivated to get the science right, but we are also motivated to get the studies published and our grants funded. We want our colleagues to find our research sufficiently interesting and important to support publishing it, and then to cite it, preferably a lot. We want jobs, promotions, and tenure. We want popular media to publicize our research and to disseminate our findings beyond the confines of our lab. We might even hope to tell a story so compelling we can produce a bestselling popular book and receive lucrative consulting and speaking engagements, or have our findings influence policy decisions.
In brief, powerful incentives exist that motivate us to achieve — or, at least, appear to achieve — a “Wow Effect”. A “Wow Effect” is some novel result that comes to be seen as having far- reaching theoretical, methodological, or practical implications. It is the type of work likely to be emulated, massively cited, and highly funded.
Compelling, persuasive narratives are amply rewarded by promotions, grants, named chairs, etc., but the relationship of “compellingness of narrative” to validity (effect size, replicability, generalizability, etc.) is currently unknown. This raises the possibility that for some unknown and possibly substantial portion of the time, we are rewarding research practices that produce Wow Effects that are false, distorted, or exaggerated. We next demonstrate how mundane explanations for the same data remain hidden in the depths of the theorizing, methodology, statistics, and conclusions of some major areas of psychological science.
A checklist for increasing confidence that our research is relatively free of motivated biases:
- What do I want to happen and why? An honest and explicit self- assessment is a good first step towards recognizing our own tendencies towards bias, and is, therefore, a first step to building in checks and balances in our research to reduce them.
JC comment: This one is the most subjective, but in many ways the most telling. Are careerist objectives paramount in publishing this paper (for yourself, or to support a student or postdoc’s career objectives)? Are you looking to support your preconceived scientific notions or ideology, or are you looking to advance the science? If your answer to any of the following questions indicate bias, then you should come back and think harder about #1.
- Am I shooting for a “Wow Effect!”? Am I painting a weak and inconsistent result as dramatic in order to tell a compelling story? Scientific ambition is not inherently problematic, and may be a powerful constructive force for scientific advancement. But we want our literature to have true, valid, Wow Effects, not ones that cannot be replicated or ones promoted as powerful and pervasive, which upon further reflection (or evidence-gathering) are, in fact, weak, fragile, and fleeting, or which can be easily called into question under critical scrutiny.
JC comment: A litmus test of this is whether you are planning the press release for your paper before it is even accepted for publication. Do you care more about whether your paper will stand the test of time, or are you more interested in short-term publicity and publication in a high impact journal that looks for ‘wow’ papers? Part of this is exacerbated by the high impact journals such as Nature and Science, with press embargoes, that are clearly going for the ‘wow’ factor. A big problem is that many of these papers (particularly in Nature Climate Change) do not survive the first week of their press release without massive flaws having been uncovered.
- Do I have a long track record of research that systematically validates a particular political or social narrative or agenda? This is not about one’s intentions but rather one’s results. If one’s results consistently validate a particular set of beliefs, values or ideology, one has failed this check, and suggests that attempts at falsification may be in order.
JC comment: Falsification is maybe not the right word here; rather ‘refutation’ should be attempted. This should be attempted as a regular practice. A scientist should always ask “how might I be wrong?” at every step of their research. When the conclusions of your research are always predictable to outsiders, then your research will appear biased.
- Am I receiving remuneration (e.g., speaking or consulting fees) for reaching a particular conclusion? Conflicts of interest, though they do not invalidate one’s conclusions, plausibly place one at greater risk of dubious research and interpretation practices more generally.
JC comment: Also consider biases that may be introduced from ideas you submitted in a federally funded grant proposal, that you are seeking to confirm. Did you submit ideas supporting the consensus on climate change, that you thought would give your proposal a better chance of funding? See this previous post
- Have I generated theoretical arguments for competing and alternative hypotheses and designed studies to incorporate and test them?* Honest tests of alternatives can go a long way to reducing personal bias.
JC recommendation: if you are unaware of competing and alternative hypotheses, check out the papers listed in my Week in Review posts , and also my posts on attribution – the topic that is the source of most of the debate.
- Have I read some of the literature highlighting the invidious ways our motivated biases, morals, and politics can creep into our scientific scholarship? Doing so can alert one to ways in which our preferences might distort our science. After having done so, have I made a good faith attempt to eliminate such biases from my scholarship?
JC recommendation: check out my collection of blog posts related to this topic, discussing relevant papers in the literature [link]
- Have I sought feedback from colleagues with very different preferences and perspectives than mine or with track records of scholarship that often contest my preferred narratives?
JC comment: If you are active on twitter and block other publishing climate scientists, that is a hint that you deserve an ‘F’ on this one. I get that there are morons in the twitosphere, by all means mute or block them. But don’t wear your bias on your sleeve by blocking other climate scientists! Take note, Michael Mann and Katherine Hayhoe. For the latest drama in this regards, see this from Ross McKitrick. UNbelievable.
“It may not always be possible for researchers to meet all of these checks. However, as a starting heuristic, meeting six of the seven probably justifies confidence that the research has kept bias mostly in check. What to do if one cannot meet at least six (or, alternatively, one fails too many of one’s own such questions?). Although that, too, is a matter of judgment, one possibility will be to start over.”
<end quotes>
JC’s struggle with ‘motivated bias’ – Webster et al. (2005)
The saga of my own fight against motivated bias begins with publication of Webster, Holland, Curry, Chang (2005): Changes in tropical cyclone number, duration and intensity in a warming environment.
Webster’s motivation for investigating this topic was that he was disturbed by Kevin Trenberth’s Science paper and public pronouncements about increasing intensity of hurricanes while he was a lead author of the IPCC AR4 (which was in the ‘discussion’ phase at the time), which Webster regarded as unsupported. He supported Chris Landsea’s decision to resign from the IPCC over Kevin Trenberth’s statement.
Webster was surprised when the result of his investigations actually supported Trenberth’s assertions.
Prior to publication of the Webster et al. (2005) paper on hurricanes, I was blissfully well outside of the scientific or public debate on climate change. As an established, tenured Professor, my main objectives in publishing papers were to produce seminal papers that would stand the test of time, and hopefully change the way scientists think about the topic I was publishing on. I was also motivated to help my students and postdocs get established in their scientific careers.
That all changed in September 2005, following publication of the Webster et al. paper, several weeks following the devastation from Hurricane Katrina. The story behind all that was recounted in an interview with Keith Kloor at the now defunct Collide-a-scape.
The relevant issue here is that I became enmeshed in a scientific and public debate that was rife with minefields that would contribute to motivated bias. The first front in the ‘war’ surrounding the Webster et al. paper was the hurricane researchers, notably Bill Gray and Chris Landsea. The attacks on us, particularly by Bill Gray, were ugly. Then we were attacked by the professional climate ‘skeptics’,from the think tanks. The ‘hurricane wars’ was a huge story in the media. (see also Chris Mooney’s book Storm World).
We were being attacked publicly; this was WAR on science. In our beleaguered state, we were ‘adopted’ by the enviro advocacy groups and the activist scientists (including RealClimate bloggers and Joe Romm). I became a ‘partisan’ on this topic; not so much the broader issue of AGW (but I decided at that time to generally accept the consensus), but on the specific issue of hurricanes and global warming.
Publication of the Webster et al. paper (also Emanuel, 2005) stimulated hundreds of publications on this topic. In the following year I was asked to review many many papers on this topic. My first reaction to receiving such a paper was to quickly figure out what ‘side’ of the debate the paper fell on. I was very hard on papers that were generally critical. I was also very hard on papers that supported our paper; after all, it wasn’t going to help ‘our side’ if weak papers got published.
I became a ‘partisan’ on this subject, and more broadly the issue of AGW. I was a soldier in the noble fight against the war on climate science. I started paying attention to social media and blogs, and I became intrigued by RealClimate.
At the same time, I was most definitely paying attention to the criticisms from Gray, Landsea and others related to the quality of hurricane intensity data. I became increasingly intrigued by the Atlantic Multidecadal Oscillation, and was puzzled by the mid-century warming ‘hiatus’.
The personal and professional shock of entering the public debate on climate change was deeply unwanted, surprising, and disturbing. I could have stayed out of all this, but I was deeply disturbed by the ‘war on science.’ Why couldn’t scientific disagreement play out in the usual way (conferences, publications), with the media acknowledging uncertainty and disagreement? Well, the answer to that question was that urgent policy decisions were at stake, including rebuilding New Orleans. With regards to AGW, for the first time the public realized that even 1 degree warming could actually matter, if it caused more intense hurricanes. This was seized on by climate change activists, with an equal but opposite response by the libertarian/conservative advocacy groups.
I was concerned about bias being introduced into the science by this partisan ‘war.’ My reflections on all this were published in the 2006 paper by Curry, Webster and Holland Mixing politics and science in testing the hypothesis that greenhouse warming is causing a global increase in hurricane intensity that was submitted in November 2005. Upon rereading this paper 13 years later, I still really like it. You can already see evidence of my readings from philosophy and social science in trying to grapple with what was going on.
Upon publication, our 2006 paper saw extensive discussion in the blogosphere. I used google to identify the blogs that were discussing this, and I stopped by each, leaving a comment stating my willingness to answer any comments. On one blog, I entered into a particularly interesting discussion, where people wanted to look at the data, asked questions about the statistical methodology, etc. A few days later I realized I was at the nemesis blog of RealClimate (ClimateAudit). I continued to engage at CA (see the Collide-a-scape interview for further details.) During the period 2006-2010, the main blogs I participated in were ClimateAudit and Collide-a-scape.
Up until 2009, I was still considered as an ‘ally’ by the AGW advocate community. (Although there were early hints from the Climategate emails of Mann’s ‘displeasure’ re my comments in a NRC committee to review a doc on temperature trends.)
Of course, this all changed in Nov 2009 with the Climategate emails, you can read my perspective on this in the Collide-a-scape interview. I was still fighting against the ‘war on science,’ but I was reconsidering who the ‘bad guys’ were. Over the next few years, the reception of the activist wing of climate science to my response to Climategate and Climate Etc. clarified all this, with serious implications for the integrity of climate science.
In August 2010, I started Climate Etc., the blog was seeded with material from my draft ‘uncertainty monster’ paper. Apart from scientific topics, my motivation was to grapple not only with any personal bias that I might have, but to understand bias in climate science caused by the politicization of the topic. Almost 9 years later , I think some things have improved, but the climate scientist activists have further entrenched their biases, to the great detriment of climate science and the climate policy debate.
So, how did I end up taking a different path and ending up in a different place than say Michael Mann, Katherine Hayhoe, or whoever?
First, as a female scientist of my generation, I wasn’t really entrained into the ‘power’ community surrounding climate science, although in the 2000’s I was named to some National Academy and other advisory committees. So my career path wasn’t invested in this kind of ‘power’ climb to influence climate science or public policy. I wasn’t editor of any journals, a lead author for the IPCC, etc. I was more interested in doing my own research. When I went to Georgia Tech in 2002, my main objective was in building a faculty and mentoring them and developing a good educational, professional and personal environment for students. So my career objectives were not really tied up in the ‘AGW enterprise.’
My generation of scientists (60+) have mostly identified as atmospheric scientists (meteorologists), oceanographers, geologists, geographers. By contrast, younger scientists (particularly those receiving Ph.D. since 2000) studying any topic related to climate pretty much have their careers defined by the AGW enterprise. As a percentage, I suspect that a far lower number of 60+ climate scientists are activists (and are more ‘skeptical’), relative to a large percentage of under 50’s (who don’t seem skeptical at all). Somebody outa do a survey.
Second, politically I’m an independent with libertarian leanings, and I have never been particularly aligned with environmental movement (while I highly value clean air and water and species diversity, the environmental movement seems motivated by other issues). I simply don’t have the soul of an ‘activist.’
Third, since my days as a graduate student I have had an abiding interest in philosophy and the social sciences, particularly as related to science.
Fourth, I care more about whether my publications will stand the test of time and contribute to deep understanding, than I care about the ‘wow’ factor, which I regard as transient and leading to nothing but trouble (e.g. Webster et al. 2005).
Fifth, at this stage of my life I can afford to buck the ‘system.’ 20 years ago, when I had a mortgage payment and college tuition to pay, there is no way I would have put myself out on such a controversial limb. There is only so much personal and professional integrity that you can afford, if your job might be at stake.
So that summarizes my personal journey, over the past 14 years, to fight against my own personal biases. Through Climate Etc. I provide resources that I hope others can use to think about, understand and challenge their own biases. Apparently trying to fight against bias in climate science gets you labeled as a ‘denier’, ‘anti-science,’ ‘serial climate disinformer.’ There seems to be no end to the perversions of ‘motivated’ climate science.
What’s next: If you are a true believer in AGW and the urgent need to act, you will think this is all irrelevant, e.g. settled science, 97% and all that. The bigger problem than motivated bias in individual scientists is when this bias gets institutionalized. The Jussim et al. paper also provides insights into this that are relevant to climate science, which will be the topic of my next post.