by Kip Hansen
John P. A. Ioannidis dropped another cluster bomb on the medical research world two weeks ago with his latest paper which concludes:
“Overall, not only are most research findings false, but, furthermore, most of the true findings are not useful.”
Ioannidis [pronounced yo-NEE-dees) ] is not some grumpy gadfly or crusty curmudgeon casting stones at his fellow medical researchers. He speaks from a high bully pulpit at Stanford University School of Medicine where he is a Professor of Medicine and of Health Research and Policy, a Professor of Statistics (Stanford University School of Humanities and Sciences) as well as the Director of the Stanford Prevention Research Center. [full bio here]
He is famous, or infamous — depending on your point of view — for his 2005 paper “Why Most Published Research Findings Are False” (.pdf here), the first cluster bomb. This earlier essay is a must read for anyone concerned about the state and efficacy of scientific research – regardless of the field. Ioannidis boldly proclaims “most research findings are false for most research designs and for most fields”. Further, Ioannidis states simply: “Claimed research findings may often be simply accurate measures of the prevailing bias”. Some of the causes for this are further discussed in a 2011 essay in Scientific American entitled “Epidemic of false claims” [link]. Judith Curry discussed this in an essay by the same name at Climate Etc. in July 2012 as it relates to climate science. It is important to understand that the basis for Ioannidis’ assertion of the falsity of most research claims is not that researchers are knowingly biased, poorly trained or engaged scientific misconduct of any kind – read it, the text of the paper is just a few pages.
In his latest essay, Why Most Clinical Research Is Not Useful [.pdf here] Ioannidis is referring specifically to clinical medical research – which he defines as “all types of investigation that address questions on the treatment, prevention, diagnosis/screening, or prognosis of disease or enhancement and maintenance of health.” He clearly distinguishes that type of research from what he terms “speculative, blue-sky research…[which]…cannot be easily judged on the basis of practical impact”. In a nutshell, “many of the features that make clinical research useful can be identified, including those relating to problem base, context placement, information gain, pragmatism, patient centeredness, value for money, feasibility, and transparency.“
Ioannidis says “’Useful clinical research’ means that it can lead to a favorable change in decision making (when changes in benefits, harms, cost, and any other impact are considered).” In this essay, I will attempt to apply, to translate, Ioannidis’ criteria for “useful clinical research” – research that does not waste research dollars and research effort – to the scientific field known as Climate Science.
Let’s acknowledge that there seems to be a more clearly defined line in medicine between clinical research and general research, Ioannidis’ blue-sky research that looks for new knowledge and better understanding of the complexities of the human body. Nevertheless, there is something for us to learn from Ioannidis’ comments about useful research. Moreover, this is not intended to be an essay of answers but rather an essay of questions.
For the purposes of this essay, we will set the definition of “useful climate research” as:
“Climate research that can lead to a favorable change in decision making regarding climate when changes in benefits, harms, cost, and other impacts are considered.”
The first criterion is that research findings be true – for climate science, that means that the issues raised in Dr. Ioannidis’ first essay must be satisfied – that there has been proper design, execution, and analysis of the right problems so that the results are correct, a faithful reflection of reality, and not just a reflection of prevailing bias in the field.
Remember, we are leaving aside “blue-sky research” (no pun) – research into the larger questions of Earth’s climate – the cause of ice ages, what causes clouds, the causes and effects of ocean currents and the like. Much of this type of research needing yet to be done revolves around and includes the getting at answers, better understandings, to the known unknowns and discovering the unknown unknowns.
Judith Curry, in her 2013 essay Pasteur’s quadrant, presents this graphic, which I have modified slightly, that helps to make the distinction between Ioannidis’ blue-sky research – the Bohr Quadrant – use-inspired research — the Pasteur Quadrant of blended values — and clinical (for us, practical) research – the Edison Quadrant:
I have labeled the lower left quadrant, which I labelled the Ioannidis Quadrant, comprising research that is low in its ability or intention to bring us fundamental or new understanding and also low in usefulness.
In the accompanying essay (linked above) Curry refers to this diagram in this way:
“….the defining characteristic of basic research is its attempt to find more general physical and natural laws to push back the frontiers of fundamental understanding.”
“With regards to climate science, the concern that I have is that there is too much research in the lower half of Stokes’ diagram, scoring low on making advances to fundamental understanding. Applied research that is useful and used is a good thing, but at the end of the day I don’t see all that much applied climate research actually getting used by decision makers. The primary problem being that there is too much focus on the climate models, and the climate models are not yet up to the task.”
“This leaves us with the unnamed 4th quadrant, which is often characterized as ‘taxonomy’, i.e. research that is neither useful nor contributes to fundamental understanding. Climate model taxonomy is characterized by endless analysis of IPCC climate model runs and projection of ‘dangerous impacts’ . If these are not being used by decision makers, then they are in the 4th quadrant.”
I would add, in the present, that many the results of climate model taxonomy may be worse than useless, they may be harmful if they are being used by decision makers to set policy based on findings that are most probably false. At best, such results certainly lead to confusion and unfounded concern for the general public.
There is a second, and possibly more harmful effect of useless research, highlighted by Ioannidis and applied to climate science by Curry in this quote:
“…I am immensely concerned by the overemphasis on climate model taxonomy, whereby scientists write papers analyzing the output of the IPCC climate model simulations, and infer future catastrophic impacts, and it seems far too easy for this kind of research to get published in Nature and Science. In the meantime, the really hard research problems are all but ignored, such as fundamental research into ocean heat transfer, multi-phase atmospheric thermodynamics, synchronized chaos in the coupled atmosphere/ocean system, etc. Not to mention the more manageable problems such as careful consideration of the attribution of climate variability during the period 1850-1970”. – from the essay Lennart Bengtsson on global climate change May 13, 2013.
When scarce, precious research dollars and research time/effort are spent on useless research, then important basic, fundamental research and pragmatic useful research do not get funded and does not get done.
In this regard, we can identify current areas of climate research that can be classified as neither “blue-sky research” or “useful” – research which can be rather readily classified as “not useful”, to use as an example for discussion. Recognizing that careers and livelihoods are at risk in such an assessment, we might want to be careful in our listing, and will not use identifiable examples, but rather, we will use Judith Curry’s example — climate model taxonomy.
An Example of Not Useful Climate Research – Climate model taxonomy … characterized by endless analysis of IPCC climate model runs and projections of ‘dangerous impacts’.
Using Ioannidis’ definition of useful above, let’s apply it to this example of “not useful research”.
It is hardly necessary to give specific examples, since they appear at least weekly in the media – yet another projection made using climate models that predict or project that in 50 years or 100 years, such things as the 6th Mass Extinction, millions of climate refugees storming borders as they are forced to move North to cooler climates, destruction of World Heritage sites, inundated island nations – you see the headlines yourself.
Remember, here we are not asking if the results are true or accurate – we are not asking, will the projection turn out to have been right? — if Ioannidis is right, nearly all are equally probably wrong – if Curry is right, such projections are based on models which are currently technically incapable of returning correct projections.
What we want to know is: Has it — the research, the money spent, the research hours spent analyzing, the expensive computer time, the effort of the publishing process – added anything useful to our knowledge of the climate? Has there been a gain in information commensurate with the cost — the expense in money and researcher-effort? Does this information help us make better decisions? Or has it been wasteful research of the same nature as the wasteful clinical medical research described by Ioannidis?
For each research proposal, Ioannidis suggests these questions, to which I offer my guesses both for our example and in a more general sense:
Problem base –> Is there a health climate problem that is big/important enough to fix?
Climate Model Taxonomy – What is the problem to being solved? What are we to do with projections so far out and so speculative? Does the projection offer solutions, even if true? These types of studies add nothing to our knowledge base.
Generally – There is a great deal of research that could be accomplished by collectively identifying climate problems/questions that are big and important enough to justify the effort.
Context placement –> Has prior evidence been systematically assessed to inform (the need for) new studies?
Climate Model Taxonomy – Unfortunately, many of these studies seem to be performed because they are relatively easy, readily publishable, and produce fame-enhancing press releases for the researchers and their institutions. They have nothing to do with the need for new studies.
Generally — Climate Science does not seem to be engaged in systematic assessments to determine need for new studies, or, possibly, is not a mature enough field to be sufficiently self-aware to see the need to review and replicate, verify, and critically assess its own prior evidence and assumptions.
Information gain –> Is the proposed study large and long enough to be sufficiently informative?
Climate Model Taxonomy – Except for curiosity value – feeding the public’s apparent love of scary stories and the activists need for alarming ‘science-based’ scenarios – there is no perceivable information gain at all in these types of studies.
Generally — In climate science, this question revolves around the available data being long enough (time), broad and dense enough (spatially) and accurate/precise enough to inform us in a useful manner — it may be necessary to use available data, even if insufficient, to incrementally improve our understanding, as long as uncertainties and imprecision are fully acknowledged. For example, topics like Ocean Heat Content, Ocean pH, General Cloudiness, Global Rainfall and many other important data sets do not have sufficient data over sufficient time periods for reliable analysis. These fields, and many others, may need blue-sky research efforts focusing on long-term data collection.
Pragmatism –> Does the research reflect real life? If it deviates, does this matter?
Climate Model Taxonomy – It is unlikely that any of the results from climate model taxonomy reflect the real world in any meaningful way as the climate models are known to be too immature to return reliable results at the claimed scales of time and space.
Generally – Current climate models are known to have problems, some quite major, and are generally acknowledged not to reflect the real life climate system adequately, certainly at least for making projections on a decadal and regional level – the type of projections most in demand by policy makers. Paying more attention to the real-worldness of research problems and research design, posing pragmatic questions for researchers to answer, would be a positive step for the field. Judith Curry recently mentioned that her private company’s “energy sector clients are already asking about next winter’s temperatures, and whether we can expect a typical La Nina pattern.” That is a pragmatic question. In a climate related field, ocean acidification, they have had to establish new research guidelines to try to improve on the applicability of their research findings to the real oceans and its biota because, “Yes, it matters”.
Patient Stake-holder and Policy Maker centeredness –> Does the research reflect top patient stake-holder and policy maker priorities?
Climate Model Taxonomy – It is doubtful that any important stake-holder or policy maker has been demanding a continuing stream of alarming projections – outside of the activist realm.
Generally — climate science does not seem to be asking stake-holders and policy makers what they need to know to make decisions, instead it seems to continue to speak-truth-to-power while simultaneously advocating political solutions. It is unwise for the field to assume that it already knows the answers to that question.
Value for money –> Is the research worth the money?
Climate Model Taxonomy – To have any value, it must first be true, or at least truer than existing knowledge. If it has passed that test, then it must supply some of the value in regards to the questions above as to utility – if it does not lend support to making better decisions, or advance our knowledge of how the climate works, then it is worthless, regardless of veracity or actual cost.
Generally – Climate Science is dealing with a question deemed ultimately important. Will we, humanity, be able to continue to survive on this planet? Or will we be forced to find another home in the future due to the consequences of climate change? There are limited resources available for climate research, and they must, because of this framing, be spent on useful, meaningful research.
Feasibility –> Can this research be done?
Climate Model Taxonomy – Unfortunately it is far too easy (comparatively) to write new code to reanalyze massive amounts of data and tune the parameters to produce differing results, even if they are entirely speculative.
Generally – The experiment many would like to do is jack up the atmospheric CO2 on Earth-Beta to 600, turn the speed knob up to 100x, and watch the results for a year. That experiment cannot be done. We cannot stop of Gulf Stream for a few years and observe what happens. We cannot call up a decade long El Niño to witness the effect on Global Surface Temperatures. As the saying goes, we only have one planet. The very nature of the climate system – its nonlinearity, its complexity, its interconnectedness – make feasibility a major issue for climate science. The corollary to this question is: If we do some feasible version of some climate experiment, will it really inform us of the thing we want to know? Will it also pass the pragmatic, real-world-ness test?
Transparency –> Are methods, data, and analyses verifiable and unbiased?
Climate Model Taxonomy – probably not (though there has been some improvement).
Generally – Climate Science has an abysmal record in this area. There are major efforts being made to improve all fields of research in regards to these questions. Because of past failures, some of our current work based on past results may be tainted beyond utility. Quoting Ioannidis (regarding clinical medical research) “Trust has been eroded whenever major subversion of the evidence has been uncovered by legal proceedings or reanalysis with different conclusions …. Biases in the design, analysis, reporting, and interpretation remain highly prevalent.” Does this apply as well to current research in climate science?
Bottom-line Questions:
Should climate research be useful?
If it should be useful, can climate research be useful?
What are the climate questions that need answering so that there can be “a favorable change in decision making” – questions that when answered provide reliably true information that facilitates decisions that will bring about real benefits to society and the environment when harms, cost, and other impacts are considered?
And for individual climate researchers and research teams: Is the research I/we are currently involved in sufficiently useful to justify the costs and effort?
Ioannidis concludes his essay with this:
“Improving the Situation: The problem of nonuseful research should not be seen as a blame game against a specific group (e.g., clinical climate researchers) but instead should be seen as an opportunity to improve. The challenges and the problems to solve involve not only researchers but also institutions, funding mechanisms, the industry, journals, and many other stakeholders, including patients policy-makers and the public. Joint efforts by multiple stakeholders may yield solutions that are more likely to be more widely adopted and thus successful.”Filed under: Sociology of science
