by Judith Curry
We survey the rationale and diversity of approaches for tuning, a fundamental aspect of climate modeling which should be more systematically documented and taken into account in multi-model analysis. – Hourdin et al.
Two years ago, I did a post on Climate model tuning, excerpts:
Arguably the most poorly documented aspect of climate models is how they are calibrated, or ‘tuned.’ I have raised a number of concerns in my Uncertainty Monster paper and also in previous blog posts.
The existence of this paper highlights the failure of climate modeling groups to adequately document their tuning/calibration and to adequately confront the issues of introducing subjective bias into the models through the tuning process.
Think about it for a minute. Every climate model manages to accurately reproduce the 20th century global warming, in spite of the fact that that the climate sensitivity to CO2 among these models varies by a factor of two. How is this accomplished? Does model tuning have anything to do with this?
Well, in 2014 the World Climate Research Programme Working Group on Coupled Modelling organized a workshop on climate model tuning. The following paper has emerged from this workshop.
Hourdin, F., T. Mauritsen, A. Gettelman, J. Golaz, V. Balaji, Q. Duan, D. Folini, D. Ji, D. Klocke, Y. Qian, F. Rauser, C. Rio, L. Tomassini, M. Watanabe, and D. Williamson, 2016: The art and science of climate model tuning. Bull. Amer. Meteor. Soc. doi:10.1175/BAMS-D-15-00135.1, in press. [link to full manuscript].
Abstract. We survey the rationale and diversity of approaches for tuning, a fundamental aspect of climate modeling which should be more systematically documented and taken into account in multi-model analysis. The process of parameter estimation targeting a chosen set of observations is an essential aspect of numerical modeling. This process is usually named tuning in the climate modeling community. In climate models, the variety and complexity of physical processes involved, and their interplay through a wide range of spatial and temporal scales, must be summarized in a series of approximate sub-models. Most sub-models depend on uncertain parameters. Tuning consists of adjusting the values of these parameters to bring the solution as a whole into line with aspects of the observed climate. Tuning is an essential aspect of climate modeling with its own scientific issues, which is probably not advertised enough outside the community of model developers. Optimization of climate models raises important questions about whether tuning methods a priori constrain the model results in unintended ways that would affect our confidence in climate projections. Here we present the definition and rationale behind model tuning, review specific methodological aspects, and survey the diversity of tuning approaches used in current climate models. We also discuss the challenges and opportunities in applying so-called ‘objective‘ methods in climate model tuning. We discuss how tuning methodologies may affect fundamental results of climate models, such as climate sensitivity. The article concludes with a series of recommendations to make the process of climate model tuning more transparent.
If ever in your life you are to read one paper on climate modeling, this is the paper that you should read. Besides being a very important paper, it is very well written and readable by a non-specialist audience.
I’m not sure where to even start with excerpting the text, since pretty much all of it is profound. Here are a few selected insights from the paper:
Climate model tuning is a complex process which presents analogy with reaching harmony in music. Producing a good symphony or rock concert requires first a good composition and good musicians who work individually on their score. Then, when playing together, instruments must be tuned, which is a well defined adjustment of wave frequencies which can be done with the help of electronic devices. But the orchestra harmony is reached also by adjusting to a common tempo as well as by subjective combinations of instruments, volume levels or musicians interpretations, which will depend on the intention of the conductor or musicians. When gathering the various pieces of a model to simulate the global climate, there are also many scientific and technical issues, and tuning itself can be defined as an objective process of parameter estimation to fit a predefined set of observations, accounting for their uncertainty, a process which can be engineered. However, because of the complexity of the climate system and of the choices and approximations made in each sub-model, and because of priorities defined in each climate center, there is also subjectivity in climate model tuning (Tebaldi and Knutti 2007) as well as substantial know-how from a limited number of people with vast experience with a particular model.
Why such a lack of transparency? Maybe because tuning is often seen as an unavoidable but dirty part of climate modeling; more engineering than science; an act of tinkering that does not merit recording in the scientific literature. There may also be some concern that explaining that models are tuned, may strengthen the arguments of those claiming to question the validity of climate change projections. Tuning may be seen indeed as an unspeakable way to compensate for model errors.
Although tuning is an efficient way to reduce the distance between model and selected obser3vations, it can also risk masking fundamental problems and the need for model improvements. There is evidence that a number of model errors are structural in nature and arise specifically from the approximations in key parameterizations as well as their interactions.
Introduction of a new parameterization or improvement also often decreases the model skill on certain measures. The pre-existing version of a model is generally optimized by both tuning uncertain parameters and selecting model combinations giving acceptable results, probably inducing compensation errors (over-tuning). Improving one part of the model may then make the ’skill’ relative to observations worse, even though it has a better formulation. The stronger the previous tuning, the more difficult it will be to demonstrate a positive impact from the model improvement and to obtain an acceptable retuning. In that sense, tuning (in case of over-tuning) may even slow down the process of model improvement by preventing the incorporation of new and original ideas.
The increase of about one Kelvin of the global mean temperature observed from the beginning of the industrial era, hereafter 20th century warming, is a de facto litmus test for climate models. However, as a test of model quality, it is not without issues because the desired result is known to model developers and therefore becomes a potential target of the development.
There is a broad spectrum of methods to improve model match to 20th century warming, ranging from simply choosing to no longer modify the value of a sensitive parameter when a match is already good for a given model, or selecting physical parameterizations that improve the match, to explicitly tuning either forcing or feedback both of which are uncertain and depend critically on tunable parameters. Model selection could, for instance, consist of choosing to include or leave out new processes, such as aerosol cloud interactions, to help the model better match the historical warming, or choosing to work on or replace a parameterization that is suspect of causing a perceived unrealistically low or high forcing or climate sensitivity.
The question whether the 20th century warming should be considered a target of model development or an emergent property is polarizing the climate modeling community, with 35 percent of modelers stating that 20th century warming was rated very important to decisive, whereas 30 percent would not consider it at all during development. Some view the temperature record as an independent evaluation data set not to be used, while others view it as a valuable observational constraint on the model development. Likewise, opinions diverge as to which measures, either forcing or ECS, are legitimate means for improving the model match to observed warming. The question of developing towards the 20th century warming therefore is an area of vigorous debate within the community.
Because tuning will affect the behavior of a climate model, and the confidence that can be given to a particular use of that model, it is important to document the tuning portion of the model development process. We recommend that for the next CMIP6 exercise, modeling groups provide a specific document on their tuning strategy and targets, that would be referenced to when accessing the dataset. We recommend distinguishing three levels in the tuning process: individual parameterization tuning, component tuning and climate system tuning. At the component level, emphasis should be put on the relative weight given to climate performance metrics versus process oriented ones, and on the possible conflicts with parameterization level tuning. For the climate system tuning, particular emphasis should be put on the way energy balance was obtained in the full system: was it done by tuning the various components independently, or was some final tuning needed? The degree to which the observed trend of the 20th century was used or not for tuning should also be described. Comparisons against observations, and adjustment of forcing or feedback processes should be noted. At each step, any occasion where a team had to struggle with a parameter value or push it to its limits to solve a particular model deficiency should be emphasized. This information may well be scientifically valuable as a record of the uncertainty of a model formulation.
The systematic use of objective methods at the process level in order to estimate the range of acceptable parameters values for tuning at the upper levels is probably one strategy which should be encouraged and may help make the process of model tuning more transparent and tractable. There is a legitimate question on whether tuning should be performed preferentially at the process level, and the global radiative budget and other climate metrics used for a posteriori evaluation of the model performance. It could be a good way to evaluate our current degree of understanding of the climate system and to estimate the full uncertainty in the ECS. Restricting adjustment to the process level may also be a good way to avoid compensating model structural errors in the tuning procedure. However, because of the multi-application nature of climate models, because of consistency issues across the model and its components, because of the limitations of process studies metrics (sampling issues, lack of energy constraints), and also simply because the climate system itself is not observed with sufficient fidelity to fully constrain models, an a posteriori adjustment will probably remain necessary for a while. This is especially important for the global energy constraints that are a strong and fundamental aspect of global climate models.
Formalizing the question of tuning addresses an important concern: it is essential to explore the uncertainty coming both from model structural errors, by favoring the existence of tens of models, and from parameter uncertainties by not over-tuning. Either reducing the number of models or over-tuning, especially if an explicit or implicit consensus emerges in the community on a particular combination of metrics, would artificially reduce the dispersion of climate simulations. It would not reduce the uncertainty, but only hide it.
JC reflections
This is the paper that I have been waiting for, ever since I wrote the Uncertainty Monster paper.
Recall my early posts, calling for verification and validation of climate models:
- The culture of building confidence in climate models
- Climate model verification and validation
- Verification, validation and uncertainty quantification in scientific computing
The recommendations made by the Hourin, Mauritsen et al. for CMIP6 to document the decisions and methods surrounding the model tuning are a critical element of climate model validation.
For too long, the job of climate modelers has seemed to be to make sure their model can reproduce the 20th century warming, in support of the need for support of highly confident conclusions regarding the ‘dangerous anthropogenic climate change’ meme of the UNFCCC/IPCC.
The ‘uncertainty monster hiding’ behind overtuning the climate models, not to mention the lack of formal model verification, does not inspire confidence in the climate modeling enterprise. Kudos to the authors of this paper for attempting to redefine the job of climate modelers to include:
- Documenting the methods and choices used in climate model tuning
- More objective methods of model tuning
- Explorations of climate model structural uncertainty
But most profoundly, after reading this paper regarding the ‘uncertainty monster hiding’ that is going on regarding climate models, not to mention their structural uncertainties, how is it possible to defend highly confident conclusions regarding attribution of 20th century warming, large values of ECS, and alarming projections of 20th century warming?
The issues raised by Hourdin, Mauritsen et al. are profound and important. Here’s to hoping that we will see a culture change in the climate modeling community that increases the documentation and transparency surrounding climate model tuning and begins explore the structural uncertainties of the climate models.
Don’t forget to read the paper.
Filed under: climate models, Uncertainty