Overconfidence(?)

by Judith Curry
Yet not once has overconfidence by actual scientists been demonstrated. You just keep making that up. – Chris Colose

Below is the sequence of tweets that have motivated this post:
Judith Curry ‏@curryja Jun 28
Jay Ambrose: Climate scientist (Mann) using court to silence skeptics
An Archi Man ‏@HalJoneser Jun 28
@curryja @MichaelEMann Judith you seem to be as uncomfortable with the law as you are with clinate science? Both evidence based I suppose.
Judith Curry ‏@curryja Jun 30
@HalJoneser @MichaelEMann I’m equally uncomfortable about trying to silence critics with lawsuits & overconfidence in scientific statements
Chris Colose ‏@CColose 14h
@curryja @HalJoneser @MichaelEMann Yet not once has overconfidence by actual scientists been demonstrated. You just keep making that up.
Chris Colose’s tweet has been retweeted by Michael Mann, among others.
Overconfidence happens all the times, when individual researchers are convinced they are correct.  The more serious issue is overconfidence in context of the major assessment reports, i.e. IPCC.
Since Chris Colose and Michael Mann seem to be unaware of the overconfidence problem, I provide some documentation of overconfidence in the IPCC reports, in context of statements made in subsequent IPCC reports.
Hockey Stick
From the TAR: New analyses of proxy data for the Northern Hemisphere indicate that the increase in temperature in the 20th century is likely to have been the largest of any century during the past 1,000 years. It is also likely that, in the Northern Hemisphere, the 1990s was the warmest decade and 1998 the warmest year.
For background on the origins of the TAR conclusion, see this post from Lead Author John Christy IPCC TAR and the hockey stick.
From the AR4: Average Northern Hemisphere temperatures during the second half of the 20th century were very likely higher than during any other 50-year period in the last 500 years and likely the highest in at least the past 1300 years. 
The  warmest period prior to the 20th century very likely occurred between 950 and 1100, but temperatures were probably between 0.1°C and 0.2°C below the 1961 to 1990 mean and significantly below the level shown by instrumental data after 1980.
 Greater uncertainty associated with proxy-based temperature estimates for individual years means that it is more difficult to gauge the significance, or precedence, of the extreme warm years observed in the recent instrumental record, such as 1998 and 2005, in the context of the last millennium.
AR5: Continental-scale surface temperature reconstructions show, with high confidence, multidecadal periods during the Medieval Climate Anomaly (year 950 to 1250) that were in some regions as warm as in the late 20th century. These regional warm periods did not occur as coherently across regions as the warming in the late 20th century (high confidence).
For average annual NH temperatures, the period 1983–2012 was very likely the warmest 30-year period of the last 800 years (high confidence) and likely the warmest 30-year period of the last 1400 years (medium confidence).
JC comment:  Since the statement made in the TAR (with likely confidence level), subsequent IPCC reports have backed off significantly, including acknowledging that confident statements about warmest year or warmest decade are not supported by the data (this was discussed in the Climategate emails), and acknowledgement of magnitude of warmth in the Medieval Climate Anomaly that are regionally as warm as the late 20th century. The TAR was overconfident in its statement regarding the recent warming relative to the past 1000 years.
Sensitivity
The IPCC AR4 conclusion on climate sensitivity is stated as:
“The equilibrium climate sensitivity. . . is likely to be in the range 2oC to 4.5oC with a best estimate of about 3oC and is very unlikely to be less than 1.5oC. Values higher than 4.5oC cannot be excluded.” (AR4 SPM)
The IPCC AR5 conclusion on climate sensitivity is stated as:
Equilibrium climate sensitivity is likely in the range 1.5°C to 4.5°C (high confidence), extremely unlikely less than 1°C (high confidence), and very unlikely greater than 6°C (medium confidence) (AR5 SPM)
JC comment: The bottom of the ‘likely’ range has been lowered from 2 to 1.5oC in the AR5, whereas the AR4 stated that ECS is very unlikely to be less than 1.5oC. It is also significant that the AR5 does not cite a best estimate, whereas the AR4 cites a best estimate of 3oC. The stated reason for not citing a best estimate in the AR5 is the substantial discrepancy between observation-based estimates of ECS (lower), versus estimates from climate models (higher). AR4 was overconfident in its conclusions regarding climate sensitivity.
Near term projections of warming
AR4: For the next two decades a warming of about 0.2°C per decade is projected for a range of SRES emission scenarios. Even if the concentrations of all greenhouse gases and aerosols had been kept constant at year 2000 levels, a further warming of about 0.1°C per decade would be expected.
Since IPCC’s first report in 1990, assessed projections have suggested global averaged temperature increases between about 0.15 and 0.3°C per decade for 1990 to 2005. This can now be compared with observed values of about 0.2°C per decade, strengthening confidence in near-term projections.
In my recent post On the AR4’s projection of 0.2C/decade temperature increase, I provided additional details on the statements made in AR4, concluding:
The IPCC AR4 stands out, among the other four Reports, as providing a projection of 0.2C/per decade for the early 21st century, with a tiny uncertainty range. The projection is qualified only by the remark that 2 decades of averaging are needed to reduce internal variability in model response.
AR5:  The global mean surface temperature change for the period 2016–2035 relative to 1986–2005 will likely be in the range of 0.3°C to 0.7°C (medium confidence). This assessment is based on multiple lines of evidence and assumes there will be no major volcanic eruptions or secular changes in total solar irradiance.
However, the implied rates of warming over the period from 1986–2005 to 2016–2035 are lower [than the model simulations] as a result of the hiatus: 0.10°C–0.23°C per decade, suggesting the AR4 assessment was near the upper end of current expectations for this specific time interval.
JC comment: To make this comparable to the AR4 numbers, 0.3 to 0.7C over a period of 30 years translates to 0.1 to 0.23C/decade. The AR5 places medium confidence on this number. The AR5 clearly did a better job than AR4, by providing a confidence level and a much larger range of uncertainty.  The AR4 projection was over confident and with a very small uncertainty range, and in light of the hiatus, turned out to diverge substantially from the observations.
Time of emergence 
The IPCC AR5 states:
In summary, it is very likely that anthropogenic warming of surface air temperature over the next few decades will proceed more rapidly over land areas than over oceans, and that the warming over the Arctic in winter will be greater than the global mean warming over the same period. Relative to background levels of natural internal variability, near-term increases in seasonal mean and annual mean temperatures are expected to occur more rapidly in the tropics and subtropics than in mid-latitudes (high confidence).
The large range of values for ToE implied by different CMIP5 models, which can be as much as 30 years, is a consequence of differences in both the magnitude of the warming signal simulated by the models (i.e., uncertainty in the climate response) and in the amplitude of simulated natural internal variability.
Subsequent to the AR5, a new paper by Mora et al. was published in Nature that made a big media splash; previous CE post Time of emergence of a warming signal.  The paper discussed unprecedented temperature shifts and gave some
rather precise ranges for when these might occur, although the times of emergence were somewhat later than the AR5 assessment.
Ed Hawkins and a cast of big name climate scientists have a comment in Nature published today, entitled Uncertainties in the timing of unprecedented climates.  The full article is available(!) [here].  Ed also has a blog post [here].  Here is the abstract:
The question of when the signal of climate change will emerge from the background noise of climate variability—the ‘time of emergence’—is potentially important for adaptation planning. Mora et al. presented precise projections of the time of emergence of unprecedented regional climates. However, their methodology produces artificially early dates at which specific regions will permanently experience unprecedented climates and artificially low uncertainty in those dates everywhere. This overconfidence could impair the effectiveness of climate risk management decisions.
Mora et al. reply [here].  Punchline:
In the accompanying Comment, Hawkins et al. suggest that our index of the projected timing of climate departure from recent variability is biased to occur too early and is given with overestimated confidence. We contest their assertions and maintain that our findings are conservative and remain unaltered in light of their analysis.
Scott Powers was invited to write News and Views on this exchange, entitled Expulsion from History [link]; behind paywall.
Mora is featured in an interview at Yale Environment 360, published today, which makes no mention of the Hawkins et al. critique??
Roz Pidcock has a post on this exchange at The Carbon Brief entitled Overconfident predictions risk damaging trust in climate science, prominent scientists warn.  Excerpts:
But the precision with which Mora et al. predict when places will emerge from historical variability is unwarranted and misleading, say the scientists who authored today’s letter. Dr Ed Hawkins from Reading University tells us:
“Mora et al. claim that they can say precisely when the “point of emergence” will occur, to within a couple of years in many tropical regions. [We], along with many other papers in the scientific literature, provide ample evidence that this precision is not justified”.
This might seem like a fairly esoteric dispute. But it matters, the researchers say, because overstating the certainty with which scientists can make predictions risks losing public trust if they turn out to be wrong.
Of course, it isn’t abnormal for scientists to disagree. As Sutton explains in the video clip below, it’s a fairly normal way that scientific understanding advances.
But on this occasion, Nature has decided to take the unusual step of including an article by an independent climate scientist – who was not involved in either piece of research – to discuss the disagreement between author teams. We understand this is the first time the journal has done this for such an exchange.
The author of this “News and Views” article, professor Scott Power, from the Australian Bureau of Meteorology, concludes that in his opinion the approach taken by Hawkins gives a “more realistic estimate of the range … that could occur in the real world”.
We asked Mora about the difference between his and Hawkins’s approaches to expressing uncertainty. He told us both are valid, but that research by other scientists suggests his is the most appropriate to use in this case. In the Nature comment, Mora and colleagues cite a paper by Dr Claudia Tebaldi and Professor Reto Knutti in support of this point.
However, Tebaldi told us today she thought her work had been taken “absurdly out of context”. In a letter to the Nature editor, seen by us but as yet unpublished, Tebaldi says:
“I would like to call your attention to an egregious case of “citation out of context” appearing in the comment/rebut by Mora et al. today on your journal website. Their use of Tebaldi and Knutti (2007) to support their projections … is laughable, since our paper actually presents arguments against it”.
It is, in short, a right old scientific ding dong.
This exchange of views may seem esoteric, but the question of just how precise scientific predictions about the future can be is one which cuts to the heart of a lot of climate science and policy discussions.
With the authors of today’s letter arguing that oversimplifying complicated climate science does no one any favours, this probably isn’t the last we’ll hear on this issue.
JC reflections
Overconfidence is rampant in climate science; I have argued that we have the IPCC consensus seeking process largely to thank for the actual institutionalization of overconfidence related to climate science.
The sequence of IPCC reports since the TAR documents overconfident conclusions that are subsequently changed or presented with reduced confidence levels.  There are other examples that I did not include, notably from WG2 and also related to the attribution of extreme weather events.
Uncertain T. Monster is certainly(!) pleased to see Roz Pidcock’s article and Ed Hawkins concerns about adequately expressing uncertainty.  However, I think the AR5, Mora and Hawkins et al. are all overconfident in their statements on time of emergence of warming signals based on climate models.  The climate models are arguably over sensitive as per the growing divergence between climate model simulations and observations that caused the AR5 to reduce its decadal projections of climate change relative to the actual climate model simulations (see my post IPCC AR5 weakens the case for AGW).
Gotta love Pidcock’s phrase ‘a right old scientific ding dong.’  Acknowledging that disagreement exists and that disagreement is what moves the science forward, is too rare in climate science – kudos to Hawkins and Sutton.  And kudos to Pidcock for grasping the essential significance of this disagreement particularly as it relates to overconfidence.
Oversimplification of complicated climate science and overconfidence in conclusions does no one any favors.
 Filed under: Uncertainty

Source