pby Judith Curry
Some reflections on the east coast U.S. snowstorm: the forecasts, their communication and the response.
About a week ago, the weather forecast models suggested a strong storm developing off the east coast of the U.S. Subsequent forecasts were issued for catastrophic snowfall amounts for the coastal northeast U.S., including New York City. Government officials responded by shutting down transportation systems. On Tuesday morning, there was relatively little snow on the ground in New York, New Jersey and Pennsylvania, although there is considerable snowfall to the North.
This has engendered considerable discussion regarding the weather prediction models, communication of the forecast, and the response to the forecast. My diagnosis of the problem and recommendation are at the end of the post.
The forecasts
Several days ago, Andy Revkin did a nice post querying a number of weather forecasters and researchers about the relative merits of the different forecast models [link], particularly since everyone seemed to be paying attention to the European model (ECMWF) rather than NOAA’s GFS model.
With each successive forecast, the National Weather Service consistently forecasted worse conditions and higher snowfall amounts; yesterday the official NWS forecast was for 24-36” of snowfall. The Weather Channel and some private forecasting companies were more conservative, predicting half that amount.
Media and politicians vied for the most dramatic predictions. New York Governor Andrew Cuomo stated [link]:
“Massive snowstorms such as the one sweeping into the US north-east on Monday are “part of the changing climate”, New York’s governor, Andrew Cuomo, declared at a press conference announcing a state of emergency.
Cuomo said on Monday that “there is a pattern of extreme weather that we’ve never seen before” – reiterating his comments in the wake of hurricane Sandy, when he said that “anyone who says there’s not a dramatic change in weather patterns is probably denying reality.”
The climate change ‘hype’
Governor Cuomo was perhaps influenced by the media hype with scientists in a strong supporting role, here are a few examples:
- Washington Post: Global warming could make blizzards worse
- Think Progress: Climate Change Is Making Blizzards Like The One Hitting New England Much Worse, Scientists Say
- Inside Climate News: Why bigger snowstorms come with global warming
- Climate Nexus: Blizzard 2015: Normal winter weather amplified by climate change
The usual suspects: Kevin Trenberth, Jennifer Francis, Michael Mann, citing their ‘opinions’. The most entertaining ‘opinion’ is from Bill Nye, ‘Science Guy':
NYE: “And everybody, I just want to introduce the idea that this storm is connected to climate change. I want to introduce that idea. I know there will be certain viewers who will become unglued, they’re throwing things at their television set and so on. But is the economic effect of storms like this is huge. Now proving any one storm is connected, specially cold-weather events is difficult. But I just want to present that.”
TOURE: “Bill, I love you for bringing that in. Thank you so much, Bill.”
In terms of actual published science, Roger Pielke Jr.[link] cites two papers that show no trend in east coast winter storms.
Several months ago, Paul O’Gorman of MIT published a relevant paper [link to press release]:
O’Gorman found that there’s a narrow daily temperature range, just below the freezing point, in which extreme snow events tend to occur — a sweet spot that does not change with global warming. This is in contrast to average snow events, which may occur over a broader temperature range.
“People may know the expression, ‘It’s too cold to snow’ — if it’s very cold, there is too little water vapor in the air to support a very heavy snowfall, and if it’s too warm, most of the precipitation will fall as rain,” O’Gorman says. “Snowfall extremes still occur in the same narrow temperature range with climate change, and so they respond differently to climate change compared to rainfall extremes or average snowfall.”
What actually happened
Well, after all the hype, what actually happened? A summary is provided in this article in Huffington Post [link]. While the storm lived up to its billing in New England and New York’s Long Island, New York City and New Jersey were largely spared, with less than 10″ in New York City. By Tuesday morning, subways were up and running and driving bans had be lifted in New York City and New Jersey.
From Slate:
In sharp contrast, the storm was a flop in New York City. It’s essentially stopped snowing there, with totals averaging between 8 and 12 inches across the city. In a midday press conference, Mayor Bill de Blasio said, “We obviously missed the worst of the storm.” Defending actions by his office and Gov. Andrew Cuomo to shut schools and freeze regional transportation, de Blasio added, “Prepare for the worst. Hope for the best.”
The reason for New York City’s low totals? The National Weather Service strongly weighted its forecast toward the historically more accurate ECMWF model and the high-resolution NAM model, which showed the Long Island snow band stalling out directly over the city instead. That didn’t happen. In constructing its forecast, the New York City office of the NWS all but ignored its own recently upgraded GFS model, which showed significantly less snow in the city. As late as Monday evening, the NWS emphasized that the storm could overperform in NYC, saying “it should be a raging blizzard.”
New York Gov. Cuomo, perhaps conditioned by the state’s slow response to the recent Buffalo snowstorm, ordered a shutdown of virtually all modes of transportation in the New York City area on the basis of the National Weather Service forecast, including the city’s subway system, which had never previously closed for a snowstorm.
On twitter, I spotted some estimates that this ‘shut down’ cost NYC $1B, I have no idea about the accuracy of this number. The NYTimes has an article today (with some very interesting comments), focused on the shutdown of the subway system.
Hindsight
Seth Borenstein (AP) has some interesting interviews:
Private meteorologist Ryan Maue of Weather Bell Analytics slammed the public agency for ratcheting up forecast storm amounts before the system arrived, instead of telling people how uncertain it was.
“The public should be upset that the forecast was blown for NYC and ask for answers,” he said in an email.
Uccellini (NWS) said the agency would review those procedures and consult with social scientists to improve messaging.
But Uccellini said he’d rather warn too much and be wrong, than not warn enough. He said the weather service’s predictions and citywide closures that they prompted made for a faster recovery.
“This was the right forecast decision to make,” Uccellini said.
New Jersey Gov. Chris Christie defended his decision to ban travel on all state roads.
“I was being told as late as 9 o’clock last night that we were looking at 20-inch accumulations in most of New Jersey. If, in fact, that is what would have happened, having these types of things in effect were absolutely the right decision to make,” Christie told WABC-TV on Tuesday. “We were acting based on what we were being told.”
New York Gov. Andrew Cuomo said he was criticized for under-reacting to the November mega storm in Buffalo, so he worked “on the theory of living learned and a little wiser.”
Irwin Redlener, the director of Columbia University’s National Center for Disaster Preparedness and an unpaid adviser to New York Mayor Bill de Blasio, said Tuesday that the way the region came to a halt ahead of the storm was good practice.
“It’s not whether the city should have prepared so much, it’s how people respond,” Redlener said. “We don’t want the population to get so cynical that they’re not heeding the warnings.”
A National Weather Service forecaster who was called a hero of 2012’s Superstorm Sandy tweeted an apology for the errant forecast.
“You made a lot of tough decisions expecting us to get it right, and we didn’t. Once again, I’m sorry,” wrote Gary Szatkowski, a National Weather Service forecaster in Mount Holly, New Jersey.
Robinson, Shepherd and others said meteorologists probably erred more in the way they relayed the forecast to the public than the prediction itself.
Ryan Maue of WeatherBell has a good assessment, excerpts:
The NWS produces a deterministic forecast with a certain range e.g. 24-36” of snowfall but information about confidence is not imparted. Were they expecting 3-feet more than 2-feet or were the values inside that range equally likely? What was the chance of 4-feet or 1-foot? Weather forecasters deal with this question every day when they issue a rain-fall forecast and call for 30% chance of rain or 100% chance of rain. That’s more easily understood by the public as they can take action based on their perceived risk of the event. An umbrella would be a good fashion accessory with 90% chance of thunderstorms at rush hour.
This indeed goes back to Hurricane Sandy which was the “worst case scenario”. Our weather models were all very confidently saying that all hell would break loose. The same NWS forecasters at Mount Holly were warning at the top of the lungs to take immediate action to save lives and property. In this respect, the public was appropriately warned and should have definitely feared the effects of this extraordinary hurricane.
The same level of media hype and perceived government overreaction surrounding this blizzard in NYC may cause future complacency with warnings. However, the weather forecasters and authorities will not hesitate to issue the same emergency declarations because a False Alarm in the “safe direction” is hardly the worst possible outcome.
While private weather forecasters including the Weather Channel were skeptical of the enormous snowfall forecasts for NYC, the risk of publicly challenging NWS and emergency management decisions would have provided “mixed messages” that could have led to the worst possible outcome: public not heeding legitimate and lawful warnings.
JC reflections
The weather forecast models generally performed adequately for this storm. The difference between a huge impact for NYC and what actually happened was a difference of about 25 km in the storm track, which is not a level of accuracy that you can expect from a weather forecast model.
There is the obvious issue of how the forecast was communicated by the NWS to the public. But there is a more fundamental issue, one that I haven’t seen mentioned by anyone, but at least hinted at by Ryan Maue, is the actual interpretation of the forecasts from the models.
The NWS presents its forecast as a deterministic forecast (either from a single run of the model, or the ensemble mean), with a ‘range’ or cone of uncertainty, that may be derived from the ensemble or from historical errors. The real failure IMO was to appropriately interpret the information contained in the forecast ensemble. If you are unfamiliar with ensemble weather forecast systems, see my previous posts How should we interpret an ensemble of models? Part I: Weather Models. Interpretation of the ensemble, including successive forecasts, helps assess uncertainty and confidence in the forecast. But more sophisticated interpretations of the ensembles can be done, such as clustering to assess the probability of different scenarios and threshold exceedences.
My company Climate Forecast Applications Network (CFAN), is at the forefront of applying sophisticated ensemble interpretation methods to predicting extreme weather events. People have asked ‘how did we do predicting snowfall for NYC?’ Well, we don’t predict snowfall, other than snowpack in the U.S. Northwest (Columbia River Basin). Quantitative snowfall is probably the most difficult parameter to forecast, and the only parameter you can get wrong by an order of magnitude.
The other issue that is out there is ‘how to deal with a blown forecast? Ryan Maue tweets:
Not sure that filibustering approach by @NWS head is best PR way to instill confidence in future forecasts. Met community needs to do some outreach here — not overt prostration but clear, coherent explanation of what went wrong & honest truth that blown forecasts will happen again (false alarm). Throwing money won’t help — need introspection & focus + better messenging.
Terse, but makes some good points. I will relate to you some experiences that my company has had with blown forecasts, both related to hurricanes in the Gulf of Mexico. Hurricane Humberto in (2007) spun up out of nowhere, and we completely missed its formation. It formed right in the main oil production region of the Gulf of Mexico, with high impact to our client. We did a post mortem; as scientists we didn’t feel bad about missing this since it was highly unpredictable. However, for our customer, this was a big deal. We had to explain why it had happened, figure out some way to improve our forecast of such storms in future (with appropriate expressions of uncertainty). Subsequently we haven’t missed the forecast for a single such storm, although our forecasts have been low probability and low confidence owing to the unpredictability of such storms.
Learn from your mistakes, understand uncertainty, and be prepared to state when you have low confidence in your prediction, or when there are two equally probable scenarios. The NWS needs to get away from making deterministic forecasts, but then the public (or at least government officials) need to be educated in interpreting ensemble forecasts. But ensemble forecasts can be far more useful for risk management.
Did the government decision makers respond appropriately to the forecasts they received? I would say yes. There is a hysteresis effect related to recent events (e.g. Hurricane Sandy, Buffalo snow storm). If there are too many false alarms, you run into the ‘boy who cried wolf’ syndrome.
And finally, I spotted this on twitter:
Should meteorologists be accountable for hyping so-called extreme #weather in ways that impact millions of lives?
I agree this is a ridiculous statement, the weather forecasters are doing the best job that they can (even if their bosses in NOAA are playing politics). But rephrase the concern to ‘climate scientists’, and then we have to wonder whether all the hyping by Trenberth, Mann et al. influenced Governor Cuomo into overreacting?
Precautions don’t come without a price, but not heeding warnings could be associated with a bigger price. With weather events, you have the opportunity to practice; even false alarms can be useful in this regard (and the expense of the precautions isn’t that overwhelming). Communities get better with practice; Florida is now a lean mean machine when it comes to responding to hurricanes (but there haven’t been any landfalls since 2005!) New York City seems to be in fighting form to reduce its vulnerability to future weather disasters.
The weather forecasting enterprise needs to get its act together in terms of better interpretation of the various models available (the UK Met Office weather forecast model is currently getting a lot of attention in the private sector weather forecasting community). Private sector forecasting continues to do better in many instances than the NWS forecasters. In the U.S. anyways, the lion’s share of the govt $$ goes to climate modeling and model interpretation and impact assessment, not to weather forecast model interpretation.
Filed under: Communication, Extreme events