Are we stating the facts? Tracing the origins of early warning statistics

Statistics can take on a life of their own, often being repackaged and rephrased so many times it is almost impossible to trace them back to their original source – and the associated nuances and caveats. The recent press release about the Early Warnings for All initiative, in which anticipatory action is a key component, included three statements for which I was keen to find the source material. 

“Early Warning Systems provide more than a tenfold return on investment” 

Various press releases for Early Warnings for All state that the source of this statement is the World Resources Institute’s 2019 Adapt Now report. However, that is a synthesis report and not the original source of the statistic. The sources cited for this statistic were background or technical papers for the main report that unfortunately can no longer be found; as these were also synthesis studies, the original source must be elsewhere. This was going to require some detective work! 

A similar statement was mentioned in a 2015 World Meteorological Organization (WMO) report, attributing it to an opening statement on the social and economic benefits of weather and climate services made by M. Jarraud, former WMO secretary general, at a conference held in Madrid, Spain, in 2007. The broader statement actually relates this cost–benefit to the wider economic benefit of national hydrometeorological services, with a previous part of the statement suggesting that for disaster risk management purposes, the cost–benefit ratio is around 1 in 7.

In the 2015 WMO report, the statement is said to be broadly in line with findings from various studies, with a range for the cost–benefit given as between 1 in 2 and 1 in 36. However, this is across all weather-sensitive sectors, including agriculture, energy and transport, and in all countries. There is no specific evidence cited for the value of early warning systems specifically, or for the disaster risk management sector.  

A 2012 World Bank study on the value of developing hydrometeorological services for early warnings in developing countries does estimate the value of improving early warning systems to European standards in the developing world. This estimation is based on the assumption that these improvements would lead to a reduction in the mortality rate for weather-related events, from the current level in developing countries to the rate in Europe. This approach gives an estimated saving of 23,000 lives per year, with a cost–benefit for disaster risk management at between 1 in 1 and 1 in 5.5. This also assumes that hazard and vulnerability profiles between developing and developed countries are similar. Care should also be taken to assume that these benefits can be achieved without broader investment in disaster risk management; for example, the benefits of the cyclone early warning in Bangladesh required parallel investment in public cyclone shelters. 

Verdict: Tracing back through these studies, there is limited evidence for a tenfold return on investment (e.g., for disaster risk management). Most studies address the overall value of hydrometeorological services for a broader set of weather-sensitive sectors.

---

“Just 24 hours’ notice of an impending hazardous event can cut the ensuing damage by 30 per cent” 

I scratched my head over this statement for some time, as I couldn’t find a single mention of these values anywhere. But then I found a compilation of similar statistics in the same World Bank study from 2012, alongside which there was a figure sourced from a technical note on a flood-warning benefit evaluation for the Susquehanna River, written by Harold Day in 1970; the figures of 30 per cent at 24 hours appear to have been read from here. Obviously, this statistic should be used with caution, as it is based solely on estimates for reduction in flood damage in one river basin in the USA, and from 50 years ago.

Verdict: Don’t use this statistic: it is specific only to floods in one river basin in the USA, and it is also ancient.

---

"The Global Commission on Adaptation found that spending just US$800 million on such systems in developing countries would avoid losses of $3 to 16 billion per year." 

First of all, the Global Commission on Adaptation was a synthesis report, so this was not a result which they themselves ‘found’. The figure of 800 million US dollars seemed quite precise, so this was where I began my detective work, and the World Bank’s 2012 report seemed like a good place to start. It mentions a figure of 800 million US dollars, but as the annual investment required for 5 years to support 80 countries. Using the lower cost–benefit ratio from this report (1 in 4), we arrive at that lower estimate for the avoided losses (3.2 billion US dollars). However, if this is the true source of the information, we need to take into account that these figures are now over ten years old, as well as being an annual figure for a requirement for long-term investment. 

Verdict: As well as being ten years old, the figure of 800 million US dollars is the annual figure, rather than a one-off investment. This needs to be updated, using the cost estimates from the Early Warning for All initiative, and the range estimated for the cost–benefit at least based on the (lower) estimates from the 2012 World Bank Study.

---

Recommendations

The use of compelling statistics to make an argument is not unique to the development of early warning systems, and anticipatory action is certainly not immune to making bold claims using unreferenced or generalized statistics. There are, though, several steps you can take to support a better use of evidence-based information. 

1. Cite the original source 

Everyone loves to have a compelling statistic to make an argument and, usually, people accept that these statistics are broad-sweeping generalizations. However, if you are using a statistic from elsewhere, you must provide the original source (not from a synthesis report). This shows your audience that you are (hopefully) aware of the context and caveats of that original estimation. It also helps that audience to make their own informed judgement as to whether they can apply it to their context. 

2. Don’t use a statistic out of context  

Statistics from specific contexts are, unfortunately, often used to make generalized statements. However, if the original statistic is about flood damage, it can’t apply to heat waves; likewise, cost–benefits calculated based on the benefit to the economy can’t be used to justify reductions in loss of life. The design and success of anticipatory action is specific to local contexts, and using a statistic out of context could lead to resources being allocated to activities that don’t provide the most benefits to communities.  

For example, investing in long-term risk-reduction measures could save more lives than allocating resources to improved early warnings, or vice versa. Similarly, investments in national hydrometeorological services might improve early warnings for flash floods in one country, but in another country, investing in community-based early warning systems might be more effective. 

3. If undertaking your own estimates, acknowledge your assumptions 

If you are estimating the benefits of different interventions, make sure you acknowledge the assumptions you are making. For example, cost–benefit analysis is highly sensitive to how a human life is valued. Acknowledging your assumptions makes it easier for others to decide whether your evidence can be used to support their own investments, as well as contributing to a more robust evidence base for the anticipatory action community.