Ensemble forecasts: reading a probability forecast

A forecast says "heavy rain on Thursday", and Thursday is dry. Another says "30 % chance of heavy rain", and the river floods anyway. Neither forecast was necessarily wrong. The first claimed a certainty that the atmosphere never offers, and the second was a fair statement of uncertainty that was then misread. Knowing what an ensemble forecast is helps with both. It is a set of many forecasts, started from slightly different conditions, whose agreement and disagreement tell us how far to trust the weather ahead. This article explains the idea, shows how a probability is counted from the members, and gives practical rules for reading one.

What is an ensemble forecast?

An ensemble forecast is a group of weather forecasts for the same time and place, each made with a slightly different starting state or a slightly different version of the model. The members are called ensemble members. The range of outcomes they produce is an estimate of the uncertainty in the forecast.

The reason is chaos. Weather models start from a picture of the atmosphere built from observations, and those observations have errors. The atmosphere is chaotic: ECMWF's account of its own ensemble notes that "due to the chaotic nature of the atmosphere, even small initial errors can grow very rapidly" [1]. A single run, called a deterministic forecast, gives one answer and hides this. ECMWF's fact sheet puts it plainly: single deterministic forecasts "can be misleading as they fail to provide this information" [2]. The WMO guidelines add that uncertainty varies widely from day to day, and that an ensemble provides an estimate of that day-to-day uncertainty [3].

How does ensemble forecasting work?

Ensemble forecasting runs the same model many times, with deliberate small changes, and treats the results as a sample of what could happen. There are two sources of change.

• Initial conditions. Each member starts from a slightly different state, within the range allowed by observation errors. ECMWF originally used singular vectors, the perturbations that grow fastest, and added an ensemble of data assimilations in 2008 to represent observation errors [1].
• The model itself. Models are imperfect, so stochastic (random) schemes nudge the model physics differently in each member. ECMWF lists schemes such as stochastic perturbation of tendencies and stochastic kinetic energy backscatter [1].

The best known ensembles are large. ECMWF's ensemble, known as ENS, has 51 members: one control run made with the best available data and unperturbed models, and 50 perturbed ones [2][6]. ECMWF began producing ensemble forecasts in real time on 19 December 1992 [1], and its medium-range ensemble was raised from 18 km to 9 km grid spacing in the model upgrade of 27 June 2023 [4].

India has its own. The National Centre for Medium Range Weather Forecasting (NCMRWF) runs NEPS-G, a global ensemble at 12 km grid spacing with 22 perturbed members plus a control, forecasting 10 days ahead (the configuration described in NCMRWF's 2022 training material). It perturbs the starting state with an ensemble transform Kalman filter and the model physics with stochastic kinetic energy backscatter and random-parameter schemes [5].

How do you turn 51 forecasts into a probability?

You count. A probability from an ensemble is usually the fraction of members that predict the event. The NCMRWF training material states it directly: "Probability of an event = Fraction of the total number of ensemble members predicting the event" [5]. The WMO guidelines describe the same thing, a simple proportion of members, and add a caution: it is not a true statistical probability, but "a useful estimate for practical purposes", and it assumes the model reflects the real distribution of events [3].

Worked example (illustrative). Suppose 51 members give the following 24-hour rainfall at a catchment point:

The chance of more than 50 mm is the members above 50 mm divided by all members:

P(rain > 50 mm) = (12 + 5) / 51 = 17 / 51 = 0.33, about 33 %
P(rain > 100 mm) = 5 / 51 = 0.10, about 10 %

Each member is worth about 2 percentage points (1/51). That is why ensemble probabilities move in small steps, and why a 2 % or 4 % chance from a single ensemble should not be over-read: it is one or two members.

What does "30 % chance" actually mean?

It means that among all the occasions on which a reliable system says 30 %, the event happens on about 30 % of them. ECMWF's example is the same idea at 70 %: a forecast is reliable if "in 70% of cases when such a forecast is made temperatures will indeed rise above that threshold" [2]. It does not mean that 30 % of the area will be rained on, or that the rain will be 30 % as heavy.

Two things follow.

• The event must be defined. The WMO guidelines say it is very important to state what the probability is for: the threshold (for example more than 50 mm of rain), the time period, and the location [3]. "30 % chance of rain" and "30 % chance of more than 50 mm in 24 hours at this catchment" are different forecasts.
• A single outcome cannot test it. A 30 % forecast followed by a flood is not proof of error; one in three such days should bring the event. The WMO guidance says probability forecasts should be verified over large samples of cases, to see how forecast probabilities relate to observed frequencies [3].

How do you read spread?

Spread is the standard deviation of the members for a variable. It is a measure of how uncertain the forecast is [3]. When members cluster tightly, the forecast is sharp: the smaller the range of predicted outcomes, the sharper it is [2]. When they scatter, predictability is low.

The WMO guidelines give practical advice [3]:

• Small spread: it is reasonable to offer more detail. The ensemble mean or control is a good guide. But small spread in one variable does not guarantee confidence in all of them; good predictability of the large-scale pressure pattern does not always mean predictability of convective rain.
• Large spread: avoid giving too much detail. If the members split into distinct scenarios, the ensemble mean may not be a realistic scenario at all.

The ensemble mean (the average of the members) smooths out the unpredictable detail, so it can look "safe", but the WMO guidelines warn that it "must not be relied on its own, as it will rarely capture the risk of extreme events" [3]. A flood warning depends on the tail of the distribution, not the average.

Common ways to see an ensemble are probability maps, a meteogram (a chart at one place showing all members through time, so the spread is visible) and the ensemble mean with spread shading.

Are ensemble probabilities reliable?

Not automatically. The raw counting described above works best where the model resolves the weather well. In mountains, small catchments and convective storms, a model can be systematically biased or too confident, with members clustered but wrong.

The WMO guidelines say direct model output from ensembles "should be used with care, as it may not provide reliable probabilistic forecasts, but nevertheless will often provide valuable information". Calibration, a statistical correction using past forecasts and observations, usually improves reliability (forecast probabilities matching observed frequencies) but can reduce resolution (the ability to tell event days from non-event days). Calibration can also degrade the probabilities of rare extremes, because there are few past cases to train on [3].

This matters most for exactly the cases that matter in the Himalaya: short, intense rain over steep catchments. See flash floods vs river floods: warning time for why these leave so little margin.

Deciding with a probability

The point of a probability is to support a decision. The classic cost-loss rule says to act when the probability of the harmful event, multiplied by the loss it would cause, exceeds the cost of acting [7]. In practice, a low probability can justify a cheap action (check the gauges, alert the duty officer), and a high probability the costly one (evacuate a camp).

Illustrative example. Suppose the ensemble gives 33 % for more than 50 mm in 24 hours over a catchment. A district officer might:

1. Place the field team and gauge readings on a higher watch and check the stations are reporting.
2. Compare with the thresholds at the river gauge and with upstream stations (how a flood early warning system works).
3. Wait for the next forecast cycle. If more members move above 50 mm, the probability rises; if they drift away, it falls.

Probabilities from successive runs also show a trend. In an illustrative case, a chance that rises from 10 % to 33 % to 60 % over three runs carries more information than any single number.

Common mistakes

• Reading one member as "the forecast". A single member is one possible outcome, not a prediction.
• Treating 33 % as "unlikely, so ignore". Three such days in ten bring the event.
• Treating a probability as a rain amount. The chance of exceeding a threshold says nothing about how far above it.
• Ignoring the location and time window. A probability for a grid box is not a probability for a particular stream.
• Trusting small spread blindly. Members can agree and all be wrong if the model misses the process.

Key points

• An ensemble forecast is many runs of one model from slightly different starting states and physics, giving a range of possible outcomes.
• ECMWF's ENS has 51 members; India's NEPS-G has 22 perturbed members plus a control.
• A probability is normally the fraction of members that show the event, a practical estimate that needs to be checked against observed frequencies over many cases.
• Spread measures uncertainty; the ensemble mean alone hides extremes.
• Always ask: probability of what, where and when?

Where ClimMaTech fits

Forecast probabilities become useful when they are checked against what the catchment is doing. At ClimMaTech we build the measurement side of that loop: river level and flow, and warnings that reach people. Our flood warning solution describes how live observations, thresholds and alerts work together with forecast information from the national agencies.

References

1. ECMWF, 25 years of ensemble forecasting at ECMWF, ECMWF Newsletter No. 153, Autumn 2017. ecmwf.int
2. ECMWF, Fact sheet: Ensemble weather forecasting, 2017. ecmwf.int
3. World Meteorological Organization, Guidelines on Ensemble Prediction Systems and Forecasting, WMO-No. 1091, 2012. wmo.int
4. ECMWF, IFS upgrade brings many improvements and unifies medium-range resolutions, ECMWF Newsletter No. 176, Summer 2023. ecmwf.int
5. NCMRWF, Ensemble prediction systems for tropical cyclone prediction (NEPS-G configuration), WMO Tropical Cyclone training, New Delhi, 2022. severeweather.wmo.int
6. ECMWF, Medium-range forecasts (description of the ENS: 51 forecasts, one control and 50 perturbed). ecmwf.int
7. A. H. Murphy, The value of climatological, categorical and probabilistic forecasts in the cost-loss ratio situation, Monthly Weather Review 105, 803-816, 1977. doi:10.1175/1520-0493(1977)105<0803:TVOCCA>2.0.CO;2105%3C0803:TVOCCA%3E2.0.CO;2)