Checking a forecast evaluation

Three checks for ecological forecast evaluation in R: coverage rewards width, sharpness needs calibration, and small test sets may not separate two forecasts.

Corrected 15 October 2026: the third check said that a ranking of two forecasts on a dozen years is not evidence of a real difference and that below about a hundred test points the difference cannot be told from zero; it now says that such a ranking is weak evidence when the gap is as small as this one, that how many test points are needed depends on the gap, and that the figure shows one simulated test set at each size, and its titles say that small test sets may not separate two forecasts.

The three earlier posts built the machinery: proper scores, the CRPS and Brier score, and skill against a baseline. This one is the checking step, the same closing move as every other cluster on this blog. A forecast evaluation can produce a clean headline number and still mislead, in three specific ways. Each has a diagnostic, and each diagnostic is a couple of lines of base R.

suppressMessages(library(ggplot2))
te_canvas <- theme(plot.background  = element_rect(fill = "#f5f4ee", colour = NA),
                   panel.background = element_rect(fill = "#f5f4ee", colour = NA))
col_main <- "#275139"; col_acc <- "#B5651D"; col_mut <- "#8C8C8C"
base_th <- theme_minimal(base_size = 12) + te_canvas +
  theme(panel.grid.minor = element_blank(),
        plot.title = element_text(face = "bold", size = 13),
        axis.title = element_text(size = 11))
crps_norm <- function(mu, sigma, y) { z <- (y - mu) / sigma
  sigma * (z * (2 * pnorm(z) - 1) + 2 * dnorm(z) - 1 / sqrt(pi)) }

Check 1: coverage on its own rewards a wide forecast

A tempting way to grade a forecast is coverage: what fraction of outcomes fell inside the stated interval? It feels intuitive, and it is a trap, because it is improper. Rank forecasters by “more coverage is better” and the winner is whoever quotes the widest interval. A forecast that reports a spread four times too large covers almost everything, and by the coverage yardstick it beats an honest one.

set.seed(909)
mu_t <- 3; sig0 <- 1
y <- rnorm(5000, mu_t, sig0)
cover90 <- function(mu, s, y) { lo <- qnorm(0.05, mu, s); hi <- qnorm(0.95, mu, s)
  mean(y >= lo & y <= hi) }

honest <- c(coverage = cover90(mu_t, sig0, y),     crps = mean(crps_norm(mu_t, sig0, y)))
wide   <- c(coverage = cover90(mu_t, 4 * sig0, y), crps = mean(crps_norm(mu_t, 4 * sig0, y)))
rbind(honest, wide)

       coverage      crps
honest   0.9042 0.5614321
wide     1.0000 1.0321118

The wide forecast covers 1.000 of outcomes against the honest forecast’s 0.904, so by coverage alone it “wins”. But its CRPS is 1.84 times worse. Coverage is a calibration check, not a ranking rule: the target is coverage equal to the nominal level, not coverage as high as possible. A proper score penalises the wasted width that pure coverage rewards.

Figure 1: Ranking by coverage prefers the wide forecast, but its CRPS is far worse. Coverage checks calibration; it does not rank forecasts.

Check 2: sharpness is a virtue only after calibration

The opposite failure is prizing sharpness: narrow intervals look confident and precise. But a sharp forecast that is wrong is overconfident, and the sign of that is a probability integral transform (PIT) that is not flat. The PIT is the forecast CDF evaluated at each outcome; for a calibrated forecast it is uniform, and for an overconfident one it piles up in the tails, because outcomes keep landing outside the too-narrow intervals.

set.seed(919)
y2 <- rnorm(5000, mu_t, sig0)
width90 <- function(s) qnorm(0.95, 0, s) - qnorm(0.05, 0, s)
pit <- function(mu, s, y) pnorm(y, mu, s)

honest_s <- c(width = width90(sig0),       crps = mean(crps_norm(mu_t, sig0, y2)),
              tail = mean(pit(mu_t, sig0, y2) < 0.05 | pit(mu_t, sig0, y2) > 0.95))
sharp_s  <- c(width = width90(0.5 * sig0), crps = mean(crps_norm(mu_t, 0.5 * sig0, y2)),
              tail = mean(pit(mu_t, 0.5 * sig0, y2) < 0.05 | pit(mu_t, 0.5 * sig0, y2) > 0.95))
rbind(honest_s, sharp_s)

            width      crps   tail
honest_s 3.289707 0.5625069 0.0996
sharp_s  1.644854 0.6087213 0.4122

The overconfident forecast is sharper: its 90 per cent interval is 1.64 wide against 3.29. If you chased sharpness you would prefer it. But 41 per cent of outcomes fall in its tails instead of the nominal 10, and its CRPS is worse (0.609 against 0.563). The discipline, from Gneiting et al. (2007), is to maximise sharpness subject to calibration: a sharp forecast is better only once the PIT confirms it is honest. A single score cannot tell you which side of that line you are on, which is why the score and the calibration plot are reported together.

Figure 2: PIT histograms. Flat means calibrated; the U-shape of the overconfident forecast shows outcomes piling up in the tails.

Check 3: a small test set may not separate two forecasts

Ecological forecasts are often evaluated on a handful of years or sites. A difference in mean score between two models is itself an estimate, with sampling uncertainty, and on a short test set that uncertainty can swamp the difference. Here model A is the honest forecast and model B is slightly too wide; A is genuinely better. Bootstrapping the per-observation CRPS difference across the test points shows when the evidence is actually there (Diebold and Mariano 1995).

crps_diff_ci <- function(ntest, seed, B = 4000) {
  set.seed(seed)
  yy <- rnorm(ntest, mu_t, sig0)
  d <- crps_norm(mu_t, 1.4 * sig0, yy) - crps_norm(mu_t, sig0, yy)   # B minus A, positive = A better
  bs <- replicate(B, mean(sample(d, replace = TRUE)))
  c(mean = mean(d), lo = quantile(bs, 0.025), hi = quantile(bs, 0.975))
}
t(sapply(c(10, 30, 100, 300), function(nt) crps_diff_ci(nt, seed = 1000 + nt)))

             mean     lo.2.5%   hi.97.5%
[1,] -0.005584860 -0.06403161 0.04713802
[2,]  0.007761495 -0.02474203 0.03743723
[3,]  0.025762271  0.01115841 0.04012961
[4,]  0.023175917  0.01465637 0.03140212

At 10 test points the interval spans zero and the point estimate even comes out the wrong sign; at 30 it still spans zero; only at 100 and beyond does the interval clear zero and confirm that A beats B. Each size is a single simulated test set, so another draw could move the size at which the interval clears zero. How many test points a comparison needs depends on how large the expected score gap is against the spread of the per-observation differences, and with a gap as small as the one between A and B, a ranking of two forecasts on a dozen years is weak evidence of a real difference. The score can favour one model on a small sample purely by chance, so the interval, not the point estimate, is what settles it.

Figure 3: Bootstrap intervals for the CRPS difference between two forecasts, one simulated test set at each size. In this run the intervals for the two smaller test sets include zero; another draw could move the size at which they clear zero.

Where this leaves the series

Put the four posts together and the workflow is complete. Forecast a distribution, not a point. Grade it with a proper score: the Brier score for presence, the CRPS for abundance. Turn the raw score into skill against a baseline that is genuinely hard, and report the baseline. Then check the evaluation itself: coverage is a calibration test and not a ranking, sharpness only counts once the PIT is flat, and a score difference on a short record needs an interval before you believe it. None of this makes a bad model good, but it stops a good score from hiding a bad forecast.

References

Gneiting T, Balabdaoui F, Raftery AE 2007. Journal of the Royal Statistical Society B 69(2):243-268 (10.1111/j.1467-9868.2007.00587.x)

Broecker J, Smith LA 2007. Weather and Forecasting 22(2):382-388 (10.1175/WAF966.1)

Diebold FX, Mariano RS 1995. Journal of Business and Economic Statistics 13(3):253-263 (10.1080/07350015.1995.10524599)

Dietze MC, Fox A, Beck-Johnson LM, et al. 2018. Proceedings of the National Academy of Sciences 115(7):1424-1432 (10.1073/pnas.1710231115)

Related tutorials

• Forecast skill and the baseline
• The CRPS and the Brier score
• Scoring ecological forecasts
• Checking predictive calibration

Reproducibility

• Code last run: 2026-10-15
• R version 4.5.3 (2026-03-11), aarch64-apple-darwin20
• Packages: ggplot2 4.0.3