Placebo tests for panel causal designs

causal inference
ecology tutorial
R
With one treated unit, the usual standard errors cannot be trusted. Permute the treatment across space instead; move it back in time to check parallel trends.
Author

Tidy Ecology

Published

2026-05-27

Modified

2026-10-14

Corrected 14 October 2026: the opening and the description presented both placebo tests as the answer to untrustworthy standard errors; they now say that the placebo across space takes their place and the placebo in time checks parallel trends, and the opening adds that the latter is itself judged by the model-based standard error.

Corrected 1 October 2026: the synthetic control weights came from a Nelder-Mead search that stopped short of the best convex pre-period fit for several units, which changed some placebo error ratios, nudged the reported ratio, placebo median and estimated effect, and let the median differ between machines; they now come from an exact quadratic programme, and the conclusion is unchanged. The opening paragraph also tied the many-groups assumption to the model-based standard errors; it belongs to the clustered ones, while the model-based ones ignore correlation within units.

The three earlier posts in this series estimated effects; this one asks whether those estimates could have arisen by chance. That question is awkward for panel designs with very few treated units. The synthetic control post had a single treated region, so there is no sample of treated units to form a sampling distribution. Difference-in-differences with a small number of clusters has a related problem: the model-based standard errors ignore correlation within units (Bertrand et al. 2004), and the clustered standard errors that allow for it assume many independent groups and can be far too small when there are few (Conley and Taber 2011; Ferman and Pinto 2019). For synthetic control the honest move is a placebo test (Abadie et al. 2010; Abadie 2021): re-run the whole analysis under conditions where the effect is known to be absent, and see whether the real estimate stands out against that reference set. The difference-in-differences placebo below does a narrower job. It moves the adoption date into the pre-period to check the parallel-trends assumption, and its single estimate is judged by the model-based standard error, which is sound here because the simulated noise is independent but would have the weakness above if the errors were correlated within units.

Placebo-in-space: permuting the treatment across units

For synthetic control, the reference set is built by pretending each donor was the treated unit. Apply the identical procedure to every donor, using the remaining donors as its pool, and record how large a post-intervention gap each produces relative to its own pre-intervention fit. If the real treated unit’s gap is unremarkable against this collection of placebo gaps, the effect is not distinguishable from noise.

The natural test statistic is the ratio of post-intervention to pre-intervention root mean squared prediction error. Dividing by the pre-period error is what makes the comparison fair: a donor that never fitted well in the pre-period has a large denominator, so a large post-period gap does not by itself make it rank highly.

library(ggplot2); library(dplyr); library(tidyr)
set.seed(173)
J <- 19; Tn <- 20; T0 <- 13; tvec <- 1:Tn        # 19 donors + 1 treated; smallest p is 1/20
F1 <- scale(cumsum(rnorm(Tn, 0.04, 0.6)))[, 1]; F2 <- sin(2 * pi * tvec / 9); trend <- (tvec - mean(tvec)) / Tn
Fmat <- cbind(F1, F2, trend)
Ld <- cbind(rnorm(J, 0.6, 0.7), rnorm(J, 0.4, 0.5), rnorm(J, 0.3, 0.9)); mud <- runif(J, 3.5, 5.5)
donors <- sapply(1:J, function(j) mud[j] + Fmat %*% Ld[j, ] + rnorm(Tn, 0, 0.18)); colnames(donors) <- paste0("D", 1:J)
wtrue <- c(0.45, 0.35, 0.20); mix <- c(2, 5, 9)
Lt <- colSums(wtrue * Ld[mix, ]); mut <- sum(wtrue * mud[mix])
treated <- as.numeric(mut + Fmat %*% Lt + rnorm(Tn, 0, 0.18)) + ifelse(tvec >= T0, 0.32 * (tvec - T0 + 1), 0)

pre <- 1:(T0 - 1); post <- T0:Tn
fit_sc <- function(y, pool) {                      # convex pre-period fit of y on pool columns
  Ppre <- pool[pre, , drop = FALSE]; m <- ncol(pool)
  D <- crossprod(Ppre) + diag(1e-8, m)             # tiny ridge: more donors than pre-period years
  A <- cbind(1, diag(m)); b <- c(1, rep(0, m))     # weights sum to one and are non-negative
  w <- quadprog::solve.QP(D, crossprod(Ppre, y[pre]), A, b, meq = 1)$solution   # exact quadratic programme
  w[w < 1e-6] <- 0; as.numeric(pool %*% (w / sum(w)))
}
rat <- function(y, s) sqrt(mean((y[post] - s[post])^2)) / sqrt(mean((y[pre] - s[pre])^2))
synth_t <- fit_sc(treated, donors); gap_t <- treated - synth_t; ratio_t <- rat(treated, synth_t)
plac <- lapply(1:J, function(j) {                  # each donor treated in turn, pool = the others
  s <- fit_sc(donors[, j], donors[, -j, drop = FALSE])
  list(gap = donors[, j] - s, ratio = rat(donors[, j], s))
})
ratios <- c(ratio_t, sapply(plac, function(x) x$ratio))
pval <- mean(ratios >= ratio_t)                    # rank of treated among J+1 units
att_t <- mean(gap_t[post])

The treated region’s error ratio is 11.1, against a placebo median of 1.6. Its rank among the 20 units gives a permutation p-value of 0.050. With 19 donors the smallest attainable p-value is 0.050, so the size of the donor pool bounds what this test can say. The estimated effect over the post-intervention years is 1.25.

te <- list(ink="#16241d",body="#2c3a31",forest="#275139",label="#46604a",sage="#93a87f",paper="#f5f4ee",
           line="#dad9ca",faint="#5d6b61",grazed="#cda23f",mown="#2f8f63",aband="#b5534e")
th <- theme_minimal(base_size = 12) + theme(text = element_text(colour = te$body),
  plot.title = element_text(colour = te$ink, face = "bold"), plot.subtitle = element_text(colour = te$faint, size = rel(0.9)),
  axis.title = element_text(colour = te$label), axis.text = element_text(colour = te$faint), panel.grid.minor = element_blank(),
  panel.grid.major = element_line(colour = te$line, linewidth = 0.3), legend.position = "bottom", legend.title = element_blank(),
  legend.text = element_text(colour = te$faint, size = rel(0.8)), strip.text = element_text(colour = te$ink, face = "bold"),
  plot.background = element_rect(fill = te$paper, colour = NA), panel.background = element_rect(fill = te$paper, colour = NA))
plac_gaps <- do.call(rbind, lapply(1:J, function(j) data.frame(t = tvec, gap = plac[[j]]$gap, who = paste0("D", j))))
tr_gap <- data.frame(t = tvec, gap = gap_t)
ggplot() +
  annotate("rect", xmin = T0 - 0.5, xmax = Tn + 0.5, ymin = -Inf, ymax = Inf, fill = te$sage, alpha = 0.12) +
  geom_hline(yintercept = 0, colour = te$faint, linewidth = 0.4) +
  geom_vline(xintercept = T0 - 0.5, colour = te$faint, linetype = "22", linewidth = 0.5) +
  geom_line(data = plac_gaps, aes(t, gap, group = who), colour = te$line, linewidth = 0.4) +
  geom_line(data = tr_gap, aes(t, gap), colour = te$aband, linewidth = 1.1) +
  geom_point(data = tr_gap, aes(t, gap), colour = te$aband, size = 1.2) +
  annotate("text", x = 1, y = max(tr_gap$gap) * 0.9, hjust = 0, colour = te$aband, size = 3.3,
           label = sprintf("treated region\nRMSPE ratio = %.1f\npermutation p = %.3f", ratio_t, pval)) +
  labs(x = "Year", y = "Gap: unit minus its synthetic control",
       title = "Placebo-in-space: the treated gap stands out from the donor placebos",
       subtitle = "Grey: each donor treated as a placebo. The treated region (red) departs only after the intervention.") + th
Gap-versus-year plot. Nineteen grey placebo lines scatter around zero, a few of them widely; one bold red line for the treated region tracks them before the year-13 marker then rises steeply after it. An annotation gives the treated region's error ratio and its permutation p-value.
Figure 1: Each grey line is a donor’s gap when it is treated as a placebo; the bold red line is the real treated region. The treated gap sits among the placebos before the intervention and separates sharply afterwards.

Placebo-in-time: moving the treatment into the past

The second placebo keeps the treated and control groups fixed but shifts the intervention date. Restrict the data to periods before the real intervention, declare a fake adoption partway through that pre-period, and estimate the difference-in-differences. Nothing happened at the fake date, so under parallel trends there is no effect for the placebo to find. If it comes out large anyway, the two groups were already drifting apart, and the design is in trouble regardless of what the real estimate says.

set.seed(2718)
Ndid <- 40; Ty <- 14; Tt <- 9; trt <- 1:20        # real adoption in year 9
al <- rnorm(Ndid, 0, 0.7); gt <- cumsum(c(0, rnorm(Ty - 1, 0.04, 0.12)))
mkp <- function(ptr) {
  d <- expand.grid(id = 1:Ndid, year = 1:Ty); d$treated <- as.integer(d$id %in% trt)
  eff <- ifelse(d$treated == 1 & d$year >= Tt, 0.9, 0)
  tr <- ifelse(d$treated == 1, ptr * (d$year - Tt), 0)
  d$y <- al[d$id] + gt[d$year] + eff + tr + rnorm(nrow(d), 0, 0.35); d
}
did_est <- function(d) {
  co <- summary(lm(y ~ factor(id) + factor(year) + post:treated, d))$coefficients["post:treated", ]
  c(b = unname(co[1]), se = unname(co[2]), p = unname(co[4]))
}
Tp <- 5                                            # fake adoption in year 5
real_time <- function(d) { d$post <- as.integer(d$year >= Tt); did_est(d) }
placebo_time <- function(d) { dd <- subset(d, year < Tt); dd$post <- as.integer(dd$year >= Tp); did_est(dd) }

Run this on two datasets: one with parallel trends, one with a small differential trend of 0.15 index units per year in the treated group.

d_clean <- mkp(0); d_ptr <- mkp(0.15)
rc <- real_time(d_clean); pc <- placebo_time(d_clean)
rp <- real_time(d_ptr);   pp <- placebo_time(d_ptr)

Under parallel trends the real estimate is 1.00 (p < 0.001) and the placebo is 0.01 (p = 0.88). Under the pre-trend the real estimate inflates to 1.87, but now the placebo is 0.54 and significant (p < 0.001). The placebo has caught the drift using pre-intervention data alone, before the real estimate could be mistaken for an effect.

d2 <- data.frame(
  scenario = rep(c("Parallel trends hold", "Differential pre-trend"), each = 2),
  which = rep(c("real (year 9)", "placebo (year 5)"), 2),
  b = c(rc["b"], pc["b"], rp["b"], pp["b"]), se = c(rc["se"], pc["se"], rp["se"], pp["se"]),
  p = c(rc["p"], pc["p"], rp["p"], pp["p"]))
d2$scenario <- factor(d2$scenario, levels = c("Parallel trends hold", "Differential pre-trend"))
d2$which <- factor(d2$which, levels = c("placebo (year 5)", "real (year 9)"))
d2$lo <- d2$b - 1.96 * d2$se; d2$hi <- d2$b + 1.96 * d2$se
d2$lab <- ifelse(d2$p < 0.001, "p<0.001", sprintf("p=%.3f", d2$p))
ggplot(d2, aes(b, which, colour = which)) +
  geom_vline(xintercept = 0, colour = te$faint, linewidth = 0.4) +
  geom_errorbar(aes(xmin = lo, xmax = hi), orientation = "y", width = 0.16, linewidth = 0.5) +
  geom_point(size = 2.8) +
  geom_text(aes(label = lab), vjust = -1.1, hjust = 0, nudge_x = 0.04,
            size = 3, show.legend = FALSE) +
  facet_wrap(~scenario, ncol = 1) +
  scale_colour_manual(values = c("real (year 9)" = te$mown, "placebo (year 5)" = te$grazed)) +
  scale_x_continuous(expand = expansion(mult = c(0.05, 0.14))) +
  labs(x = "Difference-in-differences estimate", y = NULL,
       title = "Placebo-in-time: a fake pre-period adoption should show no effect",
       subtitle = "Parallel trends: the placebo should be null. Pre-trend: the placebo is significant.") +
  th + theme(legend.position = "none")
Two stacked panels of coefficient estimates with confidence intervals. Top panel, parallel trends: the real estimate near 1 is positive, the placebo is much closer to zero. Bottom panel, differential pre-trend: the real estimate near 1.9 is larger, and the placebo is also positive and significant.
Figure 2: Real and placebo difference-in-differences estimates under each scenario. Under the differential pre-trend the placebo is clearly non-zero, which flags the design as unreliable.

The limits of a placebo

Placebo inference is honest but blunt, and it should be read with three cautions. Its validity rests on exchangeability: the permutation logic assumes that under the null the treated unit is not systematically different from the donors, which fails if the treated region was selected precisely because it was unusual. The p-value is coarse when units are few, bounded below by one over the number of units, so a single treated region against nineteen donors cannot report anything smaller than 0.05 no matter how clean the separation. And a placebo can only refute, not confirm: a null placebo-in-time is consistent with parallel trends but does not prove it, exactly as the pre-trend test in the event-study post could screen for violations without guaranteeing their absence.

None of this makes the tests optional. For a single treated unit there is often no credible alternative to a permutation, and a placebo that moves the treatment in time is among the cheapest checks available for any difference-in-differences. Reporting them, and reporting them when they fail, is what separates a comparative case study that has been stress-tested from one that merely looks convincing (Larsen et al. 2019 treat causal analysis of control-impact studies in ecology).

References

Download the R codeReport a problem with this page

Newsletter

Get updates by email

An occasional email when tutorials are added or substantially corrected. No spam; unsubscribe anytime.

By subscribing you agree to receive these emails and confirm your address once. See the privacy policy.