Summarising ecological data by group

Grouped summaries in R with dplyr: group_by and summarise, and why the mean of per-site proportions is not the pooled proportion when sampling effort varies.

Step 8 of the Tidy Ecology course, Part two: a first result, and what it is worth.

Updated 27 September 2026: a new section, When summarise() keeps a grouping, shows how the grouping that summarise() leaves behind turns the next summary on this survey into one row per habitat, and three ways to drop it.

Once your field data is a tidy table, most questions become grouped summaries: mean abundance per habitat, species richness per site, the proportion of a focal species per plot. R makes this a one-liner with dplyr, and that is exactly why it is dangerous. The one-liner hides a choice, and the wrong choice gives you a number that answers a different question than the one you asked.

This post covers group_by() and summarise(), then spends most of its time on one measurable trap that catches ecologists constantly: the mean of per-site ratios is not the same as the ratio computed on the pooled totals, and when sampling effort differs between sites the two can differ by nearly a factor of two.

group_by() then summarise()

The pattern is always the same. You split the rows into groups, then collapse each group to one number. Here is a focal-species survey across six quadrats in two habitats, where each quadrat has a total arthropod count (the sampling effort varies) and a count of the focal beetle:

library(dplyr)
surv <- data.frame(
  quadrat = c("Q1", "Q2", "Q3", "Q4", "Q5", "Q6"),
  habitat = c("edge", "edge", "interior", "interior", "interior", "edge"),
  total   = c(200,  180,   40,   30,   25,  220),   # all arthropods counted
  focal   = c( 20,   27,   20,   18,   10,   22)    # the focal beetle
)
surv

  quadrat  habitat total focal
1      Q1     edge   200    20
2      Q2     edge   180    27
3      Q3 interior    40    20
4      Q4 interior    30    18
5      Q5 interior    25    10
6      Q6     edge   220    22

group_by() marks which rows belong together and summarise() reduces each group to one row. n() counts the rows in a group; sum() and mean() do the obvious. Collapsing by habitat:

surv |>
  group_by(habitat) |>
  summarise(quadrats = n(), individuals = sum(total), focal_total = sum(focal))

# A tibble: 2 × 4
  habitat  quadrats individuals focal_total
  <chr>       <int>       <dbl>       <dbl>
1 edge            3         600          69
2 interior        3          95          48

The same verbs work on an ungrouped table, which is what you want when the quadrat itself is the unit. To get the focal proportion in each quadrat, add a column with mutate():

surv <- surv |> mutate(prop = focal / total)
round(surv$prop, 3)

[1] 0.10 0.15 0.50 0.60 0.40 0.10

The six proportions run from 0.10 to 0.60. Now suppose you want a single headline number: what fraction of the community is the focal beetle? There are two ways to compute it, and they are not the same number.

The mean of ratios is not the ratio of means

summary_tbl <- surv |>
  summarise(
    pooled         = sum(focal) / sum(total),  # ratio of the pooled totals
    mean_of_ratios = mean(focal / total),      # average of the per-quadrat proportions
    n              = n()
  )
summary_tbl

     pooled mean_of_ratios n
1 0.1683453      0.3083333 6

The pooled proportion is 0.168: of the 695 individuals counted across all quadrats, 117 were the focal beetle. The mean of ratios is 0.308, the plain average of the six per-quadrat proportions. They differ by a factor of 1.83, and neither is a rounding artefact.

The reason is that the quadrats are not the same size and the focal proportion depends on size. The correlation between quadrat total and focal proportion is -0.95: the beetle makes up a large share of the small, sparse quadrats and a small share of the big ones. The pooled proportion is dominated by the big quadrats (they contribute most of the individuals), so it lands low. The mean of ratios gives every quadrat one equal vote regardless of size, so the small high-proportion quadrats pull it up.

Neither number is wrong. They answer different questions:

• The pooled proportion answers “across all the individuals I counted, what fraction is the focal species?” It is a population-level quantity, weighted by how many individuals each quadrat contributed.
• The mean of ratios answers “in a typical quadrat, what fraction is the focal species?” It weights quadrats equally, which is what you want if the quadrat is your unit of replication and their sizes are incidental.

The mistake is reporting one while describing the other: writing “the focal beetle is 31 per cent of the community” (the mean of ratios) when “the community” means the pooled individuals (17 per cent), or claiming “a typical quadrat is 17 per cent focal” when the per-quadrat proportions average 31 per cent.

The pooled proportion is a weighted mean

The two summaries are not unrelated. The pooled proportion is exactly the mean of the per-quadrat proportions weighted by quadrat total:

weighted.mean(surv$prop, w = surv$total)   # equals the pooled proportion

[1] 0.1683453

This lands on 0.168, the pooled value again. So the real question behind the trap is: what should the weights be? If each individual should count equally, weight by effort (the pooled proportion). If each site should count equally, do not weight (the mean of ratios). Deciding that is ecology, not arithmetic, and summarise() will happily give you either without asking.

Figure 1: Focal-species proportion in each quadrat (bars), with the pooled proportion (solid line) and the mean of proportions (dashed line). The two summaries straddle the bars because the small high-proportion quadrats pull the unweighted mean up, while the pooled value follows the large quadrats.

When summarise() keeps a grouping

group_by() takes more than one column. Say you first total each quadrat within its habitat, then want one pooled proportion for the whole survey. (Each quadrat is already one row here, so the first step changes no counts. With one row per visit it would.)

per_q <- surv |>
  group_by(habitat, quadrat) |>
  summarise(focal = sum(focal), total = sum(total))

`summarise()` has regrouped the output.
ℹ Summaries were computed grouped by habitat and quadrat.
ℹ Output is grouped by habitat.
ℹ Use `summarise(.groups = "drop_last")` to silence this message.
ℹ Use `summarise(.by = c(habitat, quadrat))` for per-operation grouping
  (`?dplyr::dplyr_by`) instead.

group_vars(per_q)

[1] "habitat"

dplyr prints a message under this chunk. It is a message, not a warning and not an error: the table is correct. It says the output is still grouped by habitat. summarise() removed only the last grouping column, quadrat, and group_vars() confirms what is left. The exact wording depends on the dplyr version (it changed in dplyr 1.2.0; this page was built with dplyr 1.2.1). The leftover grouping matters at the next step:

by_hab <- per_q |> summarise(pooled = sum(focal) / sum(total))
by_hab

# A tibble: 2 × 2
  habitat  pooled
  <chr>     <dbl>
1 edge      0.115
2 interior  0.505

share_sum <- per_q |> mutate(share = focal / sum(focal)) |> pull(share) |> sum()
share_sum

[1] 2

The pooled proportion comes back as one row per habitat: 0.115 for edge and 0.505 for interior, not the survey-wide 0.168. Average the two rows and you get 0.310, far from the pooled value. It also lands close to the mean of ratios (0.308), but only because both habitats have 3 quadrats and the quadrats within a habitat are of similar size. On another survey the two need not agree. Each habitat gets one equal vote, although edge quadrats hold 600 of the 695 individuals. It is the trap from the section above, reached by accident. The share column fails the same way: inside the grouped mutate(), sum(focal) is a habitat total, so the shares add up to 2, one per habitat, not 1.

Three small changes remove the grouping. Each gives back the pooled proportion:

surv |>
  group_by(habitat, quadrat) |>
  summarise(focal = sum(focal), total = sum(total), .groups = "drop") |>
  summarise(pooled = sum(focal) / sum(total))

# A tibble: 1 × 1
  pooled
   <dbl>
1  0.168

per_q |> ungroup() |> summarise(pooled = sum(focal) / sum(total))

# A tibble: 1 × 1
  pooled
   <dbl>
1  0.168

surv |>
  summarise(focal = sum(focal), total = sum(total), .by = c(habitat, quadrat)) |>
  summarise(pooled = sum(focal) / sum(total))

     pooled
1 0.1683453

.groups = "drop" removes every grouping when summarise() finishes. ungroup() removes it afterwards. .by groups for this one call only (it needs dplyr 1.1.0 or newer), and the table it returns is not grouped, so nothing is left over. (The third result prints as a plain data frame because surv is one; the number is the same.) .by refuses a table that is already grouped. That is an error, which stops a script; here tryCatch() catches it so the page can print its text:

tryCatch(per_q |> summarise(pooled = sum(focal) / sum(total), .by = habitat),
         error = function(e) conditionMessage(e))

[1] "Can't supply `.by` when `.data` is a grouped data frame."

Where to go next

Grouped summaries assume the table is already assembled, but survey data usually arrives in two tables, one for sites and one for observations, that you have to join. The next trap lives there: an inner join silently drops the sites where the species was absent, which biases exactly the kind of mean you just computed. The joining tutorial takes that apart.

References

Wickham H, Averick M, Bryan J, et al 2019. Journal of Open Source Software 4(43):1686 (10.21105/joss.01686).

Wickham H, Cetinkaya-Rundel M, Grolemund G 2023. R for Data Science, 2nd ed. O’Reilly. ISBN 978-1-492-09740-2.

Related tutorials

• Per chick or per nest: informative cluster size
• Repeating an analysis for every site in R
• Joining ecological tables without losing zeros
• From a field sheet to your first analysis

Reproducibility

• Code last run: 2026-09-30
• R version 4.5.3 (2026-03-11), aarch64-apple-darwin20
• Packages: dplyr 1.2.1, ggplot2 4.0.3

PreviousStep 7, Your first ggplot: data, aes and geom CourseAll 44 lessons NextStep 9, From a field sheet to your first analysis