---
title: "What a shapefile loses: names, times and types"
description: "A shapefile round trip returns the geometry exactly and then rewrites the attribute table: names abbreviated, the clock dropped, logicals made integer."
date: "2026-08-12 11:00"
categories: [R, sf, spatial, data management, ecology tutorial]
image: thumbnail.png
image-alt: "Two twenty-four hour clock faces of visit times, one with the points occupying the morning and early afternoon arc and the other with the points in a single radial line at midnight."
---
The geometry is the part everyone worries about, and the geometry is the part that survives. Write a set of plot locations to a shapefile, read them back, and the coordinates are identical to the last bit and the coordinate reference system comes back intact. What changes is the table sitting beside the points, and it changes without an error.
This is the spatial twin of a problem that also afflicts plain text: the values return, the decisions you put on top of them do not. A shapefile is worse than a CSV in one specific way, though. A CSV at least gives you back the column name you wrote.
## An attribute table an ecologist would actually build
Twenty-four vegetation plots, a browse index measured in two seasons, an ordered cover class, the time each plot was visited, a logical for whether browsing was seen at all, and a soil pH with two missing values.
```{r setup}
#| message: false
#| warning: false
library(sf)
library(ggplot2)
te_paper <- "#f5f4ee"
te_ink <- "#16241d"
te_body <- "#2c3a31"
te_forest <- "#275139"
te_rust <- "#b5534e"
te_gold <- "#c9b458"
te_line <- "#dad9ca"
theme_datasheet <- function() {
theme_minimal(base_size = 12) +
theme(plot.background = element_rect(fill = te_paper, colour = NA),
panel.background = element_rect(fill = te_paper, colour = NA),
panel.grid.major = element_line(colour = te_line, linewidth = 0.3),
panel.grid.minor = element_blank(),
text = element_text(colour = te_body),
plot.title = element_text(colour = te_ink, face = "bold"),
axis.text = element_text(colour = te_body))
}
set.seed(20260812)
n <- 24
# the plots were visited during the working day, centred on mid-morning
hours <- (11 + rnorm(n, 0, 2.1)) %% 24
field <- data.frame(
site_code = sprintf("S%02d", seq_len(n)),
browse_index_spring = round(runif(n, 0, 1), 3),
browse_index_autumn = round(runif(n, 0, 1), 3),
canopy_cover_class = factor(sample(c("low", "medium", "high"), n, TRUE),
levels = c("low", "medium", "high"),
ordered = TRUE),
visit_time = as.POSIXct("2026-05-01", tz = "UTC") + 3600 * hours,
browsed = sample(c(TRUE, FALSE), n, TRUE),
soil_ph = round(runif(n, 4.2, 7.8), 2),
stringsAsFactors = FALSE)
field$soil_ph[c(4, 17)] <- NA
plots <- st_as_sf(cbind(field,
x = 400000 + runif(n, 0, 8000),
y = 5050000 + runif(n, 0, 8000)),
coords = c("x", "y"), crs = 32634)
str(field$canopy_cover_class)
```
Now deposit it twice: once as a shapefile, because that is still what most repositories and most collaborators ask for, and once as a GeoPackage, so there is something to compare against.
```{r write}
#| warning: false
dir_out <- file.path(tempdir(), "deposit")
dir.create(dir_out, showWarnings = FALSE)
shp <- file.path(dir_out, "plots.shp")
gpkg <- file.path(dir_out, "plots.gpkg")
st_write(plots, shp, quiet = TRUE, delete_dsn = TRUE)
st_write(plots, gpkg, quiet = TRUE, delete_dsn = TRUE)
from_shp <- st_read(shp, quiet = TRUE)
from_gpkg <- st_read(gpkg, quiet = TRUE)
```
## The geometry is exact and the names are not
First the good news, and check it strictly: `identical` rather than `all.equal`, because the claim is that nothing moved at all, and `all.equal` would pass a shift of several millimetres on coordinates this large.
```{r geometry}
c(coordinates = identical(st_coordinates(plots), st_coordinates(from_shp)),
crs = st_crs(plots) == st_crs(from_shp))
```
Nothing moved. Now the column names.
```{r names}
nm <- setdiff(names(from_shp), "geometry")
data.frame(deposited = setdiff(names(plots), "geometry"), shapefile = nm)
```
The DBF file that carries a shapefile's attributes allows ten characters per field name, so something has to give, and `sf` abbreviates rather than truncates: it strips vowels and repeated letters until the name fits. The name `soil_ph` was short enough to survive, and `canopy_cover_class` came back as **`r nm[4]`**, which is guessable. The pair that matters is the browse index.
```{r seasons}
nm[2:3]
```
Spring and autumn have been reduced to a trailing **`r substr(nm[2], 10, 10)`** and a trailing **`r substr(nm[3], 10, 10)`**. The two columns are still there, still in the right order, still holding the right numbers. What is gone is the only thing in the file that said which one was which, and the surviving letters do not say it either: the `t` is the third letter of autumn, so alphabetical intuition points the wrong way if anyone reaches for it.
The immediate practical consequence is that code written against the deposited object stops finding its columns, and `$` does not complain.
```{r dollar}
c(spring_column_found = !is.null(from_shp$browse_index_spring),
time_column_found = !is.null(from_shp$visit_time))
```
`NULL` is not an error. It flows into `mean()` as `NA`, into a `data.frame()` call as a dropped column, and into an arithmetic expression as a zero-length vector, and each of those failures happens somewhere other than the line that caused it.
## The clock is gone
The DBF format has a date type and no date-time type. A `POSIXct` column is therefore written as a date, and GDAL says so at write time, in a message that scrolls past in a script and is invisible in a Quarto document.
```{r classes}
cls <- function(z) sapply(st_drop_geometry(z), function(v) class(v)[1])
data.frame(column = names(cls(plots)),
deposited = unname(cls(plots)),
shapefile = unname(cls(from_shp)),
geopackage = unname(cls(from_gpkg)))
```
Three rows changed. The ordered factor became character, which loses the level ordering exactly as a CSV does. The logical became an integer, which is harmless until someone sums it and calls the result a count of sites rather than a count of `TRUE`s, or filters on `browsed == "TRUE"` and gets nothing back. The third is the expensive one.
```{r clock}
recovered <- st_drop_geometry(from_shp)[[5]]
c(class = class(recovered)[1], distinct_values = length(unique(recovered)))
```
Every visit now happened at midnight on the same day. If those timestamps were camera trap detections, or acoustic triggers, or anything else whose analysis is a distribution over the twenty-four hour cycle, the analysis is no longer possible from the deposited file. Here is what that costs, in the three numbers such a study reports.
```{r activity}
circ_mean <- function(h) {
a <- 2 * pi * h / 24
(atan2(mean(sin(a)), mean(cos(a))) %% (2 * pi)) * 24 / (2 * pi)
}
circ_r <- function(h) {
a <- 2 * pi * h / 24
sqrt(mean(sin(a))^2 + mean(cos(a))^2)
}
night <- function(h) mean(h < 6 | h >= 18)
after_h <- rep(0, n) # midnight, for every record
round(c(mean_hour_before = circ_mean(hours),
mean_hour_after = circ_mean(after_h),
r_before = circ_r(hours),
r_after = circ_r(after_h),
night_before = night(hours),
night_after = night(after_h)), 3)
```
The mean hour moves from `r sprintf("%.1f", circ_mean(hours))` to `r sprintf("%.1f", circ_mean(after_h))`, and the mean vector length, which measures how tightly the records cluster on the circle, rises from `r sprintf("%.3f", circ_r(hours))` to `r sprintf("%.3f", circ_r(after_h))`: one is the value for records concentrated in a few hours of the day, the other is the value for records at a single instant. The share that a daylight and darkness split would call nocturnal moves from `r sprintf("%.2f", night(hours))` to `r sprintf("%.2f", night(after_h))`. Nothing warns about any of this. A total collapse like this one is at least conspicuous: a mean vector length of exactly 1 is not a result anyone would publish without looking. The dangerous version is partial, where one deposited file in a series lost its clock and the others did not, and the pooled distribution simply grows a spike at midnight.
```{r fig-clock}
#| fig-cap: "Visit times on a twenty-four hour clock face, as deposited and as read back from the shapefile. The distance from the centre only separates points that would otherwise overlap."
#| fig-alt: "Two circular panels with hour labels around the rim and hours running clockwise from zero at the top. In the left panel the twenty-four points occupy just under half the rim, from the early morning hours round to the early afternoon, at scattered distances from the centre. In the right panel the twenty-four points form a single straight radial line at the zero mark."
clock <- rbind(
data.frame(hour = hours, panel = "as deposited"),
data.frame(hour = after_h, panel = "read back from the shapefile"))
clock$panel <- factor(clock$panel, levels = unique(clock$panel))
# the radius separates overlapping points and carries no information
clock$rad <- ave(seq_len(nrow(clock)), clock$panel, FUN = seq_along)
ggplot(clock, aes(x = hour, y = 0.32 + 0.03 * rad)) +
geom_point(aes(colour = panel), size = 2.2, alpha = 0.9) +
facet_wrap(~ panel) +
coord_polar(theta = "x", start = 0) +
scale_x_continuous(limits = c(0, 24), breaks = seq(0, 21, 3)) +
scale_y_continuous(limits = c(0, 1.12), breaks = NULL) +
scale_colour_manual(values = c(te_forest, te_rust)) +
labs(x = NULL, y = NULL,
title = "The same visits, before and after the deposit") +
theme_datasheet() +
theme(legend.position = "none",
panel.grid.major = element_line(colour = te_line, linewidth = 0.3),
strip.text = element_text(colour = te_ink, face = "bold"))
```
## What survives, column by column
Missing values are the case worth checking rather than assuming, because the folklore says a shapefile turns `NA` into zero.
```{r missing}
c(na_deposited = sum(is.na(field$soil_ph)),
na_shapefile = sum(is.na(from_shp$soil_ph)),
na_geopackage = sum(is.na(from_gpkg$soil_ph)))
```
They come back. A blank numeric field in the DBF is read as `NA`, and has been for many GDAL versions. The folklore comes from software that writes a fill value such as `-9999` or `0` on the way out, which is a choice made by the exporting program and not a property of the format. Worth knowing which of those your GIS does before you blame the file.
```{r fig-types}
#| fig-cap: "Class of each attribute column as deposited and as recovered from each format."
#| fig-alt: "A grid of cells with the seven attribute columns down the side and three format labels along the bottom, each cell giving an R class. Three cells are highlighted in the shapefile column, for the ordered factor, the date-time and the logical, and one in the GeoPackage column, for the ordered factor."
tab <- data.frame(
column = rep(names(cls(plots)), 3),
format = rep(c("deposited", "shapefile", "GeoPackage"), each = 7),
class = c(unname(cls(plots)), unname(cls(from_shp)), unname(cls(from_gpkg))))
tab$format <- factor(tab$format,
levels = c("deposited", "shapefile", "GeoPackage"))
tab$column <- factor(tab$column, levels = rev(names(cls(plots))))
tab$changed <- tab$class != rep(unname(cls(plots)), 3)
ggplot(tab, aes(x = format, y = column)) +
geom_tile(aes(fill = changed), colour = te_paper, linewidth = 1.4) +
geom_text(aes(label = class), size = 3.1, colour = te_ink) +
scale_fill_manual(values = c(`FALSE` = te_line, `TRUE` = te_gold)) +
labs(x = NULL, y = NULL,
title = "Where the attribute table changes class") +
theme_datasheet() +
theme(legend.position = "none",
panel.grid.major = element_blank())
```
The GeoPackage column is the control. It is a SQLite database with a real type system, so the time of day, the logical and the long field names all come back. The one thing it also loses is the factor, and that is not a format failure: no file format on that list stores an R factor, because the level ordering is a statement about the analysis and not about the data.
## Depositing so the table survives
Three things, in order of how much they buy you.
Use GeoPackage as the working format. A single file, no sidecar files to lose, no ten character limit, real types, and every GIS made in the last decade opens it. If a collaborator or a repository insists on a shapefile, write the shapefile as well and treat it as the export it is, not as the archive.
Keep the ordering out of the file. A small dictionary that names the columns, their types, their units and the level set in order is readable by a person and by a script, and it is the only place the ordering can live.
```{r dictionary}
dictionary <- data.frame(
column = c("canopy_cover_class", "visit_time", "browsed"),
type = c("ordered factor", "POSIXct, UTC", "logical"),
detail = c(paste(levels(field$canopy_cover_class), collapse = " < "),
"time of day is the analysis variable",
"TRUE means browsing seen"))
dictionary
```
Then rebuild from the dictionary rather than from whatever the file happens to return, and check the rebuild by re-running the analysis on it. If a number moves, the deposit is not the analysis.
```{r rebuild}
lv <- strsplit(dictionary$detail[1], " < ")[[1]]
rebuilt <- from_gpkg
rebuilt$canopy_cover_class <- factor(rebuilt$canopy_cover_class,
levels = lv, ordered = TRUE)
c(levels_from_gpkg = paste(levels(factor(from_gpkg$canopy_cover_class)),
collapse = " < "),
levels_rebuilt = paste(levels(rebuilt$canopy_cover_class), collapse = " < "))
```
## Honest limits
The abbreviation shown here is `sf`'s, not the shapefile's. Write the same object from QGIS and you get plain truncation to ten characters instead, which collides where `sf` does not: two columns whose first ten characters agree end up as one name plus a numbered variant. Either way the season labels are gone; the two tools lose them differently.
Nothing above depends on the values being synthetic. What changes with real data is how much information the three activity summaries carried in the first place: the more structure the timestamps held, the more the round trip destroys. A schedule that was already uniform round the clock would lose no information, although its mean vector length would still jump to one, which is a reminder that the size of the change in a summary is not the size of the loss.
The date itself survives, so a study whose time variable is the day rather than the hour is unaffected. So is any analysis that reads its attributes from the original object and uses the shapefile only to draw a map. The loss is real when the deposited file is the thing someone reanalyses, which is exactly the situation archiving is for.
Finally, the field name limit is not the only shapefile limit, only the one that shows up first. Character fields are cut at 254 characters, there is a cap on the number of fields, and the format has no way to store a null geometry. None of those announce themselves either.
## References
Pebesma E 2018 The R Journal 10(1):439-446 (10.32614/RJ-2018-009)
Bivand RS 2021 Journal of Geographical Systems 23(4):515-546 (10.1007/s10109-020-00336-0)
Wilkinson MD, Dumontier M, Aalbersberg IJ, et al 2016 Scientific Data 3:160018 (10.1038/sdata.2016.18)
## Related tutorials
- [What a CSV loses: archiving an R analysis](../what-a-csv-loses/)
- [Checking a vector layer before you measure](../checking-a-vector-layer/)
- [Running QGIS from the command line](../running-qgis-from-the-command-line/)
- [Dates and times in ecological data](../dates-and-times-in-ecological-data/)
