Measuring Freshness for Irregularly Updated Layers

Freshness monitoring assumes a cadence. Set a bound, compare the layer’s age against it, alert when it is exceeded — straightforward for a layer that refreshes nightly at 02:00. It falls apart for the layers that update when something happens: a planning-boundary set revised whenever a council passes an order, an incident layer that receives edits in bursts and then nothing for a week, a survey dataset delivered when fieldwork completes. These layers are permanently “stale” against any fixed bound and permanently silent against any bound loose enough not to fire. This guide covers how to monitor freshness when there is no cadence, using the layer’s own observed update distribution instead of a constant. It belongs to tracking spatial data freshness SLAs under spatial data freshness and quality metrics.

A fixed bound against an irregular update pattern, compared with a distribution-derived bound An age series rises and falls irregularly as updates arrive in bursts separated by long quiet periods. A fixed bound drawn low is crossed constantly, producing many alerts that all resolve on their own. A second bound derived from the ninety-fifth percentile of the layer's own observed gaps between updates sits much higher and is crossed only once, at the point where the layer genuinely stopped updating. A note states that the fixed bound produces alert fatigue while telling you nothing the distribution-derived one misses. A fixed bound fires on normal behaviour; the layer’s own distribution does not old fresh fixed bound — crossed on every quiet week p95 of observed gaps × 1.5 — crossed once, correctly genuinely stopped wk 0 wk 8 wk 16 wk 22 Seven false alerts against one true one is the usual ratio when a fixed bound meets an event-driven layer.

Problem framing: what “stale” means without a cadence

For an event-driven layer, age carries almost no information on its own. What matters is whether the current age is unusual for this layer, and there are three defensible ways to express that.

Against the layer’s own gap distribution. Collect the intervals between successive updates over a long window and take a high percentile. If the layer has historically gone at most nineteen days between updates at the ninety-fifth percentile, an age of thirty days is genuinely anomalous while an age of ten is routine. This is the general-purpose answer and works for most layers.

Against the source’s own signal. Some sources publish a “last reviewed” or “next expected” date even when they do not publish on a schedule. Where that exists it beats any statistical estimate, because it encodes intent rather than history. The check becomes “is the source’s own promised date in the past”.

Against a correlated proxy. For layers driven by an observable external process — planning boundaries following council meetings, flood extents following rainfall — the proxy tells you whether an update should have arrived. This is the most accurate approach and the most work, and it is worth building only for layers whose staleness is expensive.

All three share a property that a fixed bound lacks: they distinguish “quiet because nothing happened” from “quiet because the pipeline broke”, which is the only distinction that matters here.

Implementation: derive the bound from observed gaps

Record every update event, then compute the bound from the gap distribution rather than declaring it.

-- Update events for an irregular layer: one row per accepted batch.
-- Already available if the reconciliation ledger is in place.
WITH gaps AS (
  SELECT
    layer,
    recorded_at,
    recorded_at - LAG(recorded_at) OVER (PARTITION BY layer ORDER BY recorded_at)
      AS gap
  FROM audit.batch_reconciliation
  WHERE hop = 'published' AND recorded_at > now() - interval '2 years'
)
SELECT
  layer,
  COUNT(*)                                                          AS updates,
  percentile_disc(0.50) WITHIN GROUP (ORDER BY gap)                 AS p50_gap,
  percentile_disc(0.95) WITHIN GROUP (ORDER BY gap)                 AS p95_gap,
  MAX(gap)                                                          AS max_gap,
  -- The bound: a margin above the observed p95, so ordinary quiet
  -- periods never fire and a genuine stop always does.
  percentile_disc(0.95) WITHIN GROUP (ORDER BY gap) * 1.5           AS suggested_bound
FROM gaps
WHERE gap IS NOT NULL
GROUP BY layer;

Two design choices in that query matter.

The two-year window is deliberate: irregular layers frequently have annual structure — a planning layer quiet over the summer recess, a survey layer that only updates in the field season — and a shorter window produces a bound that fires every year at the same time. Where the seasonality is strong, computing the bound per month rather than globally is worth the extra complexity.

The 1.5 multiplier on the p95 is a margin, not a fudge. Using the p95 directly means five percent of ordinary gaps fire the alert by construction, which on a layer updating fifty times a year is two or three false alerts annually — enough to erode trust. The margin buys silence on normal behaviour at the cost of detecting a stop slightly later, which is the right trade for a layer nobody expects to update this week anyway.

Publish the derived bound back into the registry so the alert rule reads it like any other threshold.

- alert: IrregularLayerStopped
  # The bound comes from the registry, recomputed monthly from the layer's
  # own gap distribution — no per-layer rule, and no hand-set constant.
  expr: |
    (time() - max by (layer) (gis_etl_last_published_timestamp_seconds))
      > on (layer) group_left() max by (layer) (gis_layer_registry_irregular_bound_seconds)
  for: 6h
  labels: { severity: warning, data_domain: spatial }
  annotations:
    summary: >-
      {{ $labels.layer }} has not updated in {{ $value | humanizeDuration }},
      beyond its own p95 gap — check the source, not the pipeline

The six-hour sustain is appropriate here in a way it would not be on a live feed: an irregular layer crossing its bound is not an emergency, and a long sustain avoids firing on a boundary case that resolves when a delayed batch lands.

Choosing a freshness approach by what is known about the layer A three-branch decision. If the source publishes its own expected-next date, use that date directly as the bound, which is the most accurate option. If not, and an observable external process drives updates, use that process as a proxy to decide whether an update should have arrived. If neither, derive the bound from the layer's own observed gap distribution at the ninety-fifth percentile with a margin. Each branch notes the effort required and the accuracy obtained. Use the strongest signal the layer actually offers 1. the source publishes an expected-next date use it directly · effort: minimal · accuracy: highest — it encodes intent, not history alert when that date passes with no update 2. an observable process drives updates use the process as a proxy · effort: high · accuracy: high e.g. council decisions, rainfall events, field-season calendar 3. neither — use the layer’s own gap distribution p95 of observed gaps × 1.5 · effort: low · accuracy: adequate recompute monthly; segment by month where seasonality is strong Observed gap distribution for four irregular layers Four irregular layers are compared by the ratio of their ninety-fifth-percentile update gap to their median gap. A planning-boundary layer has a ratio near two, so its distribution is tight and a percentile-based bound works well. A survey dataset has a ratio near four. An incident layer has a ratio near nine, driven by long quiet periods between bursts. A grant-funded mapping project has a ratio above twenty, and the note states that a bound derived from a distribution this dispersed is barely better than a guess. The more dispersed the gap distribution, the weaker a percentile-based bound becomes planning boundaries p95 / median = 2.1 survey dataset 4.0 incident layer 8.9 grant-funded mapping 21.4 · barely better than a guess

Verification

Backtest the bound. Replay the layer’s update history against the derived value and count how many times it would have fired. Zero or one over two years is right for a layer that never actually stopped; more than three means the bound is too tight or the seasonality needs handling per month.

Then confirm it detects a real stop. Take the history, truncate the last six months of updates, and confirm the bound is crossed at approximately the expected point. A bound derived from a distribution that includes the stop will be too loose to catch it — recompute from data preceding the simulated stop.

Gotchas

Deriving the bound from a window containing an outage. The outage inflates the p95 and the bound becomes too loose to catch the next one. Exclude known incidents from the distribution, or use the p95 of a trimmed series.

Recomputing the bound continuously. A bound that updates on every event drifts upward during a slow stop and never fires. Recompute on a schedule — monthly is ample — from a window that ends before the current gap.

Applying it to layers that do have a cadence. If a layer updates predictably, use the cadence; the distribution approach is strictly worse there because it tolerates a missed run that a cadence bound would catch immediately.

Ignoring the source’s own signal. Statistical estimation is the fallback, not the first choice. A source that tells you when to expect the next update has given you a better answer than any percentile.

Paging on it. An irregular layer crossing its bound almost always means the source stopped publishing, which is not something an on-call engineer can fix at 03:00. Route to the daily queue, as the severity model in alert routing and on-call design for spatial pipelines sets out.

FAQ

What if the layer has too few updates to estimate a distribution?

Below roughly ten observed gaps the percentile is unreliable. Use the maximum observed gap with a generous margin instead, and revisit once more history accumulates. Recording that the bound is provisional in the registry stops someone treating a weak estimate as authoritative.

How does this interact with the freshness error budget?

An irregular layer should not carry a minute-based freshness budget at all, because the denominator — minutes during which the layer should have been current — is not meaningful. Track it as a count of bound crossings per year instead, and reserve budgets for layers with a real cadence, as described in setting error budgets for spatial freshness.

Should consumers be told the layer is irregular?

Explicitly, in the layer status document. A consumer that assumes a daily refresh on an event-driven layer will build a stale-data check that fires constantly and then be disabled. Publishing the expected gap alongside the age lets the consumer set its own tolerance sensibly.

What about layers that are genuinely finished?

Some spatial layers reach a terminal state — a historical survey, a decommissioned sensor network. Mark them as closed in the registry and exclude them from freshness monitoring entirely rather than letting a bound fire forever. An unmonitored-by-design layer with a recorded reason is far healthier than a permanently red one.

Can the same approach handle irregular coverage rather than freshness?

The reasoning transfers: compare the current value against the layer’s own historical distribution rather than a constant. It is the same technique the coverage checks use when a layer’s extent legitimately varies, and it is why the baseline comparison in spatial coverage and extent monitoring is expressed as a ratio to baseline.