Freshness SLA Breach Runbook

A freshness breach is the quietest incident in this collection: the data is valid, correctly projected, fully replicated, and simply late — a spatial layer has crossed the staleness budget its consumers were promised. This runbook is the procedure for that breach: reading the spatial_freshness_* metrics to confirm the budget is genuinely spent, decomposing end-to-end latency to find the one stalled stage, restarting or unblocking it, and hardening so the next stall pages while there is still budget left. It is one of the spatial pipeline incident runbooks inside the spatial incident response and tooling program, and it operationalizes the standing practice in tracking spatial data freshness SLAs.

Per-stage freshness decomposition against the SLA budget Four pipeline stages left to right — ingest, transform, index, publish — each drawn as a box with a per-stage latency contribution. The transform stage is marked stalled and shows a much larger latency bar than the others. Below the stages, a horizontal freshness budget bar fills from left to right; the fill crosses a dashed SLA line because the stalled transform stage has consumed most of the budget, marking a breach. Ingest +8 s Transform stalled +540 s Index +12 s Publish +6 s freshness budget SLA line breach

Problem Framing

Freshness is an outcome metric, and a breach tells you the promise was broken without telling you where. The spatial_freshness_seconds gauge measures the age of the newest data a layer is serving — the gap between now and the timestamp of the most recent feature that has reached the published layer. When that age exceeds the layer’s budget, consumers are looking at data older than they were guaranteed. Crucially, freshness is cumulative across every pipeline stage: ingest, transform, index, and publish each add latency, and a stall in any one of them spends the whole budget. The breach metric is the sum; the incident lives in one term of that sum.

This is what separates a freshness breach from the other runbooks. Topology and CRS incidents point at corrupt data; replication lag points at a region; a tile queue points at render workers. A freshness breach points nowhere until you decompose it. The entire skill of this runbook is attribution: turning “the layer is 9 minutes stale” into “the transform stage has been stalled for 9 minutes,” because the remediation for a stalled transform is nothing like the remediation for a stuck index build or a paused publish job.

The sibling reference triaging a freshness SLA breach drills into the attribution technique in depth; this runbook is the on-call procedure that wraps it, from the page to the post-mortem. Because a breach is a broken promise rather than broken data, the posture is restore-flow, not repair: get the stalled stage moving again and let the accumulated backlog drain, rather than quarantining or rewriting anything.

Detect

Freshness is detected as a burn ratio against the per-layer budget, not as an absolute age, because a 5-minute age is fine for a cadastral layer and a breach for a live-position feed. The rule fires a warning as the budget erodes and a critical once it is spent, so on-call has a chance to intervene before consumers are exposed.

groups:
  - name: freshness-sla-breach
    rules:
      - alert: SpatialFreshnessBudgetBurn
        expr: |
          spatial_freshness_seconds
            / on(layer) group_left() spatial_freshness_budget_seconds > 0.7
        for: 10m
        labels: { severity: warning }
        annotations:
          runbook_url: "/spatial-incident-response-and-tooling/spatial-pipeline-incident-runbooks/freshness-sla-breach-runbook/"
      - alert: SpatialFreshnessSLABreached
        expr: |
          spatial_freshness_seconds
            / on(layer) group_left() spatial_freshness_budget_seconds > 1
        for: 2m
        labels: { severity: critical }
        annotations:
          summary: "{{ $labels.layer }} freshness {{ $value | humanizePercentage }} of budget"

With a budget BB and observed age ss, the burn ratio ρ=s/B\rho = s / B crosses 11 exactly at breach; the warning at ρ≥0.7\rho \geq 0.7 buys roughly the remaining thirty percent of budget as lead time. Confirm the age is real before decomposing: spatial_freshness_seconds derived from a per-layer max(feature_timestamp) can spike falsely if the source clock jumped or a batch arrived with future timestamps, so cross-check the newest feature’s timestamp against a trusted clock.

Triage

Triage is the per-stage decomposition. Each stage emits its own timing, so the sum reconstructs the freshness age and the largest term names the stall. The query below ranks stage latencies for the breaching layer.

# Per-stage latency contribution for the breaching layer — the stall is the max
topk(1,
  max(gis_etl_stage_latency_seconds{layer="road_closures"}) by (stage))

# Confirm the stalled stage stopped producing, not just slowed
rate(gis_etl_stage_completed_total{layer="road_closures", stage="transform"}[5m])

Read the decomposition against the four stages: a large ingest term points upstream (a source feed stopped or slowed); a large transform term points at reprojection or simplification compute (a heavy batch, a lost worker, a poisoned input looping on retries); a large index term points at a GiST rebuild or vacuum blocking writes; a large publish term points at the tile or serving layer, which may itself be a queue-overflow incident. If the stalled stage’s completion rate has dropped to zero, it is stopped, not slow — a different and more urgent diagnosis than a stage that is merely behind. Scope consumer impact by how far past budget the layer has burned, and set severity accordingly.

Remediate

Remediation restores flow at the stalled stage, then lets the backlog drain. The action depends entirely on which stage triage named — there is no single fix, which is exactly why attribution came first.

# Match the remediation to the stalled stage identified in triage:
remediate:
  ingest_stalled:      # source feed stopped or slowed
    - verify upstream feed is producing; restart the connector
    - if source is down, mark the layer degraded and wait, do not fabricate data
  transform_stalled:   # reprojection / simplification blocked
    - check for a poisoned input looping on retries (see topology/CRS runbooks)
    - restore worker capacity; requeue the stuck batch with a dead-letter cap
  index_stalled:       # GiST rebuild / vacuum blocking writes
    - let the in-flight rebuild finish or cancel it off-peak; do not kill mid-write
  publish_stalled:     # serving / tile layer behind
    - hand off to the tile publish queue overflow runbook
-- Index-stage stall: is a rebuild or autovacuum holding the write lock?
SELECT pid, state, wait_event_type, query
FROM pg_stat_activity
WHERE query ILIKE '%road_closures%'
  AND (query ILIKE '%REINDEX%' OR query ILIKE '%VACUUM%');
-- A long-running REINDEX here explains an index-stage freshness stall

The one rule that spans every stage: restore flow, never fabricate freshness. It can be tempting to bump the freshness timestamp or re-publish stale data with a new clock to clear the alert — that silences the metric while lying to consumers, which is worse than the breach. If the source itself is down, mark the layer degraded honestly and wait; freshness cannot be manufactured downstream of a dead feed. Once the stalled stage is moving, the accumulated backlog drains on its own and the freshness age falls back under budget.

Post-mortem and Verification

Verify recovery on the outcome metric and its cause: the burn ratio ρ\rho must fall back below the warning line and hold, and the previously-stalled stage’s latency must return to its normal contribution. A freshness age that recovers while the stalled stage is still slow means the backlog is draining temporarily and will breach again at the next batch.

# Recovery: freshness back within budget for the affected layer
spatial_freshness_seconds{layer="road_closures"}
  / on(layer) group_left() spatial_freshness_budget_seconds < 0.7

The post-mortem’s job is to explain why the breach was detected as a breach rather than caught as an erosion. If only the critical alert fired and not the warning, the warning threshold or its for window is mistuned and on-call lost the lead time the budget was supposed to provide. Harden by confirming the two-tier detector fires in order, and by adding a per-stage latency alert on whichever stage stalled, so the next stall pages at the stage level — naming the culprit immediately — instead of surfacing only as an aggregate freshness breach after the budget is already spent. Recurring stalls in the same stage during the same window are a capacity or scheduling problem for the freshness program to fix structurally, not an incident to re-run.

Gotchas

Do not clear the alert by fabricating a timestamp. Re-stamping stale data with a fresh clock silences the metric and deceives consumers. The only honest resolution is restoring flow so genuinely new data reaches the layer; if the source is dead, mark the layer degraded.

A stalled transform is often a data-corruption incident in disguise. A poisoned geometry looping on retries can stall the transform stage and surface first as a freshness breach. If triage points at transform, check whether the topology corruption incident runbook or a CRS cascade is the real root cause before treating it as pure latency.

Aggregate freshness hides the stalled stage. The breach metric is a sum, so it tells you the layer is late without telling you where. Never remediate off the aggregate alone — decompose to the per-stage latency first, or you will restart the wrong stage and waste budget.

FAQ

Why is freshness measured as a ratio against a budget?

Because acceptable staleness is layer-specific. A cadastral layer tolerates hours; a live road-closure or vessel-position feed tolerates seconds. A single absolute threshold would either page constantly on slow-by-design layers or miss a fast layer falling behind. The burn ratio normalizes every layer to its own promise.

How do I find which stage is stalled?

Decompose the freshness age into per-stage latency contributions using gis_etl_stage_latency_seconds labelled by stage. The stage with the largest contribution is the stall. Then check that stage’s completion rate to distinguish “slow” from “fully stopped,” which need different remediations.

Can I clear the alert by re-publishing the data?

Only if re-publishing actually moves genuinely newer data through the stalled stage. Bumping the freshness timestamp on the same stale data to silence the alert is fabrication — it hides the breach from consumers who are relying on the freshness guarantee. Restore flow instead.

What if the upstream source is simply down?

Then freshness cannot be restored downstream — you cannot manufacture data that does not exist. Mark the layer degraded, notify consumers honestly, and track the source recovery. The breach is real and the correct response is transparency, not a silenced metric.

Is a freshness breach ever caused by another incident?

Frequently. A stalled transform can be a topology or CRS incident looping on a poisoned input; a stalled publish can be a tile queue overflow. Freshness is an outcome metric, so a breach is often the first visible symptom of a cause that has its own runbook. Always confirm the stalled stage is a genuine latency problem and not a corruption incident wearing a freshness mask.