Spatial Pipeline Incident Runbooks

A spatial pipeline rarely fails with a stack trace; it fails with a map that is subtly wrong — features in the wrong hemisphere, tiles that never refresh, a region quietly serving yesterday’s data. This reference collects the operational runbooks that turn those ambiguous symptoms into a repeatable detect-triage-remediate-post-mortem sequence, and it sits directly under the spatial incident response and tooling program that frames how on-call teams instrument, alert on, and recover geospatial workflows. The audience is the data engineers, GIS platform administrators, and site reliability engineers who carry the pager when a spatial layer goes bad and need a decision path rather than a debugging improvisation.

The loop above is deliberately not a straight line. Every incident this reference covers ends by feeding a sharpened threshold or a new alert back into detection, so the same failure mode does not page twice for the same reason. What follows is the shared framework — how runbooks bind to your alerting and SLO layer, a map of the five failure modes to their signal metrics, a reusable template, and the triage discipline — before the five deep runbooks take over one failure mode each.

Architecture: how runbooks bind to alerting and SLOs

A runbook is only as good as the alert that opens it. In this domain the alert is never a generic host-down page; it is a spatial signal — an invalid-geometry rate, a projection-drift counter, a queue depth, a replication-lag gauge, a freshness clock — evaluated against an objective the team already agreed to. The geospatial observability architecture and fundamentals reference defines that instrumentation layer end to end; the runbooks here assume it exists and pick up at the moment a threshold trips.

The binding has three moving parts. First, a detector: a PromQL rule (or an OpenTelemetry-derived metric) that turns raw counters into a rate or quantile the way the geospatial metric taxonomy for ETL prescribes, so the alert fires on a trend, not a single unlucky sample. Second, an SLO budget: each spatial layer publishes an objective — features clearing the trust boundary on first pass, tiles served within a staleness window, a layer meeting its freshness target — and the alert’s for duration and severity are derived from how fast the incident is burning that budget. Third, a runbook link carried on the alert annotation, so the responder opens the right page in one click rather than pattern-matching from memory at 3 a.m.

That last point is why every alert rule below carries a runbook_url annotation. An alert without a runbook is a notification; an alert with one is the first step of a procedure. When you wire the detectors from this reference, keep the severity tiers honest: a warning says the budget is eroding and a human should look within the hour, while a critical says the budget is being spent now and the remediation steps should start immediately. Teams choosing between collection strategies for these detectors will find the trade-offs weighed in choosing spatial observability tooling, which sits alongside these runbooks under the same program.

The four-stage lifecycle maps directly onto the two numbers on-call leadership tracks. Detect governs mean time to acknowledge: a detector that fires on a trend with a named layer and source label collapses the gap between “something is wrong” and “here is exactly what is wrong.” Triage and remediate together govern mean time to resolution, and the reason these runbooks front-load triage is that in spatial systems the wrong remediation is expensive — halt ingestion on a lag incident and you starve the catch-up; reroute reads on a corruption incident and you serve wrong data faster. Post-mortem governs recurrence, the number nobody puts on a dashboard but everyone feels. A practice that remediates quickly but never hardens detection will remediate the same incident forever; a practice that closes the loop shrinks its own paging volume over time. Each of the five runbooks therefore ends not with “resolved” but with a concrete detector change, and this reference treats a post-mortem that changes nothing as an incident that is not actually closed.

A second architectural concern is keeping the detector layer itself observable. The metrics that open these runbooks are themselves subject to the failure modes they detect — a collector restart drops a gauge to zero, a cardinality explosion on a per-layer label set silently drops series, a scrape timeout leaves a stale sample that reads as healthy. Every runbook’s first triage step is therefore to confirm the signal is real before acting on it, and the geospatial observability architecture and fundamentals reference covers the meta-monitoring — up counters, scrape-duration histograms, series-count gauges — that makes that confirmation a two-second check rather than a debugging detour. A runbook that trusts its input blindly will occasionally spend an hour remediating a metric bug.

Failure-mode map

Five failure modes cover the overwhelming majority of spatial pipeline incidents. Each has one primary signal metric that should be the first line of the alert, a default severity that reflects how fast it corrupts downstream state, and a dedicated runbook. The table is the index into the rest of this reference — read a symptom, follow the runbook.

Two of these — topology corruption and CRS cascade — corrupt data at rest and default to critical because every minute of continued ingestion widens the blast radius. The other three degrade freshness rather than correctness and start at warning, escalating to critical only as they burn through their staleness budget. Severity is not a property of the metric; it is a property of how much objective the incident is spending, which is why the two queue-and-lag modes carry a tiered severity in the table.

The map also encodes the single most important triage decision — whether to stop the flow or keep it running — because that decision splits cleanly along the correctness/freshness line. For the two correctness failures, continued flow is the enemy: halting ingestion for the affected layer is usually the first remediation, because every batch that lands while the fault is live adds to the cleanup. For the three freshness failures, the flow is the cure: the data is correct and only late, so stopping ingestion or replication makes the incident worse by removing the very updates that would let the layer catch up. Reading a symptom off the map therefore does two jobs at once — it points at the right runbook and it pre-loads the correct instinct about whether the responder’s first move is a brake or an accelerator. Getting that instinct backwards is the most common way a spatial incident is prolonged rather than resolved.

Generic runbook template

Every runbook in this reference instantiates the same skeleton. Keeping the shape constant means a responder who has worked one incident already knows the shape of the next, and it makes the pages diffable when you fork one to cover a new failure mode. The template below is the canonical form; each field is filled with failure-mode-specific signals and commands downstream.

# runbook.template.yaml — one file per failure mode, checked in beside the alert rules
runbook:
  id: spatial-<failure-mode>
  signal:
    metric: gis.spatial.<primary_metric>       # the first line of the alert
    detector: |                                 # PromQL that opens this runbook
      <rate-or-quantile expression> > <threshold>
    severity_ladder:
      warning:  { burn_rate: "budget eroding", response: "investigate < 1h" }
      critical: { burn_rate: "budget spending now", response: "remediate now" }
  detect:
    - confirm the signal is real, not a scrape gap or a single bad sample
    - pull the label set (layer, region, source, srid) that scopes the fault
  triage:
    - scope blast radius: which layers/regions/consumers are affected
    - decide: contain-in-place vs reroute vs halt ingestion
    - assign incident severity from the budget burn, not the raw count
  remediate:
    - stop the bleed (quarantine / shed / reroute / pause the offending stage)
    - repair or catch up the affected data
    - verify the signal metric returns below threshold and stays there
  postmortem:
    - reconstruct the timeline from the metric and the traces
    - identify the earliest signal that could have fired sooner
    - harden: add or tighten a detector so recurrence pages earlier

Two rules keep the template from rotting. Never remediate before triage has scoped the blast radius — a fix applied to the wrong layer wastes the budget you are trying to protect. And never close an incident without the post-mortem step actually changing a threshold or adding a detector; a runbook that ends at “remediated” guarantees the same page next week.

Threshold and alerting

The detectors that open these runbooks share a design philosophy: rate over raw count, quantile over mean, and a for duration long enough to survive a scrape gap but short enough to catch a burst. The composite burn-rate view is what decides severity. For a freshness or lag objective with budget $B$ (the allowed staleness in seconds) and observed staleness $s$, the burn ratio

$$\rho = \frac{s}{B}$$

crosses $1$ exactly when the objective is breached; a warning fires around $\rho \geq 0.7$ and a critical around $\rho \geq 1$. The following rules encode the top of each failure mode’s detector; the deep runbooks refine them per layer.

groups:
  - name: spatial-incident-runbooks
    rules:
      # Topology corruption — bursty invalid-geometry rate, not a daily total.
      - alert: SpatialTopologyErrorRateHigh
        expr: |
          sum(rate(gis_spatial_topology_error_total[5m])) by (layer)
            / clamp_min(sum(rate(gis_etl_features_ingested_total[5m])) by (layer), 1)
            > 0.02
        for: 10m
        labels: { severity: critical }
        annotations:
          runbook_url: "/spatial-incident-response-and-tooling/spatial-pipeline-incident-runbooks/topology-corruption-incident-runbook/"

      # CRS cascade — any silent projection change is page-worthy.
      - alert: SpatialCRSDriftDetected
        expr: increase(gis_spatial_crs_drift_total[15m]) > 0
        for: 0m
        labels: { severity: critical }
        annotations:
          runbook_url: "/spatial-incident-response-and-tooling/spatial-pipeline-incident-runbooks/crs-cascade-failure-runbook/"

      # Tile queue overflow — depth relative to drain rate, tiered.
      - alert: SpatialTileQueueBacklog
        expr: gis_spatial_tile_queue_depth > 5000
        for: 5m
        labels: { severity: warning }
        annotations:
          runbook_url: "/spatial-incident-response-and-tooling/spatial-pipeline-incident-runbooks/tile-publish-queue-overflow-runbook/"

      # Replication lag — per region and layer, worst-offender surfaces first.
      - alert: SpatialReplicationLagHigh
        expr: max(gis_spatial_replication_lag) by (region, layer) > 60
        for: 5m
        labels: { severity: critical }
        annotations:
          runbook_url: "/spatial-incident-response-and-tooling/spatial-pipeline-incident-runbooks/cross-region-replication-lag-runbook/"

      # Freshness SLA — burn ratio against the per-layer budget.
      - alert: SpatialFreshnessBudgetBurn
        expr: |
          spatial_freshness_seconds
            / on(layer) group_left spatial_freshness_budget_seconds > 0.7
        for: 10m
        labels: { severity: warning }
        annotations:
          runbook_url: "/spatial-incident-response-and-tooling/spatial-pipeline-incident-runbooks/freshness-sla-breach-runbook/"

Note the shared discipline: every rule is a ratio or a rate, every rule carries the runbook URL, and the two data-corruption modes fire at for: 0m or a short window because their blast radius grows with every ingested batch. The freshness and queue modes tolerate a longer for because a brief spike inside budget is not yet an incident. For the instrument choices behind these names — counters versus gauges, cardinality budgets, and how to keep per-layer labels from exploding — return to the geospatial metric taxonomy for ETL.

The for duration is doing more work than it appears. Set it too short and every scrape gap or transient blip pages the on-call; set it too long and a fast-burning incident spends most of its budget before the alert even fires. The resolution used across mature spatial deployments is multi-window burn-rate alerting: pair a fast, high-threshold window that catches acute incidents within minutes with a slow, low-threshold window that catches the slow leaks a single window would miss. A freshness layer, for instance, benefits from a rule that pages when the budget is burning fast over five minutes or burning gently over an hour — the first catches a sudden stall, the second catches a layer that has been quietly drifting stale all afternoon. The runbooks that follow each note where their failure mode favours a fast window (topology and CRS, which arrive as bursts) versus a dual window (freshness, queue depth, and replication lag, which can either spike or creep). The point of encoding this in the detector rather than in the responder’s judgment is that a well-tuned window turns the alert timestamp itself into triage information: an incident that tripped the fast window is acute and needs a brake now, while one that only tripped the slow window has been building and rewards a look at what changed hours ago.

Failure modes in depth

Each failure mode gets its own page because the detect-and-repair mechanics diverge sharply once you are past the shared skeleton. A short orientation to each follows so you know which page to open before the symptoms are fully legible.

Topology corruption is a spike in invalid geometries — self-intersections, unclosed rings, duplicate nodes — reaching a trust boundary and failing silently inside spatial predicates. The signal is gis.spatial.topology_error_total climbing on one layer, and the remediation leans on ST_IsValidReason, ST_MakeValid, and a quarantine lane. Work the topology corruption incident runbook, which pairs with the standing geometry validity and topology checks reference.

CRS cascade failure starts with a single upstream projection change and propagates into wrong distances, wrong areas, and empty joins across every downstream layer that consumed the mistagged feature. The signal is gis.spatial.crs_drift_total, and containment matters more than repair because the corruption spreads by join. Follow the CRS cascade failure runbook.

Tile publish queue overflow is a rendering or publish backlog — gis.spatial.tile_queue_depth growing faster than it drains — that leaves consumers looking at stale tiles even though the underlying data is current. The tile publish queue overflow runbook covers load-shedding and cache invalidation.

Cross-region replication lag is one region serving data older than its peers, isolated to a specific layer by gis.spatial.replication_lag keyed on region and layer. The cross-region replication lag runbook covers read-rerouting and catch-up, and it builds on the monitoring topology for multi-region GIS design.

Freshness SLA breach is a layer crossing its staleness budget with no structural fault — the data is valid, it is simply late. The freshness SLA breach runbook locates the stalled stage, and it extends the standing practice in tracking spatial data freshness SLAs.

Triage checklist

Before opening a specific runbook, the responder should run this scoping pass. It is failure-mode agnostic and takes under two minutes; its job is to confirm the signal is real and to hand the right label set to the deep runbook.

1. Confirm the signal is not an artifact. A missing scrape or a collector restart can look like a metric dropping to zero or spiking. Check that the underlying series has continuous samples across the alert window before treating the number as an incident. A gap is an observability bug, not a spatial one.
2. Pull the scoping labels. Every signal metric here carries the labels that localize the fault — layer, region, source, srid. Read them off the firing series first; they tell you which downstream consumers are exposed and often name the failure mode outright.
3. Decide contain versus repair versus halt. Data-corruption modes (topology, CRS) usually mean halt ingestion for the affected layer before repairing, because continued intake widens the damage. Freshness and lag modes usually mean reroute reads and catch up in place, because the data is correct, only late.
4. Set severity from the budget, not the count. A thousand invalid geometries on a throwaway staging layer is a warning; a hundred on an authoritative cadastral layer feeding compliance reports is a critical. Anchor severity to the objective the incident is spending, exactly as the defining spatial data trust boundaries work defines each layer’s guarantees.
5. Open the matching runbook and start the timeline. Note the wall-clock time the signal first crossed threshold — not when you were paged — because the post-mortem’s “could this have fired sooner” question depends on that gap.

Work the checklist, follow the failure-mode map to the right page, and let the post-mortem step feed one sharpened threshold back into detection. That feedback loop is the whole point: a spatial incident response practice is measured not by how fast it remediates, but by how rarely the same failure mode pages twice.

Related

• Spatial incident response and tooling — the parent program that frames instrumentation, tooling choices, and on-call practice for geospatial workflows.
• Choosing spatial observability tooling — how to pick the collectors and stores that feed these runbook detectors.
• Topology corruption incident runbook — detect and remediate a spike in invalid geometries at the trust boundary.
• CRS cascade failure runbook — contain an upstream projection change before it corrupts downstream joins.
• Geospatial observability architecture and fundamentals — the instrumentation and SLO layer these runbooks assume.