Spatial Incident Response & Observability Tooling

When a spatial pipeline breaks at three in the morning, the failure rarely announces itself as an exception — a tile server keeps returning 200 OK while serving holes, a reprojection job succeeds while quietly shifting features into the wrong hemisphere, and a GiST index answers every query while its bloat ratio climbs past the point where scans no longer prune. This guide is the operational reference for the moment things go wrong: how to select the right instrumentation, how to shape alerts that fire on the signals that predict incidents, and how to drive a spatial outage from detection through post-mortem. It complements the architectural groundwork in geospatial observability architecture and fundamentals and the staleness-focused program in spatial data freshness and quality metrics, turning their static instrumentation into a living response discipline.

The spine above frames the whole domain. A tooling decision on the left determines what signals you even have to work with, and everything downstream — detection sensitivity, triage speed, remediation options — is constrained by that choice. The four lifecycle stages then run in order, with a fallback lane that keeps the pipeline serving during remediation and a feedback loop that turns every incident into a tighter detection threshold. The rest of this guide walks each stage in turn, starting with the trade-offs that govern which tools you reach for.

Core Concepts: Tooling Trade-offs and the Incident Lifecycle

Spatial incident response sits on two independent axes, and confusing them is the most common reason a well-instrumented team still runs blind during an outage. The first axis is tooling capability — what your instrumentation can actually see. The second is lifecycle stage — where in the detect-to-post-mortem arc you currently stand. A tool that excels at one stage can be useless at another: distributed traces are invaluable during triage because they localize a regression to a single reprojection span, but they are a poor detection primitive because you cannot alert cheaply on a trace you sampled away.

The tooling trade-off axes

When you evaluate instrumentation for a spatial stack, four axes decide the fit. These are the axes that the tooling-selection decision framework formalizes into a repeatable scoring exercise:

• Granularity versus cost. Per-feature attributes (source CRS, vertex count, geometry type) make triage precise but explode cardinality. A gis.spatial.vertex_count histogram tagged by feature_id is unqueryable and ruinously expensive; the same histogram tagged by layer is cheap and still actionable.
• Push versus pull. Long-lived tile servers scrape cleanly, but ephemeral GDAL batch jobs finish before any scraper reaches them. The push versus pull decision for spatial pipelines turns on exactly this lifetime mismatch.
• Traces versus metrics. Traces answer “where in this one request did the reprojection stall”; metrics answer “how many reprojections stalled this hour”. You need both, and the choice between an OpenTelemetry collector and a Prometheus exporter shapes which comes first, as the OpenTelemetry versus Prometheus-native comparison for PostGIS examines.
• Storage-engine visibility. Some incidents live inside the database — GiST page splits, dead-tuple bloat, BRIN summary drift. Detecting them means querying pg_stat_user_indexes and pgstattuple, a capability orthogonal to whatever ships your application spans.

The incident lifecycle

Every spatial outage moves through four stages, and each stage has a different question and a different primary signal:

1. Detect. The question is “is something wrong”. The signal is an SLO burn-rate alert or a statistical anomaly on a leading indicator — gis.spatial.crs_drift_total incrementing, a freshness budget draining faster than its refill rate. Detection must be cheap and continuous, which is why it leans on pre-aggregated metrics rather than traces.
2. Triage. The question is “which stage broke”. Here you pivot from the aggregate alert into per-stage signals: transform-duration tails, topology-error counts by source, replication lag by layer. Traces earn their keep at this stage by localizing the fault to a single span.
3. Remediate. The question is “how do we stop the bleeding”. This is where fallback chains fire — dropping to a cached simplified geometry, opening a circuit breaker — and where a repair (ST_MakeValid, a REINDEX) is applied. The relevant playbooks live in the spatial pipeline incident runbooks.
4. Post-mortem. The question is “why did detection lag reality”. Every incident should end by tightening a threshold, adding a leading indicator, or narrowing a trust boundary so the same class of failure detects earlier next time.

The feedback arrow in the diagram is the point of the whole discipline: an incident that does not sharpen your detection is an incident you will run again. Trust boundaries are the enforcement mechanism for that sharpening — the same spatial data trust boundaries that gate ingestion become, after a post-mortem, the place you add the check that would have caught the fault at the door.

Metric Taxonomy for Incident and Tooling Signals

Incident response needs its own metric family distinct from the ingestion-quality metrics in the geospatial metric taxonomy for ETL. These signals measure the health of the response process itself and the operational state of the tooling, not the quality of the data flowing through. The canonical families are below.

Two derived quantities anchor the SLO discussion because thresholds are set against them rather than against raw counters. The mean time to resolution across an incident window is the straightforward mean of the resolution durations, but it hides bimodality — trivial page-and-ack incidents and genuine multi-hour outages average to a misleading middle. Prefer the p90:

$$\text{MTTR}_{p90} = Q_{0.90}\big(\{\, t^{\text{resolve}}_i - t^{\text{detect}}_i \,\}_{i \in W}\big)$$

The more revealing metric is the detection efficiency — the fraction of an incident’s total duration that elapsed before anyone was paged. High detection lag means your monitoring is a lagging indicator masquerading as a leading one:

$$\eta_{\text{detect}} = 1 - \frac{t^{\text{detect}} - t^{\text{onset}}}{t^{\text{resolve}} - t^{\text{onset}}}$$

where $t^{\text{onset}}$ is reconstructed during the post-mortem from the first anomalous data point. A pipeline whose $\eta_{\text{detect}}$ trends toward zero is one where incidents are discovered by users, not instruments — the signal that your detection thresholds need the feedback loop the lifecycle diagram promised. Keeping the dimensions on these metrics bounded matters just as much here as at ingestion; the same discipline described in bounding spatial metric tag cardinality applies to incident labels, since a reason dimension that admits free-text error strings will melt your time-series database mid-incident.

Instrumentation Patterns

Incident-grade instrumentation differs from steady-state instrumentation in one respect: it must survive the incident. When a pipeline is failing, telemetry volume spikes exactly when your backend is least able to absorb it, so the instrumentation has to degrade gracefully rather than amplify the outage. The patterns below assume the OpenTelemetry integration for GIS pipelines collector is already in place and focus on the incident-specific additions.

The first pattern is a dedicated incident-signal exporter that never gets sampled away. Detection metrics — fallback counts, circuit-breaker state, drift counters — must be exempt from the probabilistic sampling that heavy transforms are subject to, because a sampled-away fallback event is a blind spot precisely when it matters:

# otel-collector-config.yaml  (contrib build) — incident-signal carve-out
processors:
  # Route incident-critical metrics around the sampler entirely.
  filter/incident_keep:
    metrics:
      include:
        match_type: strict
        metric_names:
          - gis.spatial.fallback_total
          - gis.spatial.circuit_open_total
          - gis.spatial.crs_drift_total
          - gis.spatial.index_bloat_ratio
  transform/incident_labels:
    metric_statements:
      - context: datapoint
        statements:
          # Collapse high-cardinality reasons into a bounded enum before export.
          - set(attributes["reason"], "other") where attributes["reason"] == nil

service:
  pipelines:
    metrics/incident:
      receivers: [otlp]
      processors: [filter/incident_keep, transform/incident_labels]
      exporters: [prometheus]

The second pattern instruments the response process itself, so that post-mortems have data instead of anecdotes. Wrapping each remediation step in a span with a stable runbook and step attribute turns gis.etl.runbook_step_duration into a map of where operators actually lose time:

from opentelemetry import trace, metrics

tracer = trace.get_tracer("gis.incident.runbook")
meter = metrics.get_meter("gis.incident.runbook")
step_hist = meter.create_histogram("gis.etl.runbook_step_duration", unit="s")

def run_step(runbook: str, step: str, action):
    with tracer.start_as_current_span(f"runbook.{step}") as span:
        span.set_attribute("runbook", runbook)
        span.set_attribute("step", step)
        import time
        started = time.monotonic()
        try:
            return action()
        finally:
            elapsed = time.monotonic() - started
            step_hist.record(elapsed, {"runbook": runbook, "step": step})
            span.set_attribute("runbook.step_seconds", round(elapsed, 3))

The third pattern is storage-engine introspection, which no application-level span can provide. Index bloat — the slow-burn fault behind a surprising number of “the database got slow” incidents — is only visible by querying the catalog directly, then feeding the result into gis.spatial.index_bloat_ratio:

-- Emit index bloat as a gauge; a scraper lifts this into gis.spatial.index_bloat_ratio.
SELECT
  c.relname                          AS index,
  am.amname                          AS access_method,
  pg_relation_size(c.oid)            AS index_bytes,
  s.idx_scan                         AS scans,
  (pgstattuple(c.oid)).dead_tuple_percent / 100.0 AS bloat_ratio
FROM pg_class c
JOIN pg_am am        ON am.oid = c.relam
JOIN pg_index i      ON i.indexrelid = c.oid
JOIN pg_stat_user_indexes s ON s.indexrelid = c.oid
WHERE am.amname IN ('gist', 'brin');

The choice between the trace-forward OpenTelemetry path and the pull-based Prometheus path is not cosmetic — it changes which of these three patterns is cheap and which is awkward, which is why the instrumentation-comparison decision page treats it as the first fork.

Alerting and SLO Design

Alerts for incident response are not the same as alerts for data quality. A data-quality alert says “this batch is bad”; an incident alert says “the failure rate is now eroding a reliability commitment faster than the budget can absorb”. The distinction is burn rate. Rather than paging on a static threshold, page when the rate of budget consumption implies the SLO will be missed within a window you can still act on — the multi-window, multi-burn-rate pattern, applied to spatial signals.

groups:
  - name: spatial-incident-response
    rules:
      # Fast burn: fallback rate consuming the availability budget in ~1 hour.
      - alert: SpatialFallbackFastBurn
        expr: |
          (
            sum(rate(gis_spatial_fallback_total{tier!="primary"}[5m])) by (service)
            / clamp_min(sum(rate(gis_spatial_requests_total[5m])) by (service), 1)
          ) > (14.4 * 0.01)
        for: 2m
        labels: { severity: critical }
        annotations:
          summary: "Fast fallback burn on {{ $labels.service }} — 1% budget in ~1h"

      # Slow burn: sustained degradation that will breach the SLO over a day.
      - alert: SpatialFallbackSlowBurn
        expr: |
          (
            sum(rate(gis_spatial_fallback_total{tier!="primary"}[1h])) by (service)
            / clamp_min(sum(rate(gis_spatial_requests_total[1h])) by (service), 1)
          ) > (3 * 0.01)
        for: 15m
        labels: { severity: warning }

      # Index bloat crossing the point where scans stop pruning — remediate before it pages.
      - alert: SpatialIndexBloatCritical
        expr: gis_spatial_index_bloat_ratio{access_method="gist"} > 0.40
        for: 30m
        labels: { severity: warning }
        annotations:
          summary: "GiST index {{ $labels.index }} bloat >40% — schedule REINDEX"

      # Detection efficiency regression: incidents found late over the last day.
      - alert: SpatialDetectionLagRising
        expr: |
          histogram_quantile(0.90,
            sum(rate(gis_spatial_detection_lag_bucket[6h])) by (le, signal)) > 300
        for: 0m
        labels: { severity: warning }

SLO design for spatial incident response should target three outcomes, each with its own error budget. The first is availability of full-fidelity spatial responses — the fraction of requests served without a fallback, which the fast/slow burn rules above defend. The second is detection efficiency, an SLO on $\eta_{\text{detect}}$ itself: at least 80% of incidents should be instrument-detected before a user reports them. The third is remediation latency, an SLO on gis.spatial.incident_mttr p90. Sharing a single error budget across these forces the honest trade-off between building more fallbacks and building better detection. When the availability budget is healthy but detection efficiency is poor, the fix is not more redundancy — it is a leading indicator, often borrowed from the freshness program’s practice of tracking spatial data freshness SLAs, where a draining budget is itself the earliest warning.

The alert thresholds themselves deserve a spatial-specific caveat: the burn-rate multipliers of 14.4 and 3 in the rules above assume a well-behaved request rate, and spatial workloads routinely violate that assumption. A nightly tile bake or a bulk cadastral import can spike the denominator of a fallback ratio by an order of magnitude for an hour, quietly diluting a genuine degradation below the threshold. Guard against this by evaluating burn rates against a rolling baseline of the same weekday-hour rather than an absolute request count, and by exempting known batch windows from the fast-burn rule while keeping the slow-burn rule active — a batch window is exactly when a latent fault is most likely to surface, so you want the sustained signal even while you suppress the twitchy one. The same non-stationarity is why a raw count threshold on any spatial signal ages badly: what was a critical fallback rate at last quarter’s traffic is background noise at this quarter’s, and a threshold that is not periodically rebaselined slowly stops firing.

Error-budget policy closes the loop between alerting and the post-mortem stage. When a shared budget is exhausted, the policy should specify a concrete response tied to which outcome drained it: an availability breach freezes risky spatial deployments until the budget recovers, a detection-efficiency breach mandates adding a leading indicator before the next feature ships, and a remediation-latency breach triggers a runbook review. Writing this policy down before an incident is what prevents the reflexive but wrong response of pouring engineering into more redundancy when the actual deficit is in detection — the two are funded from the same budget but fixed by opposite investments, and only an explicit policy keeps that distinction visible under the pressure of a live outage.

Operational Debugging Workflow

When an alert fires, the goal of triage is to collapse the search space from “somewhere in the spatial stack” to “this stage, this layer, this source” as fast as possible. Work the checklist in order; it maps directly onto the lifecycle stages and refuses to skip ahead to remediation before the fault is localized.

1. Confirm the burn is real, not a cardinality artifact. Before anything else, check that the alert is not firing on a label explosion. A sudden new value in the reason or source dimension can spike a rate query without any underlying data fault. Cross-check the raw request rate; if it is flat while the ratio jumps, suspect the metric pipeline, not the data.
2. Localize the stage with span durations. Pull the p99 of gis.etl.transform_duration and the topology-error rate side by side. A rising transform tail with a flat error rate points at a compute-bound reprojection or a degraded query plan; a flat tail with a rising error rate points at an upstream data defect crossing the trust boundary.
3. Attribute to a source. Break gis.spatial.crs_drift_total and gis.spatial.topology_error_total down by source. A spike isolated to one provider is an upstream regression — reach for the relevant runbook rather than touching your own code. Confirm with ST_IsValidReason() on the quarantined batch and map the reason class back to the provider.
4. Check the storage engine. If the symptom is latency rather than correctness, query gis.spatial.index_bloat_ratio before assuming a code regression. A GiST index past 40% dead tuples will slow every predicate query while every application span looks nominal — the fault is below the application, in the index, and the fix is a REINDEX not a redeploy. The mechanics of catching this early are in detecting GiST index bloat from bulk inserts.
5. Decide fallback versus repair. If the fault is upstream and ongoing, step down a fallback tier to keep serving and buy time. If the fault is a discrete corrupt batch, quarantine and repair it. Never apply an in-place repair under load without first opening the fallback — a REINDEX on a hot spatial table can lock out the very queries the incident is already starving.
6. Record the timeline for the post-mortem. Capture $t^{\text{onset}}$, $t^{\text{detect}}$, and $t^{\text{resolve}}$ as the incident unfolds. Reconstructing detection lag after the fact from memory is how detection efficiency stays invisible.

Failure Modes and Fallback

Spatial pipelines fail in structured ways, and each mode has a fallback that preserves continuity while emitting a degradation signal rather than a silent substitution. The table pairs the common incident classes with their detection signal and their defined fallback. These fallbacks extend the API-level chain documented in fallback chains for spatial API failures into the pipeline interior.

Every row is a metric first and an action second — the rate of any non-primary tier is itself an SLO signal even when no request technically failed. Two of these modes cut across the other reference guides: the region-routing fallback ties directly into monitoring topology for multi-region GIS, and the freshness-breach reprocess is the operational tail of the freshness program. For the canonical structural rules that define what “corrupt” even means, keep the OGC Simple Features Specification and the PostGIS documentation within reach; a fallback that repairs geometry without honoring those rules trades one incident for another.

Related

• Choosing spatial observability tooling — the decision framework for selecting instrumentation, indexes, and collection models for a spatial stack.
• Spatial pipeline incident runbooks — step-by-step playbooks for topology corruption, CRS cascades, queue overflow, replication lag, and freshness breaches.
• Geospatial observability architecture and fundamentals — the architectural reference this response discipline instruments and defends.
• Spatial data freshness and quality metrics — the companion program whose draining freshness budgets are among the earliest incident signals.
• Fallback chains for spatial API failures — tiered graceful degradation that the pipeline-interior fallbacks here extend.