Topology Corruption Incident Runbook

When gis.spatial.topology_error_total starts climbing on a single layer, you are watching structurally broken geometries — self-intersections, unclosed rings, spikes, duplicate consecutive nodes — cross a trust boundary and enter a store that downstream queries treat as clean. This runbook is the step-by-step procedure for that specific incident: confirming the spike is real, isolating the source batch, quarantining the bad features, repairing what can be repaired, and closing the loop so the same defect pages earlier next time. It belongs to the spatial pipeline incident runbooks collection within the broader spatial incident response and tooling program.

Topology corruption detect-to-repair flow An alert box for an invalid-geometry rate spike leads to a batch-isolation step keyed by source and layer. From there a decision splits features: invalid ones go down into a quarantine lane where ST_IsValidReason diagnoses and ST_MakeValid repairs them, then a verify step re-runs ST_IsValid; valid ones flow to the clean store. Repaired features re-enter the clean store after passing verification. Rate spike topology_error_total Isolate batch by source · layer valid? ST_IsValid Quarantine ST_IsValidReason Repair ST_MakeValid Clean store verified geoms no yes verify

Problem Framing

Topology corruption is uniquely dangerous because it is invisible to the checks most pipelines already run. A self-intersecting polygon has the right geometry type, a plausible bounding box, a non-null value, and a row that increments every count you track — so schema validation, row-count reconciliation, and null checks all pass. The defect only manifests when a spatial predicate touches it: ST_Intersection returns an empty result, ST_Union throws a GEOS TopologyException, or a spatial join silently drops the malformed side and returns fewer rows than the feature count implies. By then the corrupt feature may already sit in a published layer feeding analytics or tiles.

The incident this runbook addresses is not a single bad geometry — those happen constantly and are handled by the standing gate described in geometry validity and topology checks. It is a spike: a step change in the invalid rate that signals a systemic upstream regression — a vendor switching export formats, a simplification step introduced with too aggressive a tolerance, a coordinate-precision truncation that collapses near-coincident vertices into self-touching rings. The distinguishing feature is that the invalid count jumps for one source label while others stay flat, which is exactly what makes it triageable.

The blast radius grows with every ingested batch, which is why this failure mode defaults to critical. Unlike a freshness delay, corrupt topology does not heal by waiting; it accumulates in the store and each downstream join propagates the wrong answer further. The correct instinct is to stop ingestion for the affected layer first and repair second — the opposite of the wait-and-see posture appropriate to a lag incident.

Detect

The detector is a rejection rate per layer, never a raw count. A raw count grows with volume and drowns a real spike inside normal traffic; the rate isolates the regression. The following rule opens this runbook when more than two percent of ingested features on a layer fail validity over a rolling five-minute window.

groups:
  - name: topology-corruption
    rules:
      - alert: TopologyErrorRateSpike
        expr: |
          sum(rate(gis_spatial_topology_error_total[5m])) by (layer, source)
            / clamp_min(
                sum(rate(gis_etl_features_ingested_total[5m])) by (layer, source), 1)
            > 0.02
        for: 10m
        labels: { severity: critical }
        annotations:
          summary: "Topology error rate {{ $value | humanizePercentage }} on {{ $labels.layer }} from {{ $labels.source }}"
          runbook_url: "/spatial-incident-response-and-tooling/spatial-pipeline-incident-runbooks/topology-corruption-incident-runbook/"

Keeping the source label on the rate is what turns detection into triage — when the alert fires, the offending upstream provider is already named in the annotation. Confirm the spike is real before acting: check that gis_etl_features_ingested_total has continuous samples across the window (a scrape gap can make the denominator collapse and inflate the ratio), and glance at the reason breakdown emitted alongside the counter to see whether one defect class dominates.

Triage

Once the alert is confirmed real, scope the damage in three moves. First, read the failure-reason distribution — the counter should carry a reason label mirroring ST_IsValidReason output, so you can tell a Self-intersection flood (usually a simplification or precision regression) from Ring Self-intersection or Too few points (usually a format-conversion bug). Second, name the entry batch: correlate the spike’s start time against ingestion job runs for that source and identify the specific batch or partition. Third, decide the containment posture.

-- Reason breakdown for the affected layer over the incident window
SELECT ST_IsValidReason(geom) AS reason,
       COUNT(*)               AS features,
       MIN(ingested_at)       AS first_seen
FROM staging.raw_geospatial_feed
WHERE layer = 'parcels_authoritative'
  AND NOT ST_IsValid(geom)
  AND ingested_at >= now() - interval '30 minutes'
GROUP BY 1
ORDER BY 2 DESC;

Because topology corruption corrupts data at rest, the default containment decision is halt ingestion for the affected layer, not merely alert. Pause the ingestion job for that source so continued intake stops widening the blast radius, then decide whether the already-landed features need to be quarantined out of the published layer or can be repaired in place. If the corrupt features have not yet been promoted past the trust boundary — they are still in staging — you have contained the incident already and the rest is cleanup.

Remediate

Remediation is a quarantine-then-repair sequence. Move the invalid features into an isolated table keyed by their original identifier and the incident timestamp so nothing is lost and every action is auditable, then repair with ST_MakeValid under an explicit guard so a repair never silently drops a feature to an empty geometry or violates a single-type column constraint.

-- 1. Quarantine invalid features out of the published layer
CREATE TABLE IF NOT EXISTS incident.topology_quarantine (LIKE prod.parcels INCLUDING ALL);

WITH bad AS (
  DELETE FROM prod.parcels p
  WHERE NOT ST_IsValid(p.geom)
  RETURNING p.*
)
INSERT INTO incident.topology_quarantine
SELECT *, ST_IsValidReason(geom) AS reason, now() AS quarantined_at FROM bad;

-- 2. Repair under guard — never replace a feature with emptiness
UPDATE incident.topology_quarantine
SET geom = ST_MakeValid(geom)
WHERE NOT ST_IsValid(geom)
  AND NOT ST_IsEmpty(ST_MakeValid(geom));

-- 3. Promote only rows that now pass and keep their original type
INSERT INTO prod.parcels
SELECT (t).*
FROM incident.topology_quarantine t
WHERE ST_IsValid(t.geom)
  AND GeometryType(t.geom) = 'MULTIPOLYGON';

Features that ST_MakeValid cannot repair without changing type or collapsing to empty stay in quarantine for manual review — those are the ones worth a human eye, and their count is a metric in its own right. Once repaired features are promoted, resume ingestion for the source only after adding a pre-ingestion ST_IsValid gate if one was missing, so the same batch shape cannot re-enter uncaught.

Post-mortem and Verification

Verify the incident is closed by three concrete facts, not by the absence of new pages. The rate detector must return below its two-percent threshold and stay there across a full window; the quarantine table must account for every feature that was removed from the published layer; and a spot re-run of ST_IsValid over the affected layer must return zero invalids. Only then is the layer trustworthy for downstream joins again.

-- Closure check: the published layer holds zero invalid geometries
SELECT COUNT(*) AS remaining_invalid
FROM prod.parcels
WHERE NOT ST_IsValid(geom);
-- Expect: 0

The post-mortem must change something. Reconstruct the timeline from the counter’s first threshold crossing to resolution, then find the earliest signal that could have fired — often the reason-labeled counter was already trending upward before the rate crossed two percent. Harden detection by lowering the threshold for that specific source if it is chronically noisy, or by adding a per-reason alert so a Self-intersection flood pages independently of the aggregate. The incident is not closed until a detector has been tightened; a topology corruption that recurs on the same source with the same reason means the loop was never fed.

Gotchas

ST_MakeValid can change geometry type. Repairing a self-intersecting polygon may yield a MULTIPOLYGON, a GEOMETRYCOLLECTION, or an empty geometry. Promoting repaired rows without a type filter can violate a column constraint or inject a collection where consumers expect a polygon. Always filter the promotion step by GeometryType, as the remediation SQL does.

A repair can silently shrink coverage. ST_MakeValid resolving a self-overlap may discard the overlapping sliver, reducing total area. After a bulk repair, re-run a coverage sweep from spatial coverage and extent monitoring so a topology fix does not quietly open a coverage gap.

The spike may be a precision artifact, not bad source data. If an upstream step truncated coordinate precision, near-coincident vertices collapse into self-touching rings that were fine at full precision. Repairing them masks the real bug. Check whether the source geometries validate at full precision before treating the feed itself as corrupt.

FAQ

How is this different from the standing geometry validity gate?

The standing gate in geometry validity and topology checks rejects individual bad geometries continuously as normal operation. This runbook handles the incident — a rate spike signalling a systemic upstream regression that the gate alone will quietly absorb while your quarantine table balloons. The gate is the seatbelt; this runbook is what you do after the crash.

Should I halt the whole pipeline or just the affected layer?

Just the affected source and layer. Topology corruption is almost always isolated to one upstream provider, and halting unrelated layers spends reliability budget you do not need to. The failure-reason and source labels on gis.spatial.topology_error_total are precisely what let you scope the halt narrowly.

Can I auto-repair with ST_MakeValid instead of paging?

For chronically noisy vendor feeds with minor self-intersections, an inline ST_MakeValid step is reasonable and keeps the incident from ever firing. For authoritative layers — cadastral, boundary, compliance — do not auto-repair; a silent geometry rewrite on an authoritative feature is itself a data-integrity incident. Quarantine and review instead.

Why rate instead of an absolute invalid count?

A raw count scales with ingestion volume, so a busy layer always looks worse than a quiet one and a real regression hides inside normal noise. The per-layer rejection rate normalizes for volume, which is why the detector divides by gis_etl_features_ingested_total rather than alerting on the counter directly.

What if ST_MakeValid leaves features still invalid?

That is expected for a fraction of severely corrupt inputs. Those rows stay quarantined with their ST_IsValidReason recorded for manual triage, and their count is worth its own low-severity alert — a rising unrepairable fraction points at a source producing genuinely malformed data rather than a fixable precision artifact.