Cross-Region Replication Lag Runbook

A region can report a healthy primary, a green connection pool, and a passing health check while quietly serving spatial data hours behind its peers — because the lag lives in one slow-replicating layer, not in the connection. This runbook is the procedure for that incident: reading gis.spatial.replication_lag keyed by region and layer to find the single offending dataset, rerouting reads away from the stale region while it catches up, accelerating the replication backlog, and closing the loop so a per-layer lag surfaces before consumers notice. It is one of the spatial pipeline incident runbooks within the spatial incident response and tooling program.

Per-layer replication lag and read rerouting A primary region on the left streams three layers — roads, parcels, and imagery — to two replica regions on the right. Replica region eu-west-1 shows all layers current. Replica region ap-south-1 shows roads and parcels current but the imagery layer badly lagging at 14 minutes. A rerouting arrow diverts read traffic for the imagery layer away from ap-south-1 to the healthy eu-west-1 while a catch-up arrow feeds the lagging layer. Primary roads · parcels imagery Replica · eu-west-1 roads current · parcels current imagery current Replica · ap-south-1 roads current · parcels current imagery lag 14 min replicate reroute reads

Problem Framing

Replication lag is a staleness incident with a deceptively narrow blast radius. In a multi-region spatial deployment the primary streams changes to regional replicas, and consumers in each region read locally for latency. When one replica falls behind, the data it serves is correct in structure and projection — it is simply old. The trap is that lag is almost never uniform across a region: it concentrates in one layer, usually a large one. A cadastral or imagery layer with millions of features and heavy geometry can saturate the replication stream while small point layers in the same region stay perfectly current. A region-level health check averages this away and reports green.

That per-layer concentration is why the signal must be gis.spatial.replication_lag keyed by both region and layer, exactly as the monitoring topology for multi-region GIS design prescribes. A single gauge per region hides the incident; the two-dimensional gauge names the offending dataset in the alert itself. The related standing practice of alerting on feature-level replication lag is what makes this granularity routine rather than something you scramble to add mid-incident.

The remediation posture differs sharply from a data-corruption incident. Nothing is broken, so there is nothing to quarantine or repair — the data will become correct on its own once replication catches up. The job is to protect consumers from stale reads during the catch-up (reroute them to a current region) and to accelerate the catch-up itself. Halting ingestion, the right move for topology or CRS incidents, would be exactly wrong here: it starves the replica of the very updates it needs to converge.

Detect

The detector is a per-region, per-layer gauge against a lag budget, surfacing the worst offender first. A flat threshold across all layers would either page constantly on the naturally-slow imagery layer or miss a fast layer falling behind, so the budget is per layer.

groups:
  - name: cross-region-replication-lag
    rules:
      - alert: ReplicationLagBudgetExceeded
        expr: |
          max(gis_spatial_replication_lag) by (region, layer)
            > on(layer) group_left() gis_spatial_replication_lag_budget_seconds
        for: 5m
        labels: { severity: critical }
        annotations:
          summary: "{{ $labels.layer }} in {{ $labels.region }} lagging {{ $value | humanizeDuration }}"
          runbook_url: "/spatial-incident-response-and-tooling/spatial-pipeline-incident-runbooks/cross-region-replication-lag-runbook/"
      # Trend guard: page sooner if lag is accelerating, not just high.
      - alert: ReplicationLagAccelerating
        expr: |
          deriv(max(gis_spatial_replication_lag) by (region, layer)[15m:]) > 1
        for: 10m
        labels: { severity: warning }

The group_left() join against a per-layer budget series is what makes a 14-minute lag on imagery a non-event and a 2-minute lag on a live-position layer a page. The acceleration guard matters because a lag that is high but stable is often catching up, while a lag that is climbing will breach the budget no matter its current value. Confirm the lag is measured from the primary’s write time to the replica’s apply time — a clock-skew artifact between regions can fake lag, so cross-check against a monotonic replication position, not wall clocks.

Triage

Triage answers two questions: which layer, and why is it behind. The alert already names the region and layer; the query below ranks the lagging layers and pulls the replication position so you can tell a saturated stream from a stalled one.

# Worst-offending layers in the affected region, ranked by lag
topk(5, max(gis_spatial_replication_lag{region="ap-south-1"}) by (layer))

# Is the replica applying changes at all, or fully stalled?
rate(gis_spatial_replication_applied_bytes_total{region="ap-south-1"}[5m])

If the apply rate is non-zero but the lag is growing, the stream is saturated — the layer is producing changes faster than the replica can apply them, typically a bulk update or a re-index on a heavy layer. If the apply rate is zero, replication has stalled entirely (a broken connection, a conflict, a full disk on the replica), which is a different and more urgent problem. Scope the consumer impact: identify which read traffic actually touches the lagging layer in the affected region, because that is the only traffic that needs rerouting. Everything else in the region is current and should keep reading locally for latency.

Remediate

Remediation is reroute-then-accelerate. Reroute protects consumers immediately; accelerating the stream closes the gap so you can route back.

# 1. REROUTE: send reads for the lagging layer in the affected region to a
#    current region. Leave every other layer reading locally.
routing_override:
  match:
    region: ap-south-1
    layer: imagery
  action: route_to
  target_region: eu-west-1          # nearest region within lag budget
  ttl: until_lag_within_budget

# 2. ACCELERATE the catch-up on the saturated stream:
#    - raise replication apply parallelism for the lagging layer
#    - pause non-critical bulk writes to the primary for that layer
#    - if stalled: clear the conflict / reconnect / free disk, then resume
-- Confirm the lagging layer's replication slot is advancing (PostgreSQL primary)
SELECT slot_name,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)) AS behind
FROM pg_replication_slots
WHERE slot_name = 'ap_south_imagery';
-- 'behind' should shrink over successive checks once catch-up is underway

The reroute must be surgical: override routing only for the affected layer in the affected region, leaving current layers reading locally so you do not trade a staleness problem for a latency problem across the whole region. Accelerate by raising apply parallelism for the lagging layer and, if a bulk write on the primary is the source of the flood, pausing that non-critical write until the replica converges. If triage found a fully stalled stream, the acceleration step is instead a repair — clear the replication conflict, reconnect the slot, or free disk — then let normal replication resume.

Post-mortem and Verification

Verify convergence before removing the reroute: the layer’s lag must fall back within its budget and stay there across a full window, and the replication position (confirmed_flush_lsn behind, or the equivalent) must reach near-zero. Only then route reads back to the local region, because routing back while lag is still above budget re-exposes consumers to stale data.

# Recovery: the previously-lagging layer is back within its per-layer budget
max(gis_spatial_replication_lag{region="ap-south-1", layer="imagery"}) by (layer)
  < gis_spatial_replication_lag_budget_seconds{layer="imagery"}

The post-mortem should establish why the stream saturated. A recurring lag on the same heavy layer during the same bulk-load window is a capacity or scheduling problem, not an incident — the durable fix is to throttle or chunk that bulk write, add apply parallelism permanently, or shift the load off peak. Harden detection by tightening the per-layer budget for the offending layer so it pages before the lag reaches consumer-visible levels, and by keeping the acceleration guard so a climbing lag warns while there is still time to reroute gracefully rather than reactively.

Gotchas

A green region health check hides a lagging layer. Region-level averages wash out a single slow layer. The incident is only visible on the region-plus-layer gauge, which is why feature-level lag alerting is a prerequisite, not a nice-to-have.

Do not halt ingestion to “let the replica catch up.” Halting the primary’s writes for the lagging layer starves replication of the updates it needs and does not help — the replica catches up by applying the stream faster, not by the stream stopping. Pause only non-critical bulk writes, never the layer’s normal feed.

Clock skew fakes lag. If lag is computed from wall-clock timestamps across regions, a skewed replica clock manufactures phantom lag or masks real lag. Measure against a monotonic replication position (LSN or equivalent), and reconcile clocks with the guidance in aligning timestamps across GPS feed sources.

FAQ

Why key the lag metric by layer and not just by region?

Because lag concentrates in one heavy layer while the rest of the region stays current. A per-region gauge averages the slow imagery layer against dozens of fast point layers and reports healthy, hiding an incident that a per-layer gauge names outright in the alert annotation.

Should I reroute the whole region’s reads?

No — reroute only the lagging layer. Other layers in the region are current, and sending them cross-region trades a staleness problem for added latency on data that was fine. The routing override should match the specific region and layer on the firing alert.

Is halting ingestion ever the right move here?

Only pausing a non-critical bulk write that is saturating the stream, and only temporarily. Halting the layer’s normal feed is counterproductive: replication converges by applying changes faster, not by having fewer changes to apply. This is the opposite posture from a data-corruption incident.

How do I know when it is safe to route reads back?

When the layer’s lag has fallen within its budget and held there across a full window, and the replication position shows the replica has nearly caught up. Routing back while lag is still above budget re-exposes consumers to stale reads and can flap the incident.

What if the apply rate is zero?

Then replication has stalled, not merely slowed — a broken slot, a conflict, or a full disk on the replica. That is a repair, not a catch-up: fix the underlying block, resume replication, and expect the lag to drain once the stream flows again.