Alerting on Partial-Region Failures in Spatial Pipelines

Spatial systems fail in patches. A tile pipeline stalls for one metropolitan area while the rest of the country publishes normally; a sensor feed loses one operator’s devices and keeps the other three; a replication link degrades for one continent’s read replica. Aggregate those into a single global series and the failure is arithmetically invisible — one region out of forty moves a national average by two and a half percent, well inside normal noise, and the alert only fires once the problem has spread far enough that it was never subtle to begin with. This guide covers how to build detectors that catch a single-region failure without exploding label cardinality or drowning the pager in per-region noise. It belongs to alert routing and on-call design for spatial pipelines in the spatial incident response and tooling program.

A single failed region hidden inside a healthy national aggregate A grid of twenty region tiles represents a national coverage set. Nineteen tiles are healthy and one tile in the lower middle is failed. Beside the grid, two gauges compare what each detector sees: the national aggregate reads ninety-seven and a half percent healthy, comfortably above a ninety-five percent threshold and therefore silent, while the worst-region detector reads zero percent for the failed tile and fires. An annotation states that averaging across regions is what hides the failure. One failed region out of twenty moves the average by 2.5% — and nothing fires FAILED 20 publish regions · grid_level=6 Aggregate detector avg over all regions = 97.5% healthy silent Worst-region detector min over regions = 0% healthy in one region FIRES Averaging is the bug: the aggregate answers a question nobody asked.

Problem framing: aggregation destroys the signal you need

A national or global aggregate answers “is the system broadly working”. Partial-region failure asks a different question: “is any region not working”. The second question is a minimum over regions, not a mean, and no threshold on the mean recovers it.

The arithmetic is unforgiving. With twenty publish regions, a total outage in one drops the mean to 95%. With forty regions it drops to 97.5%. With a few hundred sensor operators or tile pyramids, a complete regional outage is indistinguishable from rounding. Meanwhile the affected users experience a 100% outage, and they are all in one place — which means the complaints arrive geographically clustered and land as a customer report rather than as an alert.

There is a second, subtler failure of aggregates. Spatial workloads are non-uniform: a dense urban region carries orders of magnitude more features than a sparse rural one. A count-weighted aggregate is dominated by the dense regions, so a rural region can fail completely while contributing almost nothing to the weighted total. Weighted aggregates therefore hide exactly the regions that are hardest to notice by other means.

The counter-instinct — alert per region — has its own failure. One rule per region multiplies the pager by the region count, and a platform-wide fault fires all of them simultaneously. The workable design sits between: aggregate the detection across regions using an extremum, and preserve the region identity only in the notification.

Implementation: extremum detectors with bounded cardinality

The pattern is to compute the per-region health ratio, take the worst value across regions, and attach the offending region’s identity via the label that survives the aggregation.

# Per-region publish health for one layer, then the worst region.
# `min by (layer)` collapses the region dimension for THRESHOLDING but
# `bottomk` re-attaches the identity for the ANNOTATION, so one rule covers
# every region without one alert per region.
(
  min by (layer) (
    sum by (layer, region) (rate(gis_tile_publish_success_total[10m]))
      /
    clamp_min(sum by (layer, region) (rate(gis_tile_publish_attempt_total[10m])), 1)
  )
) < 0.90

# Companion expression used in the annotation to name the worst offenders.
bottomk(3,
  sum by (layer, region) (rate(gis_tile_publish_success_total[10m]))
    /
  clamp_min(sum by (layer, region) (rate(gis_tile_publish_attempt_total[10m])), 1)
)

Two refinements make this production-worthy.

The first is a volume floor. A region that attempted four publishes in ten minutes and failed one reads 75% healthy and fires a spurious alert. Gate the ratio on a minimum attempt count so low-traffic regions cannot trip the detector on tiny samples:

min by (layer) (
  (
    sum by (layer, region) (rate(gis_tile_publish_success_total[10m]))
      /
    clamp_min(sum by (layer, region) (rate(gis_tile_publish_attempt_total[10m])), 1)
  )
  # Only consider regions with a meaningful sample this window.
  and
  (sum by (layer, region) (increase(gis_tile_publish_attempt_total[10m])) >= 50)
) < 0.90

The second is a breadth companion. The worst-region detector says something is broken somewhere; a count of unhealthy regions says how far it has spread, which is what determines severity. Firing both, and routing on the second, gives the responder scope for free.

# How many regions are unhealthy right now — drives severity, not detection.
count by (layer) (
  (
    sum by (layer, region) (rate(gis_tile_publish_success_total[10m]))
      /
    clamp_min(sum by (layer, region) (rate(gis_tile_publish_attempt_total[10m])), 1)
  ) < 0.90
)

One unhealthy region is a regional incident: route it to the owning team with the region named. A third of regions unhealthy is a platform incident: escalate immediately, because the cause is almost certainly shared infrastructure rather than anything regional at all.

Severity determined by how many regions are unhealthy rather than by how badly one is A horizontal ladder shows three breadth bands. One unhealthy region is classed as a regional incident routed to the owning team with the region named in the page. Two to five unhealthy regions are classed as a spreading incident, which escalates and triggers a search for a shared upstream cause. More than five, or more than a quarter of all regions, is classed as a platform incident routed to the incident commander, on the reasoning that a fault affecting many regions is almost never regional. Beneath the ladder, a note explains that depth of failure in one region does not raise severity, only breadth does. Breadth sets severity — depth in one region does not 1 region regional incident page owning team region named in the page check regional worker, source feed, edge collector 2–5 regions spreading incident escalate immediately look for a shared cause: source export, schema change, shared queue saturation > 5 or > 25% platform incident incident commander stop treating it as regional check the collector fleet, database, object store A region at 0% and a region at 85% are the same severity: one region is affected either way. Depth belongs in the annotation so the responder knows what they are walking into, not in the routing decision. Absence detection catching a region that stopped reporting entirely Two regional health series are plotted. The first dips to seventy percent and recovers, which the ratio detector sees. The second simply stops: its series ends and no further samples appear, so the ratio detector sees nothing at all because there is no value to compare. A companion count-of-reporting-regions series drops by one at exactly that moment and is marked as the only signal that catches a total regional outage. A region that stops reporting has no ratio to be below a threshold region A — dips, recovers, detector fires region B — series simply ends reporting regions 20 → 19 — the only signal that fires

Verification: prove the detector sees a single-region fault

Synthesise the failure rather than waiting for it. Pick a low-traffic region, stop its publish worker in a staging environment, and confirm three things in order: the per-region ratio drops, the min by (layer) expression crosses the threshold, and the notification names the correct region. If the third fails while the first two pass, the annotation query is joining on the wrong label — a common and quiet defect.

Then invert the test. Push the region’s attempt count below the volume floor and confirm the detector goes silent rather than firing on a two-sample ratio. A detector that fires on tiny samples will be muted by whoever is on call next week, and then the real regional failure is invisible.

Finally, check the cardinality cost. The region label must be bounded — a fixed set of publish regions or grid cells, never a raw geohash or bounding box, for the reasons set out in bounding spatial metric tag cardinality. Confirm the series count for the detector’s metrics against your budget before the rule ships.

Gotchas

A region that disappears entirely reads as healthy. If a region stops emitting attempts altogether, both numerator and denominator vanish, the series is absent, and min by (layer) simply does not see it. Pair the ratio detector with an absence detector — absent() over the expected region set, or a count of reporting regions against the registry — or a total regional outage is the one case that never fires.

Grid level chosen for cartography rather than for alerting. Regions defined at too fine a grid level produce hundreds of thin series where most are below the volume floor; too coarse and a real single-city failure gets averaged away inside its region. Pick the level at which a region is the smallest unit that can independently fail — usually the publish or worker boundary, not a display grid.

The volume floor masks a slow region. A region whose traffic collapsed to below the floor because of the very fault you are hunting will be excluded by the and clause. The absence detector above covers this; without it, the floor and the fault conspire.

Severity from depth. Escalating because one region is at 0% rather than 85% inverts the priority: a shallow fault across five regions is far more likely to be a platform problem than a deep fault in one. Route on breadth, annotate with depth.

FAQ

Why not just run one alert rule per region?

Rule count grows with regions, every platform-wide fault fires all of them at once, and adding a region silently ships without coverage until someone remembers to add its rule. The extremum pattern gives one rule that automatically covers every region present in the data, including ones added yesterday.

How do I name the failing region in the page when the alert aggregated it away?

Use a companion query in the annotation, as shown above. The alert’s own labels carry layer, and the annotation evaluates bottomk at notification time to list the worst regions. If your alerting stack cannot query inside annotations, emit a second, non-paging alert that keeps the region label and correlate the two by layer — the grouping behaviour described in tuning alert grouping windows for batch GIS jobs will deliver them together.

Does this apply to freshness as well as publish success?

Directly. Replace the success ratio with per-region freshness age and take max by (layer) instead of min — the worst region is the oldest one. The same volume floor becomes a requirement that the region has reported at all in the window, and the same absence problem applies with equal force.

What about regions with legitimately different service levels?

Carry the target as a registry series and compare each region against its own target rather than a global constant, then take the extremum of the ratio to target. This keeps one rule while allowing a rural region a longer freshness allowance than a metropolitan one, in the same way the per-layer thresholds in tracking spatial data freshness SLAs are sourced from a registry rather than hard-coded.

How does this interact with multi-region replication monitoring?

They are complementary: replication lag is measured between regions, while this detector measures health within each. A replication fault typically shows as a spreading incident across several destination regions at once, which is why the breadth companion routes it away from the regional owning team and toward the platform rotation.