Tuning Alert Grouping Windows for Batch GIS Jobs

A batch geospatial job does not fail the way a request-serving system fails. Its detectors are spread across minutes — a projection check at ingestion, a validity sweep after the load, a coverage comparison once the transaction commits, a tile-invalidation lag that only becomes measurable when the renderer picks up the change. Group those with the thirty-second window that works well for a web service and the notification arrives carrying one detector, followed by three stragglers that read as new incidents. This guide covers how to size group_wait, group_interval and repeat_interval to the actual cadence of a batch spatial pipeline, and it belongs to alert routing and on-call design for spatial pipelines within the wider spatial incident response and tooling program.

Detector arrival spread across a batch run compared against two grouping windows A timeline covers four minutes of a batch load. Four detectors fire at staggered times: the projection check at fifteen seconds, the validity sweep at seventy seconds, the coverage comparison at one hundred and forty seconds, and the tile invalidation lag at two hundred and ten seconds. A short thirty-second grouping window closes before the second detector arrives and produces four separate notifications. A ninety-second window that restarts on each arrival collects all four into a single notification. Detectors arrive staggered — the window has to outlive the gaps t+0s t+60s t+120s t+180s t+240s srid validity coverage tile lag 30 s window too short 4 notifications 90 s window, restarted by each arrival — closes 90 s after the last detector sized 1 notification Window length is a property of the pipeline's stage spacing, not of how urgent the failure is.

Problem framing: the gaps are the input

The parameter you are tuning is not “how long am I willing to wait” — it is “how far apart do correlated detectors arrive”. Those are different questions, and only the second one has a measurable answer.

For any batch layer, the correlated detectors fire at points determined by the pipeline’s own structure. The projection and schema gates run inline during ingestion and fire almost immediately. Geometry validity typically runs as a post-load sweep, so it lags by however long the load takes. Coverage and row-count reconciliation compare a committed state against a baseline, so they cannot fire until the transaction closes. Tile or cache invalidation lag needs at least one scrape interval after the renderer notices the change. On a layer that loads in ninety seconds and scrapes every thirty, that spread is comfortably three to four minutes end to end.

Set group_wait shorter than the largest gap between consecutive detectors and the group closes mid-incident. Set it longer than the acceptable time-to-notify and you have traded noise for delay. The useful target is the largest consecutive gap plus a small margin — not the total span, because the window restarts each time a new alert joins the group.

That distinction matters and is frequently missed. With a ninety-second group_wait, a batch whose detectors arrive at 15 s, 70 s, 140 s and 210 s still produces one notification: each arrival is within ninety seconds of the previous one, so the timer keeps resetting. The total span is 195 seconds, but no single gap exceeds seventy seconds. Measuring the gaps, not the span, is what lets you keep the window short enough to stay responsive.

Measuring the real gaps before choosing a number

Pull the actual firing times from your alert history rather than reasoning from the schedule. The alert state series records every transition, and a short range query over past incidents on one layer gives the distribution you need.

# Firing timestamps for every spatial detector on one layer over the last 14 days.
# Run over a lookback that covers several real incidents, then read the gaps
# between consecutive alertname transitions inside each incident.
max by (alertname, layer) (
  ALERTS{alertstate="firing", layer="parcels_authoritative", data_domain="spatial"}
)

# Companion query: how long the layer's load transaction actually takes, which
# bounds when the post-load sweeps can possibly fire.
histogram_quantile(0.95,
  sum by (le, layer) (rate(gis_etl_load_duration_seconds_bucket{layer="parcels_authoritative"}[14d]))
)

Read the p95 load duration first — post-load detectors cannot fire before it. Then take the observed inter-arrival gaps from the alert transitions and use the p90 of those gaps as the basis for group_wait. Rounding up to the next convenient value gives a window that collects the family on almost every incident without waiting on the rare straggler.

For a typical nightly cadastral or boundary refresh this lands between sixty and one hundred and twenty seconds. For a streaming feed with inline-only detectors it lands under thirty. For a heavy raster or tile rebuild where the invalidation sweep runs on a five-minute cycle, it can legitimately reach three hundred — and at that point the right move is usually to split the routing rather than stretch one window, because a five-minute notification delay is unacceptable for the correctness detectors that fire early.

Implementation: window sizes that follow the cadence

The cleanest configuration expresses the cadence explicitly, with a separate route per pipeline class rather than one global compromise.

route:
  receiver: spatial-daily-queue
  group_by: ['layer', 'source']
  group_wait: 60s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    # Streaming feeds: detectors are inline, so gaps are tiny and delay is costly.
    - matchers: [pipeline_class="stream", data_domain="spatial"]
      receiver: spatial-pager
      group_wait: 20s
      group_interval: 2m
      repeat_interval: 1h

    # Nightly batch layers: post-load sweeps lag the ingest gates by ~1 minute.
    - matchers: [pipeline_class="batch", data_domain="spatial"]
      receiver: spatial-pager
      group_wait: 90s        # p90 inter-detector gap (70 s) plus margin
      group_interval: 10m    # a batch incident does not change every 5 minutes
      repeat_interval: 6h    # nobody can re-run the load before the next export

    # Tile and raster rebuilds: invalidation lag is measured in scrape cycles.
    - matchers: [pipeline_class="tile", data_domain="spatial"]
      receiver: spatial-pager
      group_wait: 180s
      group_interval: 15m
      repeat_interval: 6h

The three parameters do genuinely different jobs and are worth separating in your head. group_wait decides how long to hold the first notification for a new group so late detectors can join it. group_interval decides how long to wait before sending an update when new alerts join an already-notified group — set it too low on a batch job and every straggler re-notifies, undoing the grouping. repeat_interval decides how often an unchanged, still-firing group re-notifies, and on batch pipelines it should reflect how often a human could plausibly do something: if the upstream export only lands at 06:00, re-paging at 03:00, 04:00 and 05:00 accomplishes nothing.

The pipeline_class label is the load-bearing piece. Stamp it at the rule level from the layer’s registry entry so the routing tree never has to enumerate layer names — the same registry that supplies the per-layer thresholds described in tracking spatial data freshness SLAs.

The three grouping parameters and which part of an incident each one controls A single incident timeline is annotated in three bands. The first band, group wait, spans from the first alert to the first notification and holds the notification open so late detectors can join. The second band, group interval, spans from the first notification to an update notification and controls how quickly newly joined alerts trigger a follow-up. The third band, repeat interval, spans from the update to a re-notification of the unchanged group and controls how often a still-firing incident nags. Each band names a typical batch value. Three parameters, three different jobs first alert notify late detector joins update re-notify group_wait 90 s group_interval 10 m repeat_interval 6 h hold for stragglers throttle updates nag rate while unresolved On a batch layer, repeat_interval should not be shorter than the interval at which a human could act. If the upstream export lands at 06:00, re-paging hourly from 03:00 adds noise and no information. Notifications per incident as a standing quality metric A ratio of notifications sent to distinct incidents is plotted over twelve weeks. It sits close to one for the first eight weeks, then rises steadily toward two. An annotation marks the point where a pipeline stage got slower and pushed one detector outside the grouping window. A threshold line at one point two marks where the alert fires, well before anyone would have complained about noise. Notifications per incident: an early warning that a stage got slower 1.0 1.5 2.0 1.2 — alert threshold load duration grew; a detector left the window wk 0wk 6wk 12

Verification: prove the grouping before the next incident

Two checks confirm the window is doing what you intended.

The first is a replay. Pick a past incident on the layer, note the four or five detector firing times, and inject synthetic alerts with the same label sets at the same relative offsets against a staging alert manager. Count the notifications that reach the receiver. One is correct; more than one means the window is short somewhere in the sequence, and the offsets tell you exactly where.

# Inject a correlated family with the real spacing, then count notifications.
for offset in 0 55 125 195; do
  sleep_until=$offset
  ( sleep "$sleep_until"; curl -sS -XPOST http://alertmanager:9093/api/v2/alerts \
      -H 'Content-Type: application/json' \
      -d "[{\"labels\":{\"alertname\":\"Synthetic$offset\",\"layer\":\"replay_layer\",
            \"source\":\"replay\",\"severity\":\"warning\",\"data_domain\":\"spatial\",
            \"pipeline_class\":\"batch\"}}]" ) &
done
wait
# Expect exactly one notification for group {layer=replay_layer, source=replay}.

The second is a standing metric. Track notifications sent per distinct incident and alert when the ratio drifts above roughly 1.2. A ratio creeping toward two means a detector has moved — usually because a pipeline stage got slower and now falls outside the window — and it is a much earlier signal than an engineer complaining about noise.

Gotchas

Stretching group_wait to fix a fan-out problem. If one detector fires per region and you receive twenty notifications, the problem is the grouping key, not the window. Add the region to the group key or aggregate the detector; lengthening the wait only delays the same twenty.

Forgetting that the window delays the first page too. A ninety-second group_wait on a critical correctness alert means ninety seconds of silence after a projection break. That is usually acceptable on batch layers where nothing is served until the load completes, but it is not acceptable on a live feed. Split correctness-critical detectors onto a shorter route rather than accepting a global compromise — the same separation the severity model in the parent guide draws.

A group_interval shorter than the batch’s own retry loop. Batch jobs frequently retry a failed stage two or three times. If group_interval is shorter than the retry spacing, each retry’s failure re-notifies. Set it above the retry backoff so one job run produces one conversation.

Silent regrouping after a label change. Adding or renaming a label that appears in group_by splits an existing group in two, and mid-incident that looks like a new incident. Roll label changes with the routing change, not separately.

FAQ

How do I choose group_wait for a layer that has never had an incident?

Derive it from the pipeline structure instead of history: take the p95 load duration, add one scrape interval, and use that as the largest expected gap. Review it after the first real incident with measured arrival times and adjust. Starting conservative — say ninety seconds for any batch layer — is safer than starting short, because a slightly late page is recoverable and a fragmented page trains people to ignore the pager.

Should critical and warning alerts share a grouping window?

They can share a group key but should not share a route with the same timings. Correctness-critical detectors deserve a shorter group_wait so the page goes out fast; degradation warnings can wait for a fuller picture. Keeping them on separate routes with the same group_by gives you fast paging plus coherent batching in the daily queue.

Does a longer window risk merging two genuinely separate incidents?

Only if they share the whole group key within the window. Because the key includes layer and source, two distinct incidents merge only when the same layer from the same source breaks twice within roughly a minute — in which case they almost certainly do share a cause. The real merging risk comes from grouping on source alone.

What about alerts that resolve inside the window?

An alert that fires and resolves within group_wait never notifies at all, which is usually the desired outcome for transient blips. If a class of alert routinely does this and you still want a record, route it to the review channel with a short window rather than shortening the pager’s.

How does this interact with inhibition rules?

Inhibition is evaluated before notification, so a symptom alert that arrives inside the window and is inhibited simply never joins the group. That is the intended behaviour: grouping collects what is left after suppression, which is why correlation and suppression are described as consecutive stages in the parent guide rather than as one step.