Tile Publish Queue Overflow Runbook

When gis.spatial.tile_queue_depth climbs faster than render workers can drain it, the underlying data is fresh but the map is not — consumers keep seeing tiles built from an hours-old snapshot while the publish backlog grows behind the cache. This runbook is the load-shedding and drain procedure for that incident: confirming the queue is genuinely backing up rather than briefly spiking, shedding low-value render work, adding drain capacity, and invalidating the stale tiles once the queue recovers. It is one of the spatial pipeline incident runbooks inside the spatial incident response and tooling program.

Tile publish queue backlog and drain On the left, producers push render jobs into a queue drawn as a stack of cells with a rising fill level past a dashed high-water mark. Render workers pull from the queue at a drain rate lower than the enqueue rate. Completed tiles flow to a tile cache that currently serves stale tiles to consumers. Below, a load-shed valve diverts low-zoom and low-traffic jobs, and an added-worker box increases drain capacity, together restoring the balance. Producers enqueue rate queue depth high-water backlog growing Workers drain rate Tile cache serving stale tiles Load shed low-zoom / low-hit

Problem Framing

A tile publish queue overflow is a throughput incident wearing a freshness mask. Nothing is corrupt — every geometry is valid, every SRID is correct — but the render workers that turn updated features into map tiles cannot keep up with the rate at which invalidations are enqueued. The queue is a buffer, and while it drains slower than it fills, its depth grows without bound. The user-visible symptom is stale tiles: a consumer requests a tile, the cache returns the last-rendered version, and because the fresh render is still sitting in the backlog, the map shows data that is minutes or hours behind the database.

The reason this deserves its own runbook rather than a generic queue-length alert is the coupling to the tile cache. In most work queues, a backlog only delays completion. Here the backlog actively serves wrong answers, because a cache miss falls back to a stale on-disk tile rather than blocking. So the incident has two fronts: the queue depth that you must drain, and the stale cache entries that you must invalidate once the fresh renders finally land. Fixing the drain without invalidating the cache leaves consumers on stale tiles even after the queue recovers.

Overflow has a small number of root causes worth distinguishing during triage: a burst of invalidations from a large bulk update, a drop in worker throughput (a slow render dependency, a node lost from the pool), or a feedback loop where a mis-scoped invalidation re-enqueues far more tiles than actually changed. The distributed version of this problem — a backlog isolated to one region’s render pool — connects to monitoring topology for multi-region GIS, because a healthy global publish rate can hide one region drowning.

Detect

The naive detector — depth over a fixed threshold — pages on every harmless burst and stays silent when a slow leak builds under the threshold. The signal that matters is depth relative to drain rate: how long the current backlog would take to clear at the current worker throughput. That ratio is the estimated time-to-drain, and it is what the tiered rule below alerts on.

groups:
  - name: tile-queue-overflow
    rules:
      # Warning: backlog would take > 10 min to clear at current drain rate.
      - alert: TileQueueDrainSlow
        expr: |
          gis_spatial_tile_queue_depth
            / clamp_min(rate(gis_spatial_tiles_rendered_total[5m]) * 60, 1) > 10
        for: 10m
        labels: { severity: warning }
        annotations:
          runbook_url: "/spatial-incident-response-and-tooling/spatial-pipeline-incident-runbooks/tile-publish-queue-overflow-runbook/"
      # Critical: depth still climbing AND drain time > 30 min.
      - alert: TileQueueOverflowCritical
        expr: |
          (gis_spatial_tile_queue_depth
            / clamp_min(rate(gis_spatial_tiles_rendered_total[5m]) * 60, 1) > 30)
          and (deriv(gis_spatial_tile_queue_depth[10m]) > 0)
        for: 5m
        labels: { severity: critical }

The deriv clause on the critical rule is what separates a draining spike from a genuine overflow: a queue that is deep but shrinking will clear on its own, while a queue that is deep and still growing needs intervention now. Confirm the drain rate is real — a worker pool that has stalled entirely reports a zero render rate, which the clamp_min guards against but which you should notice as its own failure.

Triage

Triage decides which of the three root causes you are facing, because the remediation differs for each. Read the enqueue rate, the drain rate, and the invalidation source in one pass.

# Enqueue vs drain — is this a producer surge or a worker shortfall?
sum(rate(gis_spatial_tile_invalidations_total[5m])) by (region)   # in
sum(rate(gis_spatial_tiles_rendered_total[5m]))     by (region)   # out

# Which layer/zoom is flooding the queue?
topk(5, sum(rate(gis_spatial_tile_invalidations_total[5m])) by (layer, zoom))

If enqueue spiked while drain held steady, a bulk update or a mis-scoped invalidation is the cause — look at the topk to see whether one layer or a low zoom level dominates. If drain dropped while enqueue held steady, a worker-side regression is the cause — check for a lost node or a slow render dependency. If both moved, you likely have a feedback loop where invalidations trigger renders that trigger more invalidations. Scope the blast radius by region: a backlog on one region’s pool is contained and reroutable, while a global backlog needs capacity. Assign severity from how far behind the freshness budget the stale tiles have fallen, not from the raw depth.

Remediate

Remediation is shed, then drain, then invalidate — in that order. Shedding low-value work buys the drain headroom; adding workers accelerates it; invalidating the cache is the last step, done only once the fresh tiles have actually rendered.

# 1. SHED: temporarily drop low-value render jobs so high-value tiles drain.
#    Prioritise by zoom and recent hit-rate; low-zoom overviews and cold
#    tiles can wait, hot mid-zoom tiles cannot.
shed_policy:
  drop_if:
    - zoom <= 6                       # continental overviews, rarely refreshed
    - tile_hit_rate_24h < 0.01        # cold tiles nobody requests
  keep_if:
    - layer in [live_incidents, vessel_positions]   # freshness-critical

# 2. DRAIN: add render workers to the affected region's pool.
#    Scale the pool until drain rate exceeds enqueue rate.

# 3. INVALIDATE: once the backlog clears, purge stale cache entries so
#    consumers stop being served the old renders.
# Targeted cache invalidation for the layer that was behind — do NOT purge
# the whole cache, which would re-enqueue every tile and re-trigger overflow.
curl -X POST "$TILE_CACHE/purge" \
  --data '{"layer":"parcels","zoom_min":7,"zoom_max":16,"bbox":[-122.6,37.2,-121.7,38.0]}'

The single most important remediation rule is scoping the invalidation. A panicked full-cache purge re-enqueues every tile at every zoom for every layer, which floods the very queue you just drained and turns a recovering incident into a worse one. Invalidate only the layer and zoom range and bounding box that were actually behind. Shed work by zoom and hit-rate because low-zoom overview tiles and cold tiles that nobody requests cost render capacity for no user benefit, while hot mid-zoom tiles on freshness-critical layers must never be shed.

Post-mortem and Verification

Verify recovery on three axes: the queue depth returns to its normal baseline and the drain-time ratio drops below the warning threshold; the stale-tile age (the gap between a tile’s render time and the underlying data’s update time) falls back inside the freshness budget; and a spot check of previously-stale tiles confirms they now reflect current data. A queue that is empty but a cache still serving stale tiles means the invalidation step was skipped.

# Recovery check: estimated drain time back under the 10-minute warning line
gis_spatial_tile_queue_depth
  / clamp_min(rate(gis_spatial_tiles_rendered_total[5m]) * 60, 1) < 10

The post-mortem should identify why the queue was allowed to grow unbounded before anyone was paged — usually the detector was on raw depth rather than drain time, so it either cried wolf or stayed silent. Harden by adopting the drain-time ratio as the primary detector, and by adding a guard rail on invalidation scope so a bulk update cannot enqueue more tiles than the pool can drain within the freshness budget. If the incident was worker-side, the durable fix is autoscaling the render pool on the same drain-time signal.

Gotchas

A full-cache purge is the classic self-inflicted outage. Purging everything re-enqueues every tile and re-triggers the overflow, often worse than the original. Always scope invalidation to the affected layer, zoom range, and bounding box.

Zero drain rate is a stalled pool, not a slow one. If gis_spatial_tiles_rendered_total stops incrementing entirely, no amount of shedding helps — a node or dependency is down. The clamp_min in the detector hides this in the ratio, so check the raw render counter during triage.

A regional backlog hides in a healthy global rate. Aggregate publish throughput can look fine while one region drowns. Keep the region label on both the depth and render metrics and alert per region, consistent with the multi-region monitoring design, so one pool’s overflow surfaces on its own.

FAQ

Why alert on drain time instead of queue depth?

Depth alone has no meaning without throughput — a depth of 10,000 is fine if workers clear 5,000 per minute and catastrophic if they clear 50. The drain-time ratio (depth divided by render rate) expresses how long consumers will keep seeing stale tiles, which is the outcome that actually matters.

Should I add workers or shed load first?

Shed first, because it is instant and reversible, then add workers, which takes time to provision. Shedding low-zoom and cold tiles immediately frees capacity for the hot tiles that users are actually requesting, buying headroom while the pool scales up.

How do I decide which tiles to shed?

By zoom level and recent hit-rate. Low-zoom overview tiles refresh rarely and tolerate staleness; tiles with near-zero requests over the last day cost render capacity for no benefit. Never shed hot mid-zoom tiles on freshness-critical layers like live incident or position feeds.

Why not just invalidate the whole cache to be safe?

Because a global purge forces every tile to re-render at once, which re-floods the queue and re-triggers the overflow you are recovering from. Scoped invalidation of only the layer and area that were behind gives correct results without the self-inflicted second incident.

Is this incident ever a data-quality problem in disguise?

Occasionally — a mis-scoped invalidation triggered by a topology or CRS incident can flood the queue as a secondary effect. If the invalidation source is a repair job, check whether an upstream data-corruption runbook is the real root cause before treating this purely as a capacity issue.