Monitoring Cache Invalidation Lag for Map Tiles

A map tile has two truths: the object in the store and the copy at the edge. When a layer is rebuilt, only the first one changes. Until an invalidation propagates, every client is served the old copy, and every dashboard that measures the build pipeline reports success. This is the most common way a tile platform serves stale data while looking entirely healthy — the build lag is zero, the pyramid is complete, and the map is still showing last week’s boundaries. This guide covers how to measure the interval between a store write and the edge actually serving the new content, how to detect partial invalidation across regions, and how to alert on it without generating noise on every routine rebuild. It belongs to raster and tile pipeline observability under geospatial observability architecture fundamentals.

Two copies of a tile diverging between the store write and the edge purge A timeline shows a store write at the start, followed by a purge request, followed by purge completion in three edge regions at different times. Above the timeline, a bar marks the store copy as current from the moment of the write. Below it, three bars mark each edge region as serving stale content until its own purge completes, with the third region completing far later than the others. The interval between store write and the last edge completing is labelled the invalidation lag, and a note identifies the third region as a partial-invalidation failure. The store is current the moment it is written; the edge is not object store — current from t+0 edge eu — stale edge us — stale edge apac — stale far longer store write eu purged us purged apac purged invalidation lag = store write → last edge current Measuring the mean across regions would report this as healthy; the max is the only honest statistic.

Problem framing: what “invalidated” actually has to mean

Three definitions of invalidation circulate, and only the third is measurable in a way that corresponds to what users experience.

The weakest is purge requested: the pipeline called the invalidation API. This tells you nothing — the call is asynchronous and frequently partial. Platforms that alert on purge-request failures and nothing else are effectively unmonitored on this axis.

The middle definition is purge acknowledged: the CDN reported the invalidation complete. Better, but it reports on the control plane rather than the data plane, and in practice acknowledgement precedes actual eviction in some regions by a noticeable margin.

The useful definition is observed current: a request to the edge returns content matching what the store holds. That is measurable by probing, it is exactly what a client experiences, and it is the only definition under which a partial regional failure shows up at all.

Measuring it needs a content identity that survives the round trip. A strong entity tag works where the store and edge preserve it; otherwise hash the bytes. Comparing timestamps is unreliable, because a cache may serve an old body with a fresh header.

Implementation: a probe that compares edge against store

The probe picks a small set of canary tiles per layer — tiles guaranteed to change on every rebuild — fetches each from the store and from every edge region, and reports the age of the difference.

# invalidation_probe.py — measure observed staleness per edge region.
import hashlib, time
import httpx
from opentelemetry import metrics

meter = metrics.get_meter("gis.tile")
stale_seconds = meter.create_observable_gauge(
    "gis.tile.invalidation_lag_seconds",
    callbacks=[lambda opts: _observe(opts)],
    description="Seconds since the store copy changed while an edge still serves the old body",
)

_first_seen_divergent: dict[tuple[str, str], float] = {}

def _digest(body: bytes) -> str:
    return hashlib.blake2b(body, digest_size=16).hexdigest()

def _observe(_options):
    now = time.time()
    for layer, canaries in registry.canary_tiles():
        for url_path in canaries:
            origin = _digest(httpx.get(f"{STORE_BASE}/{url_path}").content)
            for region, base in EDGES.items():
                # Bypass any local cache in the probe itself; we are measuring
                # the EDGE, not our own client.
                edge = _digest(httpx.get(f"{base}/{url_path}",
                                         headers={"Cache-Control": "no-cache"}).content)
                key = (layer, region)
                if edge == origin:
                    _first_seen_divergent.pop(key, None)
                    yield metrics.Observation(0.0, {"layer": layer, "region": region})
                else:
                    # Start the clock the first time we see divergence, so the
                    # gauge reports how LONG the edge has been wrong, not merely
                    # that it is wrong right now.
                    started = _first_seen_divergent.setdefault(key, now)
                    yield metrics.Observation(now - started,
                                              {"layer": layer, "region": region})

Choosing canary tiles well is most of the design. They must change on every rebuild, or the probe reports zero lag while the rest of the pyramid is stale. A tile carrying a build identifier stamped into its metadata, or a low-zoom overview tile that necessarily changes whenever anything beneath it does, both work. Pick two or three per layer at different zoom levels: an overview tile catches whole-layer invalidation failures, a deep tile catches the case where only the top of the pyramid was purged.

The alert takes a maximum over regions, for the same reason every other spatial alert takes an extremum.

groups:
  - name: tile-invalidation
    rules:
      - alert: TileInvalidationStalled
        # Max over regions: one stale edge is the incident, regardless of the rest.
        expr: max by (layer) (gis_tile_invalidation_lag_seconds) > 900
        for: 10m
        labels: { severity: critical, data_domain: spatial }
        annotations:
          summary: >-
            Edge serving stale tiles for {{ $labels.layer }}
            ({{ $value | humanizeDuration }} behind the store)
          worst_regions: '{{ with query "topk(3, gis_tile_invalidation_lag_seconds)" }}{{ range . }}{{ .Labels.region }} {{ end }}{{ end }}'
          runbook_url: "/spatial-incident-response-and-tooling/spatial-pipeline-incident-runbooks/tile-publish-queue-overflow-runbook/"

      # A region that has never converged is different from one that is slow.
      - alert: TileInvalidationRegionStuck
        expr: min_over_time(gis_tile_invalidation_lag_seconds[2h]) > 300
        for: 30m
        labels: { severity: critical, data_domain: spatial }
        annotations:
          summary: "Region {{ $labels.region }} has not converged for {{ $labels.layer }} in 2h"

The second rule matters more than it looks. A region that is slow recovers and its gauge returns to zero; a region that is stuck never does, and a threshold-plus-duration alert on the instantaneous value can flap around the boundary without ever making the stuck state obvious. Taking a minimum over a long window asks “has this region been correct at any point in the last two hours”, and a negative answer is unambiguous.

Three invalidation failure signatures distinguished by the shape of the lag series Three small line charts sit side by side, each showing invalidation lag over time for one failure signature. The first, healthy, shows a sawtooth that rises briefly after each rebuild and returns to zero. The second, slow propagation, shows the same sawtooth with a much taller peak that still returns to zero. The third, stuck region, shows the lag rising and never returning to zero. Beneath each chart, the diagnosis and first action are named: no action, tune the purge concurrency, and force a regional purge then investigate the control plane. The shape of the lag series names the fault healthy returns to zero after each rebuild no action slow propagation tall peaks, still converges raise purge concurrency stuck region never returns to zero force regional purge, check control plane min_over_time separates the third case from the second; an instantaneous threshold cannot. Canary tile choice and what each choice can detect Three canary choices are compared. A tile over a rarely-changing rural area reports zero lag through a total invalidation outage, because its content never changed. A tile stamped with the build identifier changes on every build and detects any invalidation failure. A low-zoom overview tile changes whenever anything beneath it does and additionally detects partial rebuilds. The second and third are marked as usable, the first as a trap. A canary that never changes reports perfect freshness forever rural tile that rarely changes detects nothing — content identical before and after, so the probe always agrees tile stamped with the build identifier changes every build — detects any invalidation failure, in any region low-zoom overview tile changes when anything beneath it does — additionally detects partial rebuilds

Verification: prove the probe detects a stale edge

Write a changed canary tile to the store without issuing a purge, and confirm the lag gauge starts climbing for every region within one probe interval. This is the single most important test, because it exercises the exact condition the probe exists to catch and it is trivial to arrange.

Then issue the purge and confirm the gauge returns to zero in every region. A region whose gauge stays elevated after a successful purge is either serving from a layer of cache the probe is not reaching or has a stale intermediate proxy, both of which are findings worth having.

Finally, confirm the probe is not itself cached. Run it twice in quick succession against an unchanged tile and check that both requests reached the edge — a probe served from a local HTTP cache reports perfect freshness forever. Setting an explicit no-cache header, as the implementation does, and asserting on the response’s cache-status header where available, covers this.

Gotchas

Canary tiles that do not change. A canary over an area that rarely updates reports zero lag through a total invalidation outage. Prefer tiles whose content is guaranteed to change per build — a low-zoom overview, or a tile stamped with the build identifier.

Comparing headers instead of bodies. Last-modified and cache-age headers are rewritten by intermediaries and can be fresh while the body is old. Hash the body.

One probe location. Probing from a single network location exercises a single edge, and a partial regional failure is invisible from anywhere except the failing region. Probe from each region you serve, or from a synthetic client that can target regional endpoints directly.

Alerting on the mean lag. Averaging across regions is how a completely stuck region gets reported as a small increase. Use max, and use the region label in the annotation so the responder knows where to look — the same reasoning set out in alerting on partial-region failures.

FAQ

How often should the probe run?

Frequently enough that the reported lag is precise relative to your alert threshold. Probing every sixty seconds against a fifteen-minute threshold is ample. Probing more often mostly buys precision you will not act on, and each probe is a small number of requests, so cost is rarely the binding constraint.

Should I probe every layer?

Probe every layer whose staleness would matter to a consumer, which is usually all served layers but not internal intermediates. Two or three canaries per layer keeps the total request volume trivial even across dozens of layers.

What if my CDN does not support targeted purges?

Then invalidation is a time-to-live problem rather than an event, and the metric to watch is whether observed staleness stays below the configured lifetime. The probe is unchanged; only the expected shape of the series differs, converging on a sawtooth bounded by the lifetime rather than by purge latency.

How does this interact with pyramid completeness?

They are independent and both necessary. Completeness answers “was it built”; invalidation lag answers “is what clients receive the built thing”. A layer can be perfectly complete in the store and entirely stale at the edge, which is why the sampled probe in detecting tile pyramid gaps at high zoom is recommended against both the store and the serving endpoint.

Does a planned rebuild need to suppress this alert?

No, provided the threshold exceeds normal purge propagation. If routine rebuilds trip it, the threshold is too tight rather than the alert being wrong — and if propagation genuinely takes longer than users can tolerate, that is a real finding rather than something to silence with a maintenance window.