Raster and Tile Pipeline Observability

Vector observability asks whether a feature is valid, current and correctly projected. Raster and tile observability asks a different set of questions, because the unit of work is not a feature but a pyramid: a set of derived images or coverages at multiple resolutions, each level built from the one below, each level cached independently, and each level capable of being stale, missing or wrong while its neighbours look perfect. A tile pipeline can be entirely healthy at zoom 6 and entirely broken at zoom 14, in one metropolitan area, for one style, and every aggregate metric you own will look fine.

This topic covers the signals that make raster and tile pipelines observable: pyramid completeness, per-level publish lag, nodata and band integrity, cache invalidation correctness, and the render-time metrics that tell you whether a tile is merely present or actually right. It belongs to geospatial observability architecture fundamentals and shares its instrumentation conventions with the geospatial metric taxonomy for ETL.

Core concepts: the pyramid is the unit of correctness

Three properties distinguish raster and tile observability from its vector equivalent.

Correctness is per level, not per dataset. A pyramid is a set of derived products. Rebuilding zoom 12 without rebuilding zoom 6 leaves the overview showing yesterday’s world at low zoom and today’s at high zoom — a state that no dataset-level freshness metric detects, because something was published recently. Every completeness and freshness signal therefore needs a zoom or level dimension, and the alert has to consider the worst level rather than the average.

Absence is the dominant failure. In vector pipelines the common defect is a wrong value; in tile pipelines it is a missing tile. A gap in the pyramid renders as blank map area, which users notice immediately and monitoring frequently does not, because a tile that was never written produces no error, no log line and no metric — only the absence of one. Completeness must therefore be measured against an expected set derived from the layer’s extent, not counted from what happens to exist.

Staleness survives repair. Once a wrong tile is in an edge cache, fixing the source and rebuilding the pyramid changes nothing for clients until the cache is invalidated. The cache is a second, independent copy of the truth with its own age, and the interval between object-store write and cache refresh is where most user-visible tile incidents actually live.

A fourth property is worth naming because it distorts every cost metric: tile counts grow geometrically with zoom. A single zoom level increment quadruples the tile count, so a pyramid to zoom 16 contains overwhelmingly more tiles at its deepest level than everywhere else combined. Any metric that counts tiles without a level dimension is, in practice, a metric about the deepest level only.

Metric taxonomy for raster and tile pipelines

Keep the namespace consistent with the rest of the platform: gis.raster.* for coverage-level signals and gis.tile.* for pyramid and serving signals. Every series carries layer and, where meaningful, zoom and region.

The completeness ratio deserves comment because it is the one metric that requires computing an expectation. For a layer whose extent is known, the expected tile count at zoom $z$ over a bounding box is derived from the tile indices covering that box:

$$N_z = (x^{max}_z - x^{min}_z + 1)\,(y^{max}_z - y^{min}_z + 1)$$

with $x_z = \lfloor 2^{z}\,(\lambda + 180)/360 \rfloor$ for longitude $\lambda$ and the corresponding Mercator expression for latitude. Computing $N_z$ from the layer’s registered extent — rather than from a previous run’s tile count — is what makes the ratio detect a shrinking pyramid rather than merely tracking it.

The tile size distribution is an unusually good correctness proxy and worth instrumenting even though it looks like a cost metric. A style regression that renders an empty basemap produces tiles of nearly constant, very small size; a labelling fault that repeats a label across the pyramid inflates the p99. Watching the size histogram catches a class of “the tiles are all there and all wrong” failures that completeness cannot see.

Instrumentation: computing completeness against an expectation

The build worker knows the layer extent and the levels it was asked to produce, so it can compute the expected tile set directly and compare it to what it wrote.

# tile_completeness.py — expected-versus-present completeness per pyramid level.
import math
from opentelemetry import metrics

meter = metrics.get_meter("gis.tile")
complete = meter.create_observable_gauge(
    "gis.tile.level_complete_ratio",
    callbacks=[lambda opts: _observe(opts)],
    description="Tiles present divided by tiles expected for the layer extent",
)

def tile_range(west, south, east, north, zoom):
    """Web Mercator tile index range covering a WGS84 bounding box."""
    def xy(lon, lat):
        n = 2 ** zoom
        x = int((lon + 180.0) / 360.0 * n)
        lat_r = math.radians(max(min(lat, 85.05112878), -85.05112878))
        y = int((1.0 - math.asinh(math.tan(lat_r)) / math.pi) / 2.0 * n)
        return x, y
    x0, y1 = xy(west, south)
    x1, y0 = xy(east, north)
    return x0, x1, y0, y1

def expected_count(extent, zoom):
    x0, x1, y0, y1 = tile_range(*extent, zoom)
    # Expectation comes from the REGISTERED extent, never from a previous run —
    # otherwise a pyramid that shrank yesterday looks complete today.
    return (x1 - x0 + 1) * (y1 - y0 + 1)

def _observe(_options):
    for layer, extent, levels in registry.layers():
        for zoom in levels:
            present = store.count_tiles(layer, zoom)
            yield metrics.Observation(
                present / max(expected_count(extent, zoom), 1),
                {"layer": layer, "zoom": str(zoom)},
            )

The zoom label is stringified deliberately: it is a small bounded set, and keeping it as a label rather than folding it into the metric name lets one alert rule cover every level. Deep pyramids should still cap the labelled range — instrumenting every level to zoom 20 individually multiplies series count without adding much signal, and grouping levels above a threshold into a single zoom="15+" bucket keeps cardinality within the budget discussed in bounding spatial metric tag cardinality.

Alerting: worst level, not average level

Every pyramid alert takes an extremum over the level dimension for the same reason regional alerts take an extremum over regions: the average hides the failure.

groups:
  - name: tile-pyramid
    rules:
      # Completeness: the WORST level decides, and the annotation names it.
      - alert: TilePyramidIncomplete
        expr: min by (layer) (gis_tile_level_complete_ratio) < 0.98
        for: 15m
        labels: { severity: critical, data_domain: spatial }
        annotations:
          summary: "Pyramid gap on {{ $labels.layer }} — worst level {{ $value | humanizePercentage }} complete"
          runbook_url: "/spatial-incident-response-and-tooling/spatial-pipeline-incident-runbooks/tile-publish-queue-overflow-runbook/"

      # Publish lag: the OLDEST level decides.
      - alert: TilePublishLagHigh
        expr: max by (layer) (gis_tile_publish_lag_seconds) > 3600
        for: 20m
        labels: { severity: warning, data_domain: spatial }

      # Invalidation: the edge is serving something the store no longer holds.
      - alert: TileCacheStale
        expr: |
          max by (layer, region) (gis_tile_cache_age_seconds)
            - on (layer) group_left() max by (layer) (gis_tile_publish_lag_seconds)
          > 1800
        for: 10m
        labels: { severity: critical, data_domain: spatial }

      # Correctness proxy: a style regression collapses the size distribution.
      - alert: TileSizeDistributionCollapse
        expr: |
          histogram_quantile(0.5, sum by (le, layer, zoom) (rate(gis_tile_bytes_bucket[30m])))
          < 0.4 *
          histogram_quantile(0.5, sum by (le, layer, zoom) (rate(gis_tile_bytes_bucket[30m] offset 1d)))
        for: 30m
        labels: { severity: warning, data_domain: spatial }

The completeness threshold sits at 0.98 rather than 1.0 on purpose. Real pyramids carry a small number of legitimately absent tiles — ocean-only cells in a land-cover layer, tiles outside a clipped administrative boundary — and demanding perfection produces a permanently firing alert. Where the layer genuinely should be complete, tighten to 1.0 and enjoy the stronger signal.

The tile size comparison against a day-ago baseline is the cheapest correctness check available for a styled raster or vector-tile layer, and it catches the failure mode where every tile renders successfully and every tile is blank.

Scale considerations: instrumenting a pyramid without paying for it

The geometric growth of tile counts makes naive instrumentation expensive in a way that vector pipelines rarely experience. Three disciplines keep the cost bounded while preserving the signal.

Never label by tile index. A metric carrying x and y produces one series per tile, which at zoom 14 over a national extent is tens of millions of series. The correct dimensions are layer, zoom and a coarse region — the last drawn from a small fixed set of publish regions or a low-zoom grid cell, never from the tile’s own coordinates. This is the same discipline that governs vector attributes, and the reasoning is set out fully in bounding spatial metric tag cardinality.

Aggregate deep levels. Levels beyond roughly zoom 15 rarely repay individual instrumentation: they behave alike, they fail together, and they dominate every unlabelled count. Collapsing them into a single zoom="15+" bucket keeps the interesting levels distinguishable while capping the label range at a dozen values.

Sample the expensive checks, count the cheap ones. Manifest counters emitted by the build worker cost nothing because the worker already knows the numbers. Probing the object store or the edge costs a request per probe, so it belongs on a schedule with a bounded sample rather than on every scrape. Mixing the two — exact counting where it is free, sampling where it is not — gives most of the coverage of a full audit at a small fraction of the cost.

A fourth consideration is retention rather than cardinality. Correctness signals such as per-level completeness and warp rejection reasons are the series a post-incident review will need to query weeks later, and they are small. Retaining them longer than the high-volume serving metrics costs little and repeatedly turns out to be the difference between a reconstructable incident and a guess, as the post-incident review practice depends on.

Multi-region behaviour and the degraded-serve decision

Tile platforms almost always serve from more than one location, and the pyramid at each location is an independent copy with its own build state, cache state and failure modes. That has a practical consequence: the question “is this layer healthy” has no single answer, and the platform needs an explicit policy for what to do when the answer differs by region.

The workable policy is a small ladder. When a region’s pyramid is complete and its cache current, serve normally. When a region’s pyramid is complete but its cache lags beyond the tolerated staleness, serve anyway and mark the layer degraded in the response headers, because stale tiles are usually more useful than no tiles. When a region’s pyramid has a gap, prefer serving the corresponding tile from the nearest region that has it — cross-region fill costs latency but hides the gap entirely from the client. Only when no region holds the tile should the request fall through to a placeholder, and that event deserves its own counter, because a placeholder served is a user-visible failure regardless of what every other metric says.

Implementing the ladder needs one signal the build pipeline does not naturally produce: a per-region view of completeness and cache age, rather than a global one. That is why both gis.tile.level_complete_ratio and gis.tile.cache_age_seconds carry a region dimension in the taxonomy above, and why every alert over them takes an extremum rather than an average. It also connects the tile stack to the general degradation design used elsewhere on the platform, described in fallback chains for spatial API failures, which treats the choice between a degraded answer and no answer as a design decision rather than an accident.

The one behaviour to avoid is silent cross-region fill without instrumentation. A platform that quietly serves European tiles to Asian clients when the Asian pyramid has a gap will look perfectly healthy on every completeness dashboard while its latency percentiles drift and nobody knows why. Count the fills, label them by source and destination region, and alert when the rate rises — the fill is a legitimate mitigation and a poor steady state.

Failure modes and fallback behaviour

A level rebuilt without its overviews. High zoom shows new data, low zoom shows old. The tell is per-level publish lag diverging across the pyramid while completeness stays at one. Fallback: serve from the level whose lag is lowest and mark the layer degraded rather than mixing epochs.

Cache invalidation partially completed. The purge succeeded in most edge locations and failed in one. This surfaces as gis.tile.cache_age_seconds diverging by region — a partial-region failure in the sense covered by alerting on partial-region failures, and detectable only if the region label survives.

Nodata expansion after a projection change. A warp into a different projection can push valid pixels outside the target grid, silently increasing the nodata ratio while every tile still renders. Watch gis.raster.nodata_ratio against the source’s own ratio, not against zero.

Band reordering. A source that swaps band order produces tiles that render in wrong colours with no error at any stage. Assert band count and band metadata against a contract at warp time; this is the raster equivalent of a schema fingerprint.

Queue overflow at deep levels. Because tile counts quadruple per level, a backlog that is trivial at zoom 10 is catastrophic at zoom 14. Queue depth alerts must be scaled per level, and the recovery procedure is the subject of the tile publish queue overflow runbook.

A style change that renders successfully and blankly. Every tile is written, the pyramid is complete, publish lag is zero, and the map is empty because a layer reference in the style no longer resolves. Only the size distribution notices, which is the argument for treating gis.tile.bytes as a correctness signal rather than a cost one.

Operational debugging workflow

1. Establish which levels are affected: query gis_tile_level_complete_ratio by zoom and note whether the gap is one level or a contiguous range.
2. Establish which regions are affected: query completeness by region at the worst level; a single region points at a worker, a global gap points at the builder.
3. Compare object-store contents against the expectation directly for a sample of missing tile indices — a tile absent from the store is a build failure, a tile present in the store but missing at the edge is an invalidation failure.
4. Check gis.tile.render_error_total by reason over the build window; a dominant reason usually names the fault outright.
5. Check the source coverage: band count, nodata ratio and projection against contract, in that order.
6. Inspect the size histogram for the affected level against the previous day to distinguish “missing” from “present but empty”.
7. Confirm the invalidation completed by requesting a known-changed tile from each edge region and comparing its content hash against the object store.

Related

• Geospatial observability architecture fundamentals — the parent section defining the metric conventions used here.
• Tile publish queue overflow runbook — the incident procedure for a saturated publish queue.
• Spatial coverage and extent monitoring — the vector-side analogue of pyramid completeness.
• Observability scoping rules for vector data — the sampling discipline that keeps this instrumentation affordable.