Detecting Tile Pyramid Gaps at High Zoom

A missing tile produces no error. Nothing logs, nothing counts, nothing fails — the build worker simply never wrote the object, and the map shows a blank square to whoever pans over it. That silence is why pyramid gaps are usually reported by users rather than by monitoring, and why they cluster at high zoom, where the tile count is largest and the build queue is most likely to have shed work. This guide covers how to detect gaps by comparing against a computed expectation rather than counting what exists, how to make that check affordable at zoom levels holding millions of tiles, and how to distinguish a genuine gap from a legitimately empty cell. It belongs to raster and tile pipeline observability under geospatial observability architecture fundamentals.

Expected tile grid compared against written tiles, revealing a contiguous gap A grid of tile cells covering a layer extent at high zoom is drawn twice. On the left, the expected set derived from the registered extent shows every cell filled. On the right, the written set shows the same grid with a contiguous block of six cells missing in the lower right and two isolated cells missing elsewhere. Beneath the grids, a note distinguishes the contiguous block, which indicates a failed worker shard, from the isolated cells, which indicate individual render failures. Expected minus written — the shape of the difference names the cause expected (from registered extent) written (counted in the object store) contiguous block → a worker shard died mid-range isolated cells → individual render failures, check render_error_total by reason

Problem framing: counting is not detecting

The naive check compares today’s tile count against yesterday’s. It fails in the two cases that matter.

It fails when the extent grows. A layer that gained a new region should have more tiles; a count comparison reports growth and passes, while the new region is entirely missing and the old one grew by a rebuild artefact. Only a comparison against the extent-derived expectation notices.

It fails when the gap is old. A pyramid that has been missing four hundred tiles for a month has a stable count, so a day-over-day comparison sees nothing wrong. The gap becomes invisible precisely because it persisted, which is the opposite of what you want.

The expectation-based check has one real cost: at high zoom the expected set is enormous. A modest national layer at zoom 14 covers millions of tiles, and listing an object store to count them on every scrape is neither cheap nor kind to the store. The practical designs all reduce that cost by sampling or by maintaining the count incrementally rather than by measuring it.

Implementation: an affordable expectation check

Three techniques, in increasing order of cost and precision. Most platforms want the first two.

Manifest accounting. The build worker already knows how many tiles it intended to write and how many writes succeeded. Emitting both as counters gives an exact completeness figure for the run at no additional cost, and it catches every gap introduced by that run.

# Emitted by the build worker — exact, free, and blind to pre-existing gaps.
tiles_expected = meter.create_counter("gis.tile.build_expected_total")
tiles_written  = meter.create_counter("gis.tile.build_written_total")
tiles_failed   = meter.create_counter("gis.tile.build_failed_total")

for zoom in levels:
    planned = expected_count(extent, zoom)
    tiles_expected.add(planned, {"layer": layer, "zoom": str(zoom)})
    for index in plan(extent, zoom):
        try:
            store.put(render(index))
            tiles_written.add(1, {"layer": layer, "zoom": str(zoom)})
        except RenderError as exc:
            # `reason` is what turns a count into a diagnosis later.
            tiles_failed.add(1, {"layer": layer, "zoom": str(zoom),
                                 "reason": exc.kind})

Stratified sampling against the store. Manifest accounting cannot see a tile that was written and later deleted, or a gap that predates your instrumentation. A periodic sampled probe covers that: draw a few hundred tile indices uniformly from the expected set at each level, HEAD each one, and report the hit ratio. Uniform sampling of a few hundred cells detects any gap covering more than roughly one percent of a level with high probability, which is the size range that matters, and it costs a few hundred requests per level per sweep rather than millions.

import random

def sampled_completeness(layer, extent, zoom, n=400):
    x0, x1, y0, y1 = tile_range(*extent, zoom)
    total = (x1 - x0 + 1) * (y1 - y0 + 1)
    n = min(n, total)
    hits = 0
    for _ in range(n):
        x = random.randint(x0, x1)
        y = random.randint(y0, y1)
        if store.exists(layer, zoom, x, y) or is_legitimately_empty(layer, zoom, x, y):
            hits += 1
    return hits / n            # gauge: gis.tile.level_sampled_complete_ratio

Full enumeration at low zoom only. Levels below roughly zoom 10 have small enough tile counts to enumerate exhaustively, and a gap at low zoom is disproportionately visible to users because it covers a large area. Enumerate cheaply where you can; sample where you cannot.

The is_legitimately_empty predicate is what keeps the ratio honest. A land-cover layer has no tiles over open ocean, and a layer clipped to an administrative boundary has none outside it. Deriving the predicate from the same geometry that produced the extent — a coverage mask, not a hand-maintained exclusion list — is what stops the check from decaying as the layer changes.

Cost and coverage trade-off across three completeness techniques Three techniques are compared on two axes rendered as labelled bars. Manifest accounting has negligible cost and detects only gaps created by the current run. Stratified sampling has low cost proportional to a few hundred requests per level and detects any gap larger than about one percent of a level. Full enumeration has cost proportional to the tile count and detects every gap including single tiles, which makes it affordable only at low zoom. A recommendation line suggests combining manifest accounting on every run, sampling on a schedule, and enumeration below zoom ten. Pick the technique by what it can see, then by what it costs Manifest accounting cost: none · already known by the worker sees: gaps this run created Stratified sampling cost: ~400 HEADs per level per sweep sees: any gap over ~1% of a level Full enumeration cost: proportional to tile count sees: every gap, down to one tile relative sweep cost at zoom 14 manifest sampling full enumeration Run all three: manifest on every build, sampling hourly, enumeration nightly below zoom 10. Probability of missing a gap against sample size, for three gap sizes Three curves plot the probability that a uniform sample fails to detect a gap, against the number of cells sampled. For a gap covering five percent of a level, detection is near certain by fifty samples. For one percent, four hundred samples reduce the miss probability to under two percent. For one tenth of one percent, even two thousand samples leave a substantial miss probability. A note states that gaps below one percent of a high-zoom level are rarely user-visible, which is why four hundred is a sensible budget. Miss probability falls fast for gaps that matter gap = 5% of the level gap = 1% gap = 0.1% 50400 1 0002 000 samples miss

Verification: prove the check can see a gap you made

Delete a known block of tiles in a staging pyramid and confirm each mechanism reacts as designed. The sampled probe should show the ratio fall in proportion to the deleted fraction; the manifest counters should be unaffected, since the deletion happened outside a build; the low-zoom enumeration should report the exact missing indices if the block reaches that far up the pyramid.

That contrast is the point of the exercise. It demonstrates concretely that manifest accounting cannot see post-hoc deletion, which is the argument for keeping the sampled probe even though the manifest numbers look authoritative.

Then verify the empty-cell predicate by deleting a block that should be empty — ocean cells in a land layer — and confirming the ratio does not move. A predicate that is too narrow produces a permanently degraded ratio that people learn to ignore.

Gotchas

Sampling uniformly over a heavily clipped extent. If most of the bounding box is legitimately empty, uniform sampling spends nearly all its budget on cells that do not exist and the effective sample size collapses. Sample from the coverage mask rather than from the bounding rectangle.

Reporting completeness as a layer average. A gap confined to one level disappears into an average across levels, exactly as shown in the parent topic. Always report and alert on the worst level.

Counting a zero-byte object as present. A failed write that left a truncated object satisfies an existence check while rendering blank. Where the store exposes object size cheaply, treat implausibly small objects as absent; the size distribution alert in the parent topic catches the systemic version of this.

Letting the expectation drift from the registry. If the extent used by the checker is cached separately from the one used by the builder, a boundary change silently makes the two disagree and completeness reads above or below one for reasons unrelated to any gap. Read both from the same registry entry.

Probing only the deepest level. High zoom holds the most tiles, so it feels like the place to look, but a gap at low zoom covers vastly more ground per missing tile and is far more visible to users. Sweep every instrumented level on the same schedule rather than concentrating the sample where the count is largest.

FAQ

How large a sample do I need per level?

For detecting a gap covering a fraction pp of a level, a sample of nn cells misses it with probability (1−p)n(1-p)^n. At p=0.01p = 0.01 and n=400n = 400 that is about 1.8%, which is comfortable for an hourly sweep. Halving the detectable gap size requires doubling the sample, so pushing much below one percent gets expensive quickly — and gaps smaller than one percent of a high-zoom level are rarely user-visible.

Should the probe hit the object store or the serving endpoint?

Both, for different reasons. Probing the store answers “was it built”; probing the serving endpoint answers “can a client get it”, which additionally covers cache and routing faults. Running the same sampled index set against both and comparing is a cheap way to localise a fault to build versus serve.

What about layers where the extent legitimately changes daily?

Recompute the expectation from the registry on every sweep rather than caching it. The cost is one extent lookup; the benefit is that a shrinking extent shows up as an extent change rather than as a mysterious completeness gain.

How does this relate to coverage monitoring for vector layers?

They answer the same question in different media: is the data present everywhere it should be. The vector-side machinery, including the baseline-extent comparison, is covered in spatial coverage and extent monitoring, and the two checks should share the registered extent so a boundary change updates both.

Can I skip this if the builder reports success?

No — that is exactly the manifest-accounting blind spot. A builder reports on what it attempted; it cannot report on a tile that a later lifecycle policy expired, an operator deleted, or a partial multipart upload left truncated. The sampled probe exists precisely to cover the gap between “the build said it worked” and “the tiles are there”.