Detecting Divergent Writes Across Spatial Replicas

Replication lag is a well-understood signal: the replica is behind, you can measure by how much, and it catches up. Divergence is different and worse. Two regional stores that both accepted writes during a partition, or a replica that silently dropped a batch and kept serving, do not converge on their own — they hold different answers to the same spatial query, indefinitely, while every lag metric reads zero. This guide covers how to detect divergence between spatial replicas using content fingerprints rather than lag, how to localise the disagreement to a region and a batch, and how to decide which side wins. It belongs to monitoring topology for multi-region GIS under geospatial observability architecture fundamentals.

Lag versus divergence: two replicas that agree on lag and disagree on content Two replicas are compared on two rows. The lag row shows both replicas reporting zero seconds behind the primary, so a lag-based check reports both healthy. The content row shows per-grid-cell fingerprints for the same layer: most cells match between the replicas, but three cells hold different digests. An annotation states that lag has fully caught up while the content permanently disagrees, and that only a content comparison can see it. Zero lag, permanent disagreement replica eu · lag 0 s a lag check reports healthy replica apac · lag 0 s a lag check reports healthy per-cell content digests 3 cells differ the same 3 cells differ Grid-cell digests localise the disagreement immediately — and a whole-table checksum would only say “different”.

Problem framing: lag and divergence are different faults

Lag is a temporal difference that resolves itself. Divergence is a content difference that does not, and it arises from three mechanisms in spatial platforms.

Accepted writes on both sides of a partition. Multi-region deployments that permit regional writes for latency reasons will, during a network partition, accept conflicting edits to the same feature. When the partition heals, last-writer-wins silently discards one edit and the regions agree — or the conflict resolution differs per region and they do not.

A dropped or partially-applied batch. A replica that failed mid-apply and resumed from the wrong position ends up missing a slice of features, catches up on lag, and serves a permanently smaller layer.

Divergent derived state. Even with identical base tables, regionally-computed derived layers — simplified geometries, tile pyramids, materialised join results — can diverge because a regional worker used a different tolerance, a different boundary version, or a different library build.

None of these move a lag metric. All of them produce regionally inconsistent answers to the same spatial query, which manifests to users as “the map shows different things depending on where I am”.

There is a fourth mechanism worth naming because it is the hardest to accept: deliberate regional difference that was never written down. A regional team applies a local correction — a boundary that the national dataset gets wrong for their jurisdiction, an address alias, a suppressed feature — directly in their replica, entirely reasonably, and never tells anyone. From the platform’s perspective this is indistinguishable from corruption. Detecting it is still valuable, because the right response is to move the correction upstream into the source of truth where every region gets it, rather than to keep discovering it during incidents.

Implementation: grid-cell content fingerprints

A whole-table checksum detects divergence and localises nothing, so it is nearly useless operationally. Fingerprinting per spatial grid cell gives the same detection plus an immediate answer to where.

-- Content fingerprint per coarse grid cell. Run identically on every replica.
-- The cell key must be deterministic and coarse enough to keep the row count
-- small (a few thousand cells), and it must NOT be a raw geohash.
SELECT
  ST_SnapToGrid(ST_Centroid(geom), 0.5)::text                     AS cell,
  COUNT(*)                                                        AS features,
  md5(string_agg(
        parcel_id || ':' || md5(ST_AsBinary(geom)) || ':' || COALESCE(land_use, ''),
        ',' ORDER BY parcel_id))                                  AS digest
FROM prod.parcels
GROUP BY 1;

Three details make this reliable. The aggregate must be ordered — string_agg without ORDER BY produces a different digest for identical content depending on scan order, which yields permanent false divergence. The geometry must be hashed from a canonical binary form rather than from text, because text representations differ in precision between versions. And the cell size must be coarse: half a degree over a national extent gives a few thousand rows, which compares in milliseconds, whereas a fine grid produces a comparison as expensive as the divergence it is looking for.

Comparing replicas is then a join over the fingerprint output.

-- Collected centrally: one fingerprint set per replica, compared pairwise.
SELECT
  COALESCE(a.cell, b.cell)                       AS cell,
  a.features                                     AS eu_features,
  b.features                                     AS apac_features,
  CASE
    WHEN a.cell IS NULL          THEN 'missing_in_eu'
    WHEN b.cell IS NULL          THEN 'missing_in_apac'
    WHEN a.digest <> b.digest
     AND a.features = b.features THEN 'content_differs'
    WHEN a.digest <> b.digest    THEN 'count_and_content_differ'
  END                                            AS disagreement
FROM fingerprints.parcels_eu   a
FULL OUTER JOIN fingerprints.parcels_apac b USING (cell)
WHERE a.digest IS DISTINCT FROM b.digest;

Separating content_differs from count_and_content_differ is worth the extra branch. Equal counts with different digests points at conflicting edits to existing features — the partition case. Different counts points at a dropped or partial batch. The two have different remediations and the query names which one you have.

Emit the disagreement count as a gauge dimensioned by replica pair, and alert on any non-zero sustained value.

- alert: SpatialReplicaDivergence
  expr: max by (layer, replica_pair) (gis_replica_divergent_cells) > 0
  for: 15m
  labels: { severity: critical, data_domain: spatial }
  annotations:
    summary: >-
      {{ $value }} grid cells differ between {{ $labels.replica_pair }}
      for {{ $labels.layer }}
    runbook_url: "/spatial-incident-response-and-tooling/spatial-pipeline-incident-runbooks/cross-region-replication-lag-runbook/"

The fifteen-minute sustain matters: a replica mid-apply will legitimately show transient disagreement, and alerting instantly turns normal replication into a page. What must not be tolerated is disagreement that persists past the expected apply window, since that is divergence rather than lag.

Decision path from a divergence signal to the correct remediation A decision tree starts from detected divergence. The first question asks whether feature counts differ. If they do, the diagnosis is a dropped or partial batch and the remediation is to re-apply the missing range from the source of truth. If counts are equal, a second question asks whether the divergent cells cluster in one geographic area. If they do, the diagnosis is conflicting regional writes during a partition and the remediation is to resolve by the declared conflict policy and re-emit. If they are scattered, the diagnosis is a derived-state difference such as a differing simplification tolerance or library build, and the remediation is to rebuild the derived layer from a pinned configuration. The shape of the disagreement names the remediation counts differ?   cells clustered geographically? dropped or partial batch re-apply the missing range from the source of truth; check the apply position conflicting regional writes resolve by the declared conflict policy, re-emit, and review whether regional writes are worth it derived-state difference rebuild from a pinned tolerance and library build; the base tables are fine yes no yes no Comparison cost against grid cell size for a national layer Four grid cell sizes are compared on the cost of a full fingerprint comparison. A half-degree grid produces a few thousand cells and compares in milliseconds. A tenth of a degree produces tens of thousands and compares in under a second. A hundredth produces millions and takes minutes. A thousandth produces hundreds of millions and is impractical. The note states that localisation improves only slightly across these while cost rises by orders of magnitude. Coarse cells detect just as well and compare orders of magnitude faster 0.5 degree cells ~4k cells, milliseconds 0.1 degree cells ~90k cells, under a second 0.01 degree cells ~9M cells, minutes 0.001 degree cells impractical

Verification

Manufacture each divergence class in staging. Delete a hundred features from one replica and confirm the count-and-content branch fires with the correct cells. Update a single feature’s geometry on one side only and confirm the content-only branch fires on exactly one cell. Rebuild a derived layer with a different simplification tolerance on one replica and confirm the scattered-cell signature appears.

Then confirm the fingerprint is stable. Run it twice on an unchanged replica and assert identical digests. Instability almost always means an unordered aggregate or a text geometry representation, and it will otherwise generate a permanent false alert that trains everyone to ignore the real one.

Gotchas

Unordered aggregation in the digest. Produces different digests for identical content. The single most common defect in this check.

Grid cells too fine. A comparison over hundreds of thousands of cells is expensive enough that it gets run rarely, which defeats the purpose. Coarse cells detect equally well and localise well enough.

Comparing during the apply window. Transient disagreement is normal; alert only on disagreement that outlives the expected replication delay.

Fingerprinting only the geometry. Attribute divergence is just as damaging and invisible to a geometry-only digest. Include the attributes that consumers actually read.

No declared conflict policy. When conflicting regional writes are found, there must already be a rule for which side wins. Deciding it during an incident produces inconsistent outcomes across incidents.

FAQ

How often should the comparison run?

Hourly for most layers, and immediately after any failover or partition heal. The fingerprint query is cheap enough to run more often, but divergence is not a fast-moving fault and hourly gives a bounded detection window without adding load.

Should divergence page overnight?

Yes for layers where regional consistency is a stated guarantee, because divergence does not self-heal and every hour widens the set of consumers who received an inconsistent answer. For layers explicitly documented as eventually consistent with a long bound, the daily queue is adequate.

Does this replace lag monitoring?

No. Lag catches the common, transient case cheaply; divergence catches the rare, permanent one. Running both is the point — a platform with only lag monitoring is blind to exactly the faults that need human intervention.

What about three or more replicas?

Fingerprint each against a designated reference rather than pairwise, which keeps the comparison count linear. Where no natural reference exists, comparing each against the primary and reporting per-replica divergence is usually clearer than a full mesh.

Can the fingerprints be reused for anything else?

They make an excellent restore check. After recovering a replica from backup, comparing its fingerprints against the primary answers “is this restore complete and correct” in one query, rather than by counting rows and hoping. The same output also supports a cheap regression check when upgrading a spatial library: fingerprint before and after, and any change in a digest means the upgrade altered geometry serialisation, which is worth knowing before it reaches a replica comparison as mysterious divergence.

How does this interact with the replication lag runbook?

It supplies the distinction the runbook needs at its first step: whether the replica is behind or wrong. The remediation paths differ completely, and starting the cross-region replication lag runbook on a divergence incident wastes the first twenty minutes chasing a lag that is already zero.

One last practical note: keep the fingerprint function itself under version control and pinned per layer. A change to how the digest is computed makes every replica appear divergent simultaneously, which is an unmistakable signature once you know to look for it and a bewildering incident when you do not.