Fallback Chains for Spatial API Failures

A fallback chain for spatial API failures is the stateful routing layer that keeps a geospatial pipeline serving correct geometry when an upstream geocoding, reverse-lookup, isochrone, or tile provider degrades — instead of letting one vendor’s latency surge or 5xx storm cascade into a global pipeline halt. The hard part is not the retry; it is degrading without silently losing spatial fidelity. A naive fallback that swaps a high-precision provider for a simplified-geometry mirror will pass every transport-layer health check while quietly shifting coordinates, collapsing vertices, or returning the wrong CRS. This page is for the data engineers, GIS platform administrators, and SREs who own that risk. It sits under Geospatial Observability Architecture & Fundamentals and shows how to build a tiered router, instrument each hop, and alert on geometric degradation — not just availability.

Architecture

Resilient spatial ETL pipelines require deterministic fallback topologies that isolate upstream provider degradation from downstream analytics, mapping, and routing services. The fallback chain operates as a stateful routing layer positioned between ingestion orchestrators and external geospatial APIs. It implements a tiered circuit breaker model that evaluates endpoint health through active synthetic probing, passive error sampling, and real-time quota utilization tracking. When a primary geocoding, reverse-lookup, or isochrone API exhibits sustained latency degradation or HTTP 5xx surges, the router atomically shifts traffic to a secondary vendor or regional mirror. If both external tiers exhaust their error budgets, requests route to a locally materialized spatial cache backed by PostGIS or DuckDB spatial extensions. These tiers are the reusable degradation primitives that the defining spatial data trust boundaries model invokes when a boundary is breached, so the same routing layer protects both ingress validation and external enrichment calls.

Each tier maintains isolated connection pools, independent rate limiters, and strict request serialization contexts to prevent cross-tier state contamination. The routing layer enforces immutable spatial contracts: coordinate reference systems (CRS), bounding box constraints, and precision tolerances are validated before and after every hop, so fallback execution never silently alters geometric fidelity or attribute schemas. Bounding-box scoping at the router edge — the same admission logic described in observability scoping rules for vector data — rejects out-of-scope or malformed coordinates before they consume a provider’s quota and trigger needless fallback activation. Multi-region deployment patterns further isolate failure domains by anchoring fallback chains to specific availability zones, allowing traffic to bypass regional network partitions or provider outages; the cross-region reconciliation rules are detailed in monitoring topology for multi-region GIS.

The router uses a three-state machine (CLOSED, OPEN, HALF_OPEN) with spatial-aware thresholds. A boundary is only as trustworthy as its declared contract, so the same versioned policy file drives both the runtime router and the alert thresholds — drift between “what we route on” and “what we alert on” is impossible by construction:

# fallback_router.yaml
routing_policy:
  primary:
    endpoint: "https://api.vendor-a.com/v3/geocode"
    health_check:
      active_interval: 15s
      probe_payload: '{"address": "1600 Amphitheatre Pkwy, Mountain View, CA"}'
      success_criteria:
        latency_p95: 800ms
        http_success_rate: 0.98
  secondary:
    endpoint: "https://geo-mirror.vendor-b.com/v2/lookup"
    activation_trigger:
      consecutive_failures: 5        # OPEN the breaker after 5 in a row
      quota_utilization: 0.85        # shed early before the primary 429s
  local_cache:
    engine: "duckdb_spatial"
    table: "spatial_cache.geocode_materialized"
    fallback_trigger:
      circuit_state: "OPEN"
      max_age_hours: 72              # stale-but-located beats null geometry
  spatial_contract:
    enforce_crs: "EPSG:4326"         # validated on entry AND exit of every tier
    max_precision_loss_cm: 5         # vertex precision budget across the hop
    bbox_validation: strict

Metric Specification

Fallback observability requires strict separation between transport-layer telemetry and domain-specific spatial measurements. Standard API metrics (request duration, retry counts, error rates) provide necessary infrastructure visibility but fail to capture geometric degradation, projection mismatches, or silent coordinate truncation that occur during tier transitions. The measurement framework uses the canonical namespace from the geospatial metric taxonomy for ETL so that a gis.spatial.* series means the same thing whether it was served by the primary, the mirror, or the cache, and it categorizes telemetry into reliability, fidelity, and provenance dimensions.

Reliability metrics track circuit-state transitions and how often traffic leaves the primary tier. Fidelity metrics compute geometric divergence between the primary’s expected output and the fallback’s actual output using a spatial distance algorithm — the normalized Hausdorff distance — so a “successful” 200 response that returns the wrong polygon still raises an alarm. Provenance metrics enforce immutable lineage, logging which tier served each request alongside data-residency tags, attribute-mapping checksums, and precision-loss quantification.

To collapse a fallback decision into a single comparable health number, score each served response by combining its tier availability with its geometric fidelity. Let $a_t \in [0,1]$ be the rolling availability of tier $t$ and let $\delta$ be the measured Hausdorff distance in metres against a tolerance ceiling $\delta_{\max}$. The effective fidelity-weighted service score is:

$$S_{\text{fallback}} = a_t \cdot \left(1 - \min\!\left(\frac{\delta}{\delta_{\max}}, 1\right)\right)$$

A response served from a healthy mirror ($a_t \approx 1$) but with $\delta$ approaching $\delta_{\max}$ scores near zero — correctly flagging that the pipeline is “up” but geometrically wrong. The degradation index $\delta / \delta_{\max}$ is what the histogram above records, and any value above 0.5 m for geocode operations pages the owning team before the corrupted coordinate reaches a published layer.

Pipeline Integration & Configuration

Instrumenting a fallback chain means propagating spatial context across every hop so a payload is traceable from the failed primary call through the tier that ultimately served it. Standard OpenTelemetry semantic conventions cover HTTP transport but lack native spatial attributes, so custom dimensions must be injected into spans to maintain trace continuity across fallback hops — the broader wiring is covered in OpenTelemetry integration for GIS pipelines, and the build-versus-buy trade-off between vendor-agnostic tracing and domain-specific fidelity tracking is weighed in comparing OpenTelemetry vs custom metrics for GIS.

Deploy the routing layer as a sidecar or API-gateway plugin so every spatial API call traverses the proxy before hitting an external endpoint. Pre-warm the PostGIS/DuckDB cache from historical request logs, index geometries with GiST, and apply ST_Transform to enforce a unified CRS. Inject X-Spatial-Fallback-Tier and X-Request-Trace-ID headers and map them to OTel baggage so lineage survives async workers. The validator below records the Hausdorff distance between the primary’s expected geometry and whatever the fallback tier returned, attaches the serving tier as a span attribute, and emits the fidelity histogram:

# spatial_fallback_router.py — runs in the proxy, one span per fallback hop
from opentelemetry import trace, metrics
from shapely.geometry import shape

tracer = trace.get_tracer("gis.spatial.fallback_router")
meter = metrics.get_meter("gis.spatial.fallback")

degradation_histogram = meter.create_histogram(
    "gis.spatial.fidelity.degradation_index",
    unit="m",
    description="Hausdorff distance between primary and fallback geometries",
)

def validate_spatial_fidelity(primary_geom: dict, fallback_geom: dict, fallback_tier: str):
    with tracer.start_as_current_span("spatial_fidelity_check") as span:
        g1 = shape(primary_geom)
        g2 = shape(fallback_geom)
        # hausdorff_distance on a Polygon walks the exterior ring automatically
        hausdorff = g1.hausdorff_distance(g2)

        span.set_attribute("gis.spatial.crs", primary_geom.get("crs", "EPSG:4326"))
        span.set_attribute("gis.spatial.fallback_tier", fallback_tier)
        span.set_attribute("gis.spatial.hausdorff_distance_m", hausdorff)

        degradation_histogram.record(
            hausdorff, attributes={"tier": fallback_tier, "operation": "geocode"}
        )

        if hausdorff > 0.5:
            span.add_event("gis.spatial.fidelity_breach", {"distance_m": hausdorff})
            return False
        return True

The local-cache tier must store coordinates at full precision or it becomes its own source of silent truncation. Materialize the cache with explicit precision and double precision storage, and validate vertex counts on read so a degraded cache rebuild is caught before it serves:

-- spatial cache materialization — preserve precision, enforce CRS
CREATE TABLE spatial_cache.geocode_materialized AS
SELECT
  request_key,
  ST_SetPrecision(ST_Transform(geom, 4326), 1e-7) AS geom,  -- ~1cm grid, EPSG:4326
  ST_NPoints(geom) AS expected_vertices,
  now() AS cached_at
FROM ingest.geocode_log
WHERE ST_IsValid(geom) AND ST_SRID(geom) = 4326;

CREATE INDEX geocode_materialized_gix
  ON spatial_cache.geocode_materialized USING GIST (geom);

Threshold Design & Alerting Logic

Fallback thresholds are tiered by severity so recoverable noise does not page humans while unrecoverable corruption does. A WARNING flags a fallback-activation uptick the cache can absorb; a CRITICAL fires on geometric-fidelity breaches or a breaker stuck open that will corrupt downstream layers; a DYNAMIC_BASELINE tier compares live activation against a rolling historical envelope so a vendor’s routine maintenance window does not constantly re-tune static numbers. These Prometheus rules read the gis_etl-namespaced series exported by the collector:

# WARNING — fallback chain is serving >15% of requests from non-primary tiers
sum by (operation) (rate(gis_etl_gis_spatial_fallback_activation_ratio[5m])) > 0.15

# CRITICAL — Hausdorff degradation p95 exceeds the 0.5 m geocode tolerance
histogram_quantile(0.95,
  rate(gis_etl_gis_spatial_fidelity_degradation_index_bucket[5m])) > 0.5

# CRITICAL — primary tier circuit breaker stuck OPEN for over an hour
max by (tier) (gis_etl_gis_spatial_circuit_open_duration_seconds) > 3600

# CRITICAL — fallback returned geometry in an unapproved CRS
sum by (tier) (rate(gis_etl_gis_spatial_projection_mismatch_total[2m])) > 0

# DYNAMIC_BASELINE — activation ratio is >3x its 7-day average for this operation
sum by (operation) (rate(gis_etl_gis_spatial_fallback_activation_ratio[15m]))
  > 3 * avg_over_time(
      sum by (operation) (rate(gis_etl_gis_spatial_fallback_activation_ratio[15m]))[7d:15m]
    )

Severity assignment must reflect spatial-workload non-linearity: a 0.5 m fidelity drift on a routing-isochrone layer that feeds dispatch decisions has a far larger blast radius than the same drift on a low-zoom choropleth, so page on the operation dimension, not just the global ratio. Where activation latency itself threatens an SLO, the rule should trigger cache promotion rather than merely notifying — a stale-but-located response beats a timeout that propagates back up the ingestion graph.

Failure Modes & Edge Cases

Fallback chains fail in characteristic, diagnosable ways. The following patterns account for most production incidents, and each is wired to a fallback or remediation path so the pipeline degrades instead of collapsing.

• Silent CRS swap on the secondary tier. A mirror returns geometries in EPSG:3857 while the contract expects EPSG:4326; vertex counts and validity look fine, so the response passes every transport check while landing hundreds of metres off. Diagnose by asserting ST_SRID on every fallback payload and watching gis.spatial.projection.mismatch_total; a non-zero counter with green availability is the signature. Validation rules here should align with the coordinate reference system validation checks in the freshness and quality reference.
• Simplified-geometry mirror masking degradation. A cheaper secondary provider returns Douglas–Peucker-simplified polygons that pass ST_IsValid but collapse vertices, inflating gis.spatial.fidelity.degradation_index. Treat any Hausdorff value above the tier’s tolerance as a contract violation, require precision=high in the secondary request, and prefer cached primary outputs over a lossy live mirror. The deeper geometry rules live under geometry validity and topology checks.
• Local cache precision loss. A cache rebuilt without ST_SetPrecision or stored in single precision silently truncates decimal places, so the lowest tier becomes the least faithful. Diagnose with SELECT COUNT(*) FROM spatial_cache.geocode_materialized WHERE ST_NPoints(geom) < expected_vertices; rebuild with explicit precision and double precision storage.
• Circuit breaker stuck OPEN. The active health probe keeps failing because its synthetic payload drifted out of sync with the provider’s current schema, so the breaker never tests recovery and traffic is pinned to the cache long after the primary healed. Inspect gis.spatial.circuit.open_duration_seconds; if it never resets, realign the probe payload and re-verify the HALF_OPEN transition criteria.
• Quota-exhaustion thrash across regions. When a regional mirror is itself degraded, naive failover routes there anyway and inflates the degradation index. Anchor each tier to a specific availability zone and confirm the chain is not routing into an already-degraded region — the regional reconciliation pattern is detailed in monitoring topology for multi-region GIS.

Troubleshooting Checklist

When a fallback alert fires or spatial telemetry lags, work the steps in order:

1. Confirm which tier is serving. Read gis.spatial.fallback.activation_ratio by tier. Sustained ratio above 0.4 means the primary is effectively down, not flapping — treat it as an outage, not a transient.
2. Validate the served CRS. Assert ST_SRID equals the contract target on a sample of fallback payloads and check gis.spatial.projection.mismatch_total. A non-zero counter is a silent-corruption emergency: stop the affected tier.
3. Measure fidelity, not just availability. Compute SELECT ST_HausdorffDistance(geom_primary, geom_fallback) FROM validation_set and compare against the tier tolerance. Drift above the ceiling means the fallback provider’s geometry is unfit even though it answered 200.
4. Audit the breaker state machine. If gis.spatial.circuit.open_duration_seconds never resets, the active probe payload is stale — align it with the current provider schema and re-test the HALF_OPEN recovery path.
5. Inspect cache integrity. Run the ST_NPoints(geom) < expected_vertices check; a degraded cache rebuild is a common hidden source of truncation. Rebuild with ST_SetPrecision(geom, 1e-7) and double precision storage.
6. Flush OTel batches. Verify the exporter is not dropping fidelity spans under backpressure: set OTEL_BSP_MAX_EXPORT_BATCH_SIZE=512 and OTEL_BSP_SCHEDULE_DELAY=1000, and offload heavy Hausdorff computation to a dedicated metrics sidecar so aggregation workers do not block on ST_Distance.
7. Trace the lineage. Once stabilized, confirm every served response carries its serving-tier provenance tag so the breach can be attributed to a specific provider and request window.

Adhere to the OGC Simple Features Access specification when defining the spatial contracts each tier must honour, so enforcement stays interoperable across providers, spatial databases, and downstream analytics, and reference the official OpenTelemetry Python SDK documentation for production-grade context propagation across fallback hops.

Related

• Geospatial Observability Architecture & Fundamentals — the parent guide to instrumenting spatial pipelines end to end.
• Defining Spatial Data Trust Boundaries — the boundary model that invokes these fallback tiers on a breach.
• Geospatial Metric Taxonomy for ETL — canonical gis.spatial.* metric names every tier emits.
• Monitoring Topology for Multi-Region GIS — anchoring fallback chains to healthy availability zones.
• Comparing OpenTelemetry vs Custom Metrics for GIS — choosing instrumentation for fidelity tracking across hops.