Architecting Deterministic Fallback Routing for Spatial Feature Servers

When a primary OGC API - Features server degrades, the failure rarely announces itself as a clean 5xx. It leaks: p95 latency creeps past the tile-publish SLO, a replica returns geometries in the wrong SRID, or a cached snapshot serves topology that was valid yesterday. This page tackles one narrow operational problem — building a fallback routing layer that deterministically shifts traffic across feature-server tiers without silently downgrading coordinate-reference-system fidelity, attribute completeness, or geometry validity. It sits under Observability Scoping Rules for Vector Data and within the broader Geospatial Observability Architecture & Fundamentals domain, and it assumes you already emit the gis.spatial.* signals those pages define. The audience is the SRE or data-platform engineer who owns the gateway in front of a PostGIS-backed feature service and needs failover that an auditor can reconstruct request-by-request.

Deterministic fallback routing across three spatial feature-server tiers An incoming /ogc/features request reaches a routing gateway. The gateway routes to Tier 1, the primary cluster, while it is healthy; failing over to Tier 2, a cross-region replica, when serving-zone p95 latency or the error budget is breached; and finally to Tier 3, a static snapshot, when the replica is also unhealthy. Tier 1 emits gis.spatial.routing.tier=1, Tier 2 emits tier=2 with CRS validation, and Tier 3 emits tier=3 with a lineage tag. All three tiers feed an observability perimeter that tags and traces every served request. /ogc/features request Routing gateway Tier 1 · primary cluster emits gis.spatial.routing.tier=1 Tier 2 · cross-region replica tier=2 · CRS validation Tier 3 · static snapshot tier=3 · lineage tag healthy p95 / budget breach replica unhealthy Observability perimeter every tier tagged & traced

Problem Framing

A naive primary/secondary toggle treats every non-200 as interchangeable, which is exactly wrong for spatial workloads. A feature server under index-rebuild contention can return structurally valid GeoJSON that is hours stale; a misconfigured replica can return fresh data reprojected to the wrong EPSG code. Both pass an HTTP health check. The signal that should drive failover is therefore not “is the origin up” but “is the origin serving geometry that still satisfies the trust contract” — the same distinction defining spatial data trust boundaries draws between a payload that is reachable and one that is trustworthy.

This scenario arises whenever you place more than one feature server behind a single endpoint: a regional primary, a cross-region read replica, and a degraded snapshot cache. The pipeline stage affected is the serving zone, the last hop before web-map clients and downstream ETL consumers read features. The signals that indicate you need deterministic routing are concrete: a rising gis.spatial.geometry_invalid_ratio on responses, gis.spatial.crs_mismatch_count spiking on one upstream, or serving-zone p95 crossing the threshold while the database itself reports healthy. Fallback routing that ignores those signals will happily promote a degraded tier and convert a recoverable blip into a contaminated tile cache. The degradation primitives themselves — what each tier is allowed to drop — are catalogued in fallback chains for spatial API failures; this page is about wiring the routing decision that selects among them.

Implementation

The routing layer is an Envoy aggregate cluster that enforces a strict priority order — Tier 1 primary, Tier 2 cross-region replica, Tier 3 static snapshot — with outlier detection and health checks tuned to spatial timeouts rather than generic web defaults. The connect_timeout and per_try_timeout values are deliberately tight because a feature query that has not returned a first byte in 500 ms is, for an interactive map client, already a failure.

# fallback_routing_config.yaml — Envoy aggregate cluster, deterministic tier priority
static_resources:
  clusters:
  - name: tier1_primary
    type: STRICT_DNS
    connect_timeout: 2s
    lb_policy: ROUND_ROBIN
    outlier_detection:
      consecutive_5xx: 3              # eject after 3 consecutive 5xx — fast, deterministic
      interval: 10s
      base_ejection_time: 120s        # matches the circuit-breaker half-open window below
      max_ejection_percent: 50        # never blackhole the whole primary on a partial fault
      enforcing_consecutive_5xx: 100
    health_checks:
    - timeout: 5s
      interval: 10s
      unhealthy_threshold: 3
      healthy_threshold: 2
      http_health_check:
        path: /healthz                # must assert CRS + freshness, not just process liveness
        expected_statuses: [{ start: 200, end: 299 }]

  - name: tier2_replica
    type: STRICT_DNS
    connect_timeout: 3s               # cross-region: allow one extra second of RTT
    lb_policy: ROUND_ROBIN
    health_checks:
    - timeout: 5s
      interval: 10s
      unhealthy_threshold: 3
      http_health_check: { path: /healthz }

  - name: tier3_snapshot
    type: STATIC                      # degraded vector cache — last-resort read availability
    load_assignment:
      cluster_name: tier3_snapshot
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address: { address: 10.0.5.10, port_value: 8443 }

  - name: spatial_failover             # aggregate: deterministic tier1 -> tier2 -> tier3
    connect_timeout: 2s
    lb_policy: CLUSTER_PROVIDED
    cluster_type:
      name: envoy.clusters.aggregate
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.clusters.aggregate.v3.ClusterConfig
        clusters: [tier1_primary, tier2_replica, tier3_snapshot]

  listeners:
  - name: spatial_gateway
    address:
      socket_address: { address: 0.0.0.0, port_value: 8080 }
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: spatial_ingress
          route_config:
            name: local_route
            virtual_hosts:
            - name: spatial_api
              domains: ["*"]
              routes:
              - match: { prefix: "/ogc/features" }
                route:
                  cluster: spatial_failover
                  timeout: 1.5s
                  retry_policy:
                    retry_on: "5xx,reset,connect-failure,retriable-status-codes"
                    num_retries: 3
                    per_try_timeout: 0.5s   # a feature query past 500ms TTFB is already failed

The decisive detail is the /healthz contract: a check that only asserts process liveness will keep a stale or mis-projected tier marked healthy. The probe must return non-2xx when the upstream’s gis.spatial.crs_mismatch_count is non-zero or its freshness lag exceeds the serving SLO, so outlier detection ejects on spatial faults, not just crashes.

Failover alone is not recovery. To avoid flapping the primary back into rotation the instant it returns one good response, govern reinstatement with a half-open circuit breaker whose isolation window equals the base_ejection_time above.

Half-open circuit-breaker state machine for primary tier reinstatement A three-state circuit breaker. Closed sends all /ogc/features traffic to Tier 1. When p95 latency is high or the error budget is burned, the breaker transitions to Open and all traffic shifts to Tier 2. After 120 seconds of isolation it transitions to Half-Open, probing Tier 1 with 10 percent of traffic. If the probe success rate exceeds 98 percent over a 60-second window it returns to Closed; if probe failures persist it reverts to Open and extends isolation by a further 240 seconds. Closed 100% → Tier 1 Open 100% → Tier 2 Half-Open 10% probe → Tier 1 p95 high · budget burned after 120s isolation probe failures persist · +240s isolation probe success rate ≥ 98% over 60s → Closed
  1. Closed — 100% of /ogc/features traffic to Tier 1. Monitor p95 latency and the serving-zone error budget.
  2. Open — entered when p95 exceeds 1,500 ms for three consecutive 30-second windows, or the rolling 60-second error rate exceeds 3.5%. All traffic shifts to Tier 2.
  3. Half-Open — entered after 120 s of isolation. Exactly 10% of requests probe Tier 1. If the probe success rate exceeds 98% over a 60-second window, transition to Closed; if the failure rate exceeds 2%, revert to Open and extend isolation by 240 s.

Classify 400 Bad Request responses whose body carries InvalidGeometry or CRSMismatch as circuit-breaking events, not client errors — per the OGC API - Features specification these indicate the upstream is serving payloads the gateway must reject. Then stamp routing lineage onto every request so the tier that served it is reconstructable. The wiring that propagates these attributes is covered in OpenTelemetry integration for GIS pipelines; the resource attributes themselves use the canonical names from the geospatial metric taxonomy for ETL:

# otel-collector-config.yaml — contrib build, stamp routing lineage on every feature span
processors:
  resource:
    attributes:
    - { key: gis.spatial.routing.tier, value: "1", action: insert }
    - { key: gis.spatial.crs, value: "EPSG:4326", action: insert }
    - { key: gis.spatial.topology.validated, value: "true", action: insert }

Verification & Testing

Confirm the routing layer behaves deterministically before you trust it in an incident. Drive failover synthetically by ejecting the primary (block its /healthz at the host firewall) and assert that traffic moves to Tier 2 within one outlier-detection interval. The PromQL rules below are what should fire — wire them to the runbook that flips the circuit breaker, and verify each one triggers against injected load:

# prometheus_alert_rules.yml
groups:
- name: spatial_fallback_routing
  rules:
  - alert: SpatialFeatureServerLatencyDegradation
    expr: >
      histogram_quantile(0.95,
        sum(rate(gis_spatial_request_duration_seconds_bucket{path=~"/ogc/features.*"}[30s]))
        by (le)) > 1.5
    for: 90s
    labels: { severity: critical, routing_action: "trigger_tier2_fallback" }
    annotations: { summary: "p95 latency over 1500ms for 3 consecutive 30s windows" }

  - alert: SpatialFeatureServerErrorRateSpike
    expr: >
      sum(rate(gis_spatial_requests_total{status=~"5..|400",path=~"/ogc/features.*"}[60s]))
      / sum(rate(gis_spatial_requests_total{path=~"/ogc/features.*"}[60s])) > 0.035
    for: 60s
    labels: { severity: warning, routing_action: "initiate_circuit_breaker" }
    annotations: { summary: "Rolling 60s error rate over 3.5%" }

Then validate that the content of the fallback response is trustworthy, not merely that bytes arrived. Run a targeted PostGIS assertion against whatever tier currently serves the endpoint, comparing geometry validity and SRID against the pipeline baseline:

-- Confirm the active fallback tier serves valid, correctly-projected geometry
SELECT ST_IsValid(geom) AS valid, ST_SRID(geom) AS srid, ST_NPoints(geom) AS vertices
FROM fallback_feature_cache
WHERE feature_id IN (SELECT id FROM recent_fallback_requests LIMIT 100);

If any row returns valid = false or an SRID that deviates from the baseline EPSG code, the tier is unfit to serve — halt promotion and pin traffic to the last known-good tier. Finally, inspect Envoy’s own counters to confirm the ejection timing matches the circuit-breaker window:

curl -s http://localhost:9901/stats \
  | grep -E "cluster.tier1_primary.outlier_detection|circuit_breakers"

The reported base_ejection_time should align with the 120 s half-open threshold; if max_ejection_percent caps earlier than expected, a partial fault is tripping more hosts than intended and the outlier parameters need widening.

Gotchas & Failure Modes

  • A liveness-only /healthz masks stale and mis-projected tiers. The most common failure is a health check that returns 200 while the upstream serves hours-old or wrong-SRID features. Outlier detection then never ejects, and clients read degraded geometry indefinitely. The probe must assert freshness and CRS, not just that the process is alive — otherwise failover is reacting to crashes only, the rarest spatial failure mode.
  • Retries amplify load on a database already in topology repair. A num_retries: 3 policy against a primary stuck in a long ST_MakeValid or REINDEX turns one slow query into four, accelerating the very collapse you are trying to survive. Keep per_try_timeout tight (0.5 s) so retries fail fast onto Tier 2 rather than queueing behind a contended primary, and confirm the retry budget cannot exceed the upstream’s connection pool.
  • Tier 3 snapshots drift out of CRS alignment silently. A static vector cache built before a reprojection or schema migration will serve structurally valid geometry in an obsolete SRID — passing ST_IsValid while failing the trust contract, exactly the cross-region drift tracked in monitoring topology for multi-region GIS. Tag every snapshot with its build-time EPSG and a lineage ID, and reject reads whose tag deviates from the live baseline rather than assuming the cache is safe because it responds.

FAQ

Why route on spatial signals instead of HTTP status alone?

Because the most damaging feature-server failures return HTTP 200. A replica serving stale tiles or geometry reprojected to the wrong EPSG is reachable and “healthy” by any generic check, yet it corrupts every downstream map and join. Driving failover from gis.spatial.crs_mismatch_count, geometry-validity ratios, and freshness lag catches the faults that a status code cannot see.

How do I keep the circuit breaker from flapping the primary?

Tie the isolation window to the base_ejection_time (120 s here) and require a probe success rate above 98% over a full 60-second window before returning to Closed. Probing with exactly 10% of traffic means a still-degraded primary fails the evaluation on a small blast radius and re-isolates with an extended 240 s window, instead of oscillating users between a good and a bad tier.

Should Tier 3 ever serve writes?

No. The static snapshot is a read-availability tier of last resort. Allowing writes against a degraded cache creates divergent state that cannot be reconciled when the primary recovers. Fallback routing for the /ogc/features read path should hard-fail writes during Tier 3 operation and surface that explicitly to clients.

What does an auditor need to reconstruct a fallback event?

Every request must carry the gis.spatial.routing.tier span attribute plus an X-Data-Lineage-ID header. With those, the audit trail shows which tier served each feature and in what CRS during the outage window. Missing lineage metadata means a proxy misconfiguration — treat it as a rollback trigger, because you can no longer prove which data tier answered a given request.

How do I tune the 1,500 ms p95 and 3.5% error thresholds?

Derive them from the serving-zone SLO, not from defaults. Set the p95 gate just above the latency at which interactive map clients begin abandoning tile requests, and set the error-rate gate a few points above the clean-baseline 4xx/5xx rate so transient validation rejections do not trip failover. Feeds with chronic minor geometry faults justify a higher gate paired with an upstream ST_MakeValid repair step.