Tuning Circuit Breakers for Geocoding Fallbacks

A circuit breaker that is tuned wrong is worse than none at all: set the failure threshold too tight and a geocoding provider’s routine 800 ms latency blip trips the breaker and dumps traffic onto a lower-fidelity mirror for no reason; set it too loose and the breaker stays closed through a real outage, burning quota and stalling ingestion while every request times out. This page is the focused guide to picking the threshold counts, latency budgets, and half-open recovery parameters that make a geocoding fallback chain shed load at the right moment and heal cleanly. It sits under Fallback Chains for Spatial API Failures and within Geospatial Observability Architecture & Fundamentals, for the data engineers, GIS platform admins, and SREs who own geocoding availability.

Circuit breaker states and the parameters that gate each transition A three-state machine. CLOSED serves the primary geocoder and counts failures over a rolling window. When the failure rate crosses the threshold or the p95 latency exceeds its budget, it transitions to OPEN, which serves the fallback mirror and starts a cooldown timer. When the cooldown elapses the breaker moves to HALF_OPEN, which admits a small number of trial probes. If the trials succeed it closes back to primary; if any trial fails it re-opens and doubles the cooldown. CLOSED serve primary count failures OPEN serve mirror cooldown timer HALF_OPEN admit trial probes limited concurrency fail rate > thr or p95 > budget cooldown done trial success × N → close trial fails → re-open, double cooldown

What you are actually tuning

A geocoding circuit breaker has four knobs, and tuning is the act of matching each to the provider’s real behaviour rather than to a copied default. The failure threshold decides how many failures over a rolling window trip the breaker open. The latency budget decides when a slow-but-succeeding provider counts as failing, because a geocoder that answers every request in four seconds is down for practical purposes even though it never returns a 5xx. The cooldown decides how long the breaker stays open before it risks a probe. The half-open trial policy decides how many probes it admits and how many must succeed before it fully closes. Get the threshold and budget right and the breaker trips on genuine degradation; get the cooldown and trial policy right and it recovers without flapping.

The mistake that dominates production incidents is tuning on counts instead of rates. A fixed “5 consecutive failures” threshold behaves completely differently at 10 requests per second than at 1,000: at high volume five failures is noise, at low volume it is a meaningful signal that takes forever to accumulate. Tune on a failure ratio over a rolling window with a minimum request floor, so the breaker needs both a bad enough ratio and enough traffic to be confident. This mirrors the reliability-versus-fidelity split the Fallback Chains for Spatial API Failures architecture draws: the breaker guards availability, while the geometry contract on each hop guards fidelity, and the two must be tuned independently or a breaker that closes on availability will happily serve a mirror that is geometrically wrong.

Every tripped breaker and every fallback hop has to be observable, so the tuning is instrumented with two counters: gis.api.fallback_total, incremented each time a request is served by a non-primary tier, and gis.api.circuit_open_total, incremented each time the breaker transitions to OPEN. Read together they tell you whether the breaker is tripping appropriately (open transitions rare, fallback volume bounded) or thrashing (open transitions frequent, fallback volume spiky). Whether these live as domain-specific counters or ride on generic OTel spans is the trade-off weighed in comparing OpenTelemetry vs custom metrics for GIS.

Threshold and latency-budget math

Express the trip condition as a rolling failure ratio against a floor, plus a latency-budget breach. Let ff be failures and nn total requests in the window WW, with a minimum-volume floor nmin⁡n_{\min}; let Lp95L_{p95} be the observed p95 latency against budget LbudgetL_{\text{budget}}. The breaker trips OPEN when

(fn≥θ  ∧  n≥nmin⁡)  ∨  Lp95>Lbudget\left( \frac{f}{n} \ge \theta \;\wedge\; n \ge n_{\min} \right) \;\vee\; L_{p95} > L_{\text{budget}}

A workable starting point for a geocoding tier is θ=0.5\theta = 0.5, nmin⁡=20n_{\min} = 20 over a W=30sW = 30\text{s} window, and LbudgetL_{\text{budget}} set to roughly 1.5× the provider’s healthy p95 — if the vendor answers in 800 ms when healthy, budget 1,200 ms and treat sustained breach as a trip. The floor nmin⁡n_{\min} is what stops a single failed request at 3 a.m. from opening the breaker; the ratio is what makes the threshold volume-independent. Set the cooldown from the provider’s typical incident length, not a round number: a geocoder that recovers in bursts wants a short cooldown so it probes often, while one with hour-long outages wants exponential backoff so it does not hammer a dead endpoint.

Implementing breaker state with tuned parameters

The breaker below keeps a rolling window of outcomes, trips on the ratio-or-latency condition, and manages the half-open trial with exponential cooldown backoff. Each transition emits the counters so the tuning is measurable in Prometheus.

# geocode_breaker.py — ratio-based circuit breaker with half-open recovery
import time
from collections import deque
from opentelemetry import metrics

meter = metrics.get_meter("gis.api.geocode_breaker")
fallback_total = meter.create_counter("gis.api.fallback_total")
circuit_open_total = meter.create_counter("gis.api.circuit_open_total")

class GeocodeBreaker:
    def __init__(self, theta=0.5, n_min=20, window_s=30,
                 latency_budget_ms=1200, base_cooldown_s=15,
                 half_open_trials=3, half_open_successes=2):
        self.theta = theta
        self.n_min = n_min
        self.window_s = window_s
        self.latency_budget_ms = latency_budget_ms
        self.base_cooldown_s = base_cooldown_s
        self.half_open_trials = half_open_trials
        self.half_open_successes = half_open_successes
        self.state = "CLOSED"
        self.outcomes = deque()          # (ts, ok: bool, latency_ms)
        self.opened_at = 0.0
        self.consecutive_opens = 0       # drives exponential cooldown
        self.trial_ok = 0
        self.trial_seen = 0

    def _prune(self, now):
        while self.outcomes and now - self.outcomes[0][0] > self.window_s:
            self.outcomes.popleft()

    def _cooldown(self):
        # exponential backoff, capped at 8x the base
        return self.base_cooldown_s * min(2 ** self.consecutive_opens, 8)

    def allow(self, tier: str) -> bool:
        now = time.monotonic()
        if self.state == "OPEN":
            if now - self.opened_at >= self._cooldown():
                self.state = "HALF_OPEN"
                self.trial_ok = self.trial_seen = 0
            else:
                fallback_total.add(1, {"tier": tier, "reason": "open"})
                return False              # serve the mirror
        if self.state == "HALF_OPEN" and self.trial_seen >= self.half_open_trials:
            fallback_total.add(1, {"tier": tier, "reason": "half_open_saturated"})
            return False                  # trial slots full; keep shedding
        return True                       # CLOSED, or a HALF_OPEN trial slot

    def record(self, ok: bool, latency_ms: float, endpoint: str):
        now = time.monotonic()
        self.outcomes.append((now, ok, latency_ms))
        self._prune(now)

        if self.state == "HALF_OPEN":
            self.trial_seen += 1
            self.trial_ok += 1 if ok else 0
            if not ok:
                self._open(endpoint)      # a single trial failure re-opens
            elif self.trial_ok >= self.half_open_successes:
                self.state = "CLOSED"
                self.consecutive_opens = 0
            return

        n = len(self.outcomes)
        f = sum(1 for _, o, _ in self.outcomes if not o)
        p95 = sorted(l for _, _, l in self.outcomes)[int(0.95 * (n - 1))] if n else 0
        tripped = (n >= self.n_min and f / n >= self.theta) or p95 > self.latency_budget_ms
        if self.state == "CLOSED" and tripped:
            self._open(endpoint)

    def _open(self, endpoint: str):
        self.state = "OPEN"
        self.opened_at = time.monotonic()
        self.consecutive_opens += 1
        circuit_open_total.add(1, {"endpoint": endpoint})

The two parameters that most often need adjusting after launch are half_open_trials and half_open_successes. Admit too many trials and a still-sick primary absorbs a burst of real traffic that fails; admit too few and recovery detection is slow. Requiring two successes out of a maximum of three trials is a reasonable default — it demands corroboration before fully closing but does not stall on a single unlucky probe. The exponential cooldown on consecutive_opens is what prevents the flapping failure mode where a marginally-healthy primary is repeatedly probed, briefly closes, fails again, and re-opens.

Alerting on breaker behaviour

Tuning is not done at deploy time; the counters tell you whether the parameters still fit the provider’s behaviour weeks later. The Prometheus-flattened series are gis_api_circuit_open_total and gis_api_fallback_total. Alert on the rate of open transitions and on sustained fallback volume, tiered by severity.

# WARNING — breaker is flapping: more than one open transition per minute.
# A healthy breaker opens rarely; frequent opens mean theta or cooldown is too tight.
sum by (endpoint) (rate(gis_api_circuit_open_total[5m])) * 60 > 1

# CRITICAL — fallback tier is serving the majority of geocode traffic for 10m.
# The primary is effectively down; this is an outage, not a transient.
sum by (tier) (rate(gis_api_fallback_total[10m]))
  / sum (rate(gis_api_geocode_requests_total[10m])) > 0.5

# CRITICAL — breaker stuck OPEN: opens but the fallback volume never subsides,
# meaning half-open probes keep failing and the primary never heals.
sum by (endpoint) (increase(gis_api_circuit_open_total[30m])) > 5
  and sum by (endpoint) (rate(gis_api_fallback_total[5m])) > 0

# WARNING — latency-budget trips dominate: primary is slow, not erroring.
# Pair with the CRS/fidelity checks so a slow-but-correct primary is preferred
# over a fast-but-wrong mirror.
sum by (endpoint) (rate(gis_api_circuit_open_total{reason="latency"}[15m])) > 0

A flapping breaker (first rule) almost always means θ\theta is too low or the cooldown too short for the provider’s jitter — raise the floor nmin⁡n_{\min} or lengthen the base cooldown. A breaker stuck open (third rule) is the geocoding analogue of the stale-probe failure the parent guide documents: the half-open trial keeps failing either because the primary is genuinely down or because the probe payload drifted from the provider’s current schema. When the fourth rule fires, the primary is slow rather than erroring, and the tuning question becomes whether the latency budget is realistic — cross-check against the coordinate reference system validation rules so you never let a fast mirror serving the wrong CRS look preferable to a slow but correct primary. Route these alerts through the trust-boundary escalation defined in defining spatial data trust boundaries, since a geocoder feeding an authoritative address layer has a lower tolerance for fallback than one feeding a best-effort enrichment.

FAQ

Should the breaker count slow responses as failures?

Yes — for a geocoder, latency is a failure mode in its own right. A provider that answers every request in four seconds never returns a 5xx but is unusable for an interactive pipeline, and a breaker that only watches error codes stays closed through it. Include a p95 latency-budget term in the trip condition, set to roughly 1.5× the provider’s healthy p95, so sustained slowness trips the breaker just as an error surge would.

How do I stop the breaker from flapping?

Flapping comes from a trip threshold that is too sensitive relative to recovery. Widen the rolling window, raise the minimum-volume floor so low-traffic noise cannot trip it, and apply exponential backoff to the cooldown keyed on consecutive opens. Requiring multiple half-open successes before fully closing also damps flapping, because one lucky probe no longer declares the primary healthy.

What half-open concurrency is safe?

Admit only a few trial probes — two or three — and cap concurrency so a still-sick primary is not hit with a burst of real traffic the moment the cooldown elapses. The half-open state is a probe, not a reopening of the floodgates; full traffic returns only after the required number of trials succeed. If your traffic is high-volume, keep the trial count absolute rather than proportional so recovery testing stays gentle.

Should each geocoding provider have its own breaker instance?

Yes. A shared breaker across providers conflates independent failure domains, so one vendor’s outage would shed traffic away from a healthy one. Instantiate a breaker per endpoint with parameters tuned to that provider’s healthy latency and typical incident length, and key gis.api.circuit_open_total on the endpoint so you can see which provider is unstable.

How does breaker tuning interact with geometric fidelity?

They are orthogonal and must both hold. The breaker decides whether to fall back on availability grounds; the spatial contract decides whether the tier it falls back to is correct. A breaker can close cleanly on availability while the mirror it prefers returns the wrong CRS or simplified geometry, so pair every breaker with the CRS and bounding-box assertions on each hop, and prefer a slow correct primary over a fast wrong mirror when the latency budget forces the choice.