Choosing Spatial Observability Tooling

Selecting instrumentation for a spatial stack is a decision teams usually make by inertia — they reach for whatever their web services already use and discover, mid-incident, that a geometry pipeline breaks the assumptions that tooling was built on. High-vertex histograms blow up pull-based cardinality budgets, ephemeral reprojection jobs vanish before a scraper arrives, and index-level faults never surface in an application span at all. This guide is a structured decision framework for choosing spatial observability tooling deliberately: it walks the forks that actually matter for geospatial workloads and hands each fork off to a focused comparison. It sits within spatial incident response and observability tooling, turning that guide’s abstract trade-off axes into a repeatable selection you can defend in a design review.

The tree above is the shape of the decision. Three roughly independent questions — how the workload lives, whether triage needs causal traces, and how the underlying spatial table is written — each resolve to a concrete tool, and their combination is a composite profile rather than a single product choice. The sections that follow give each fork an architecture sketch, a scoring rubric, configuration you can lift, threshold guidance, and the failure modes that show up when the profile is wrong for the workload.

Architecture

A spatial observability stack has four layers, and the tooling choice at each is largely separable from the others, which is what makes the decision tractable. Treating them as one monolithic “which vendor” question is the mistake that leads to inertia selection.

• Collection layer. How telemetry leaves the workload: a scraper pulling a /metrics endpoint, or the workload pushing OTLP to a collector. This is decided almost entirely by workload lifetime, and it is the fork the push versus pull comparison for spatial pipelines resolves.
• Signal layer. What kind of signal you privilege: causal traces that reconstruct one request’s path through a reprojection, or pre-aggregated metrics that count reprojections in bulk. The OpenTelemetry versus Prometheus-native comparison for PostGIS is the deep dive on this fork.
• Storage-engine layer. How the spatial data is indexed determines what index-health signals even exist and how they behave. An append-only GPS-trace table and a randomly-updated parcel table want different index access methods, which is the subject of the GiST versus BRIN index comparison.
• Backend layer. Where signals land and are queried. This is the most fungible layer and the least spatial-specific, so it is deliberately out of scope here — the upstream three choices constrain it far more than it constrains them.

Because the layers are separable, a valid profile might be “push OTLP + Prometheus metrics + GiST”, another “pull scrape + OTel traces + BRIN”. There is no single right stack; there is a right stack for a given workload’s lifetime, triage needs, and write pattern. The composite profile in the diagram is exactly that tuple, and it feeds directly into the detection and triage stages described in the parent guide on spatial incident response and observability tooling.

The independence of the forks is a useful approximation rather than a strict law, and knowing where it leaks prevents two common mistakes. The collection and signal forks couple weakly: choosing traces as your primary signal nudges you toward push, because a trace is naturally emitted by the workload as it executes, whereas forcing traces through a pull model means buffering spans on a scrape endpoint that a short-lived job may never expose. The signal and storage-engine forks are more genuinely independent — index-health gauges are catalog queries that both a trace pipeline and a metric pipeline can carry equally well — which is why storage-engine visibility should never be the tail that wags the collection-model dog. The one hard coupling is between workload lifetime and everything downstream: an ephemeral job constrains the collection model, which constrains how cheaply you can retain its signals, which constrains what triage is even possible after it exits. Resolve lifetime first, and the remaining forks open up; resolve it last, and you will find you have painted the other two into a corner.

There is also a temporal dimension the layered view flattens. The right profile for a workload at launch is often not the right profile a year later, because the workload’s own characteristics drift — a pipeline that begins as a nightly batch and evolves into a streaming service crosses the lifetime boundary that governs the collection fork, and a table that starts append-only and later begins receiving edits crosses the write-pattern boundary that governs the storage-engine fork. Treating the composite profile as a snapshot to be re-derived, not a permanent inscription, is what keeps the instrumentation honest as the thing it observes changes underneath it.

Decision Metrics

Score each candidate against the criteria below rather than arguing about products in the abstract. The weights reflect what actually determines fit for spatial workloads; adjust them to your context, but do not drop the spatial-specific rows, because those are the ones generic evaluations miss.

A simple weighted score keeps the comparison honest. For each candidate the fit is the weighted sum of its per-criterion score $s_{ij} \in \{0, 1, 2\}$:

$$\text{Fit}_j = \sum_{i} w_i \, s_{ij}$$

The point of writing it down is not the arithmetic — it is that a low score on a high-weight, spatial-specific row (ephemeral capture, storage-engine visibility) is exactly the row a team skips when it selects by inertia, and it is the row that produces the mid-incident surprise. When two profiles score within a few points, prefer the one that is simpler to operate during an incident, because the tooling you understand under pressure beats the tooling that scores marginally higher on paper.

A worked example makes the rubric concrete. Suppose a team runs a fleet that is two-thirds nightly reprojection batches and one-third long-lived tile servers, with a small on-call rotation. Score a push-plus-traces profile against a pull-plus-metrics profile. Ephemeral capture (weight high) scores 2 for push and 0 for pull, because the batches dominate and pull loses them. High-cardinality vertex metrics (weight high) scores 1 for both, since neither is free and both need sampling or aggregation. Per-request causality (medium) scores 2 for push, 0 for pull. Storage-engine visibility (medium) scores 1 for both, since each can run the catalog SQL. Operational simplicity (medium) scores 0 for push and 2 for pull, reflecting the small rotation’s limited appetite for a collector fleet. Ingestion cost (medium) scores 1 for push and 2 for pull. With high weighted at 3 and medium at 1, push totals $3\cdot2 + 3\cdot1 + 1\cdot2 + 1\cdot1 + 1\cdot0 + 1\cdot1 = 13$ and pull totals $3\cdot0 + 3\cdot1 + 1\cdot0 + 1\cdot1 + 1\cdot2 + 1\cdot2 = 8$. Push wins decisively — but the two-point operational-simplicity gap is the flag that the team must invest in collector runbooks before that win is real, or the profile that scores higher on paper will be the one nobody can operate at 3 a.m. That tension between capability and operability is the recurring theme of every honest tooling decision, and it is why the score is a starting point for the conversation, not the end of it.

The weights themselves are worth revisiting quarterly. A team’s workload mix shifts — a batch-heavy shop that graduates its pipelines into always-on streaming services will find its ephemeral-capture weight falling and its steady-state-cost weight rising, which can flip the recommended profile without any tooling changing. Re-scoring on a cadence, rather than treating the original decision as permanent, is what keeps the instrumentation matched to the workload it actually observes rather than the workload it observed at inception.

Configuration

The decision reduces to configuration at each layer. Below is the collection-layer wiring for both branches so the trade-off is concrete rather than abstract. For a long-lived tile server, a pull scrape is the natural fit:

# prometheus.yml — pull branch for a long-lived spatial service
scrape_configs:
  - job_name: postgis-tileserver
    scrape_interval: 15s
    static_configs:
      - targets: ["tileserver:9187"]
    metric_relabel_configs:
      # Drop any accidental per-feature label before it hits the TSDB.
      - source_labels: [feature_id]
        action: labeldrop

For an ephemeral reprojection batch, that scrape never lands — the job is gone in seconds. The push branch routes the same class of metric through a collector via OTLP instead:

# otel-collector-config.yaml (contrib) — push branch for ephemeral spatial jobs
receivers:
  otlp:
    protocols:
      grpc:
processors:
  batch/spatial:
    timeout: 5s
  # Bound the reason cardinality that batch failures otherwise generate.
  transform/bound_labels:
    metric_statements:
      - context: datapoint
        statements:
          - keep_keys(attributes, ["layer", "operation", "reason"])
exporters:
  prometheusremotewrite:
    endpoint: "http://tsdb:9090/api/v1/write"
service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch/spatial, transform/bound_labels]
      exporters: [prometheusremotewrite]

At the storage-engine layer, the configuration is the index DDL itself. A quick reference — the full trade-off analysis is in the dedicated comparison — is that an append-only feed takes a BRIN on its geometry-derived sort key, while a randomly-updated table takes a GiST:

-- Append-only GPS trace: BRIN on the naturally-clustered ingest order.
CREATE INDEX idx_trace_brin ON gps_trace USING brin (recorded_at, geom);
-- Randomly-updated parcels: GiST for arbitrary spatial predicates.
CREATE INDEX idx_parcel_gist ON parcels USING gist (geom);

Threshold and Alerting

Tooling choices carry their own operational thresholds — the tooling can itself become the incident. Alert on the health of the observability pipeline, not only on the data it carries, so that a blind spot is a page rather than a silent gap during the next outage.

groups:
  - name: spatial-tooling-health
    rules:
      # Scrape targets vanishing — the pull-branch failure signature.
      - alert: SpatialScrapeTargetDown
        expr: up{job=~"postgis.*"} == 0
        for: 3m
        labels: { severity: warning }
        annotations:
          summary: "Scrape target {{ $labels.instance }} down — metrics going dark"

      # Collector dropping datapoints under load — the push-branch failure signature.
      - alert: SpatialCollectorDropping
        expr: |
          rate(otelcol_processor_dropped_metric_points[5m]) > 0
        for: 5m
        labels: { severity: critical }

      # Cardinality budget breach on a spatial metric — the shared failure signature.
      - alert: SpatialSeriesCardinalityHigh
        expr: |
          count by (__name__)({__name__=~"gis_spatial_.*"}) > 50000
        for: 10m
        labels: { severity: warning }

Set the cardinality ceiling from your TSDB’s headroom, not from a round number, and treat a breach as a design signal that a label needs bounding, exactly as bounding spatial metric tag cardinality prescribes. The scrape-down threshold should sit below the interval at which your detection SLO can still act; a target dark for ten minutes on a five-minute burn window is a detection blind spot, not a minor blip.

Failure Modes

The characteristic failures of a mis-chosen profile are predictable, and naming them is the point of choosing deliberately:

• Ephemeral job blindness. A pull profile applied to batch jobs silently loses every metric the job emitted, because the scrape never coincides with the job’s lifetime. The symptom is a dashboard that looks healthy because it only ever shows the long-lived services.
• Cardinality collapse. A trace-forward profile that attaches per-feature attributes without sampling melts the backend under load — precisely when you need it. The symptom is telemetry latency rising in lockstep with the incident it should be illuminating.
• Index-fault invisibility. Any profile that lacks storage-engine introspection cannot see GiST or BRIN degradation, so “the database got slow” incidents have no root-cause signal and get misattributed to application code.
• Operational overload. A maximally capable profile — every signal type, every layer instrumented — is unusable if the on-call engineer cannot reason about it at 3 a.m. Capability that is not operable under pressure is negative capability.

Rollout and Reversibility

The layered structure that makes the decision tractable also makes it reversible, and planning for reversibility up front is what turns a tooling choice from a bet into an experiment. Because the collection, signal, and storage-engine layers are separable, you can migrate one without disturbing the others. A team that started pull-and-metrics can adopt traces for triage by adding an OpenTelemetry collector alongside the existing scraper, dual-running until the trace-based triage proves itself, and only then deciding whether to retire any pull-side detection. Nothing forces a rip-and-replace, and the dual-run window is where you discover the operational cost the rubric could only estimate.

Sequence the rollout to protect detection, which is the layer an incident cannot tolerate losing. Never cut over the detection path and the triage path in the same change; land the new triage capability while the proven detection alerts keep firing, verify the new signals against a known incident replayed from history, and migrate detection last if at all. The reverse order — swapping detection first and validating triage later — is how a migration itself becomes an outage, because the window where the new detection is unproven is exactly the window an undetected fault can grow. Keep the old profile’s alert rules in place through at least one full incident cycle so you can compare detection lag between the two before committing.

Reversibility also disciplines the storage-engine layer. A table that migrates from a GiST to a BRIN index, or the reverse, should keep both indexes live long enough to compare query plans under real traffic before the old one is dropped, since the fragmentation curves the GiST versus BRIN index comparison describes only reveal themselves under production write patterns. Treating every layer’s choice as a testable, reversible hypothesis rather than a permanent commitment is what keeps a tooling decision from calcifying into the same inertia it was meant to escape.

Troubleshooting Checklist

When the observability stack itself is suspect during an incident, work this list before doubting the data:

1. Verify signals are arriving. Check up for pull targets and otelcol_receiver_accepted_metric_points for push. A green dashboard on a dead feed is the worst failure mode because it reads as health.
2. Check for silent sampling. Confirm that incident-critical metrics (fallback, drift, circuit-breaker counters) are exempt from the sampler. A sampled-away detection signal is invisible exactly when it matters.
3. Audit label cardinality. Run the per-name series count. A recent spike usually traces to a new free-text label value leaking into a reason or source dimension.
4. Confirm storage-engine scraping. Verify the SQL that emits gis.spatial.index_bloat_ratio is actually running; index metrics are the first to be quietly dropped when a config is trimmed.
5. Reconcile with the runbooks. If the tooling checks out and the data is still wrong, move to the matching playbook in the spatial pipeline incident runbooks; the problem is in the pipeline, not the instrumentation.

Related

• GiST vs BRIN indexes for spatial observability — the storage-engine fork: which spatial index access method fits random-access versus append-only data.
• OpenTelemetry vs Prometheus-native instrumentation for PostGIS — the signal fork: traces with spatial span attributes versus pull-based exporters.
• Push vs pull metric collection for spatial pipelines — the collection fork: OTLP push versus scrape for high-cardinality and ephemeral spatial jobs.
• Spatial incident response and observability tooling — the parent guide whose trade-off axes this framework operationalizes.
• Bounding spatial metric tag cardinality — the cardinality discipline every tooling profile must enforce.