Geospatial Metric Taxonomy for ETL

A geospatial metric taxonomy is the shared vocabulary that lets every stage of a spatial ETL pipeline emit comparable, machine-routable signals about geometry health, coordinate integrity, volume, and freshness. Without a canonical namespace, each ingestion job invents its own counter names, thresholds drift apart, and a self-intersecting polygon or a silently re-projected feed reaches a published tile layer before any dashboard reacts. This page is for the data engineers, GIS platform administrators, and SREs who instrument those pipelines. It sits under Geospatial Observability Architecture & Fundamentals and defines the gis.spatial.* and gis.etl.* metric families, their units and dimensions, the composite scoring that rolls them into an SLO, and the alerting and troubleshooting logic that turns the taxonomy into operational guardrails.

The diagram traces a feature from heterogeneous spatial sources through ingest, reprojection, topology validation, spatial indexing, and publication. Each stage emits a metric drawn from the taxonomy to a contrib-build OpenTelemetry collector, which samples and forwards to the observability backend where composite scores and alert rules live.

Architecture

Geospatial ETL pipelines operate across distributed compute clusters, cloud storage tiers, and heterogeneous coordinate reference systems, so the taxonomy has to capture both infrastructure state and geometric integrity in one envelope. The foundation is a clean separation between metric emission and the data plane: lightweight telemetry sidecars run alongside the spatial operators (GDAL/OGR workers, PostGIS transform pods) and route structured counters, histograms, and traces to a centralized sink over asynchronous batched gRPC, so that backpressure on the observability path never stalls feature throughput.

Where each metric is allowed to originate is governed by the spatial data trust boundaries that segment the pipeline into integrity zones. Trust boundaries dictate metric authority: raw WKB ingestion emits baseline geometry counts and SRID validation flags, while post-join stages own topology preservation ratios and ring-orientation compliance. The same boundaries decide which features get full validation versus sampled checks, a decision formalized by the observability scoping rules for vector data — point, line, and polygon datasets carry distinct baselines, so linework pipelines track segment-snapping tolerance violations while polygon pipelines monitor hole-containment ratios. The transport and attribute conventions below are shared with the broader OpenTelemetry integration for GIS pipelines, so a metric named once here is recognizable everywhere downstream.

In distributed deployments, regional edge caches, replicated tile servers, and cross-availability-zone replication introduce latency asymmetries that distort spatial freshness signals. To keep metrics comparable across regions, the telemetry envelope standardizes four spatial dimensions on every emission: spatial.srid, spatial.geom_type, spatial.bbox (as a serialized extent), and pipeline.stage. Those tags are what let an SRE correlate an infrastructure latency spike with a specific geometry class in a specific zone rather than chasing an unlabeled aggregate.

Metric Specification

The taxonomy extends beyond row-count and latency into spatial state transitions, organized into four operational dimensions — structural, volumetric, temporal, and quality. Every metric carries a canonical name under the gis.spatial.* or gis.etl.* namespace, an OpenTelemetry instrument type, a unit, and production-ready warning and critical thresholds.

To roll these into a single SLO signal, each dimension contributes a normalized breach term to a Spatial Integrity Index $Q \in [0, 1]$, where $1$ is fully conformant. For dimension metrics $m_i$ with critical thresholds $\tau_i$ and weights $w_i$ (with $\sum w_i = 1$):

$$Q = 1 - \sum_{i} w_i \cdot \min\!\left(1, \frac{m_i}{\tau_i}\right)$$

Weighting structural and quality terms above volumetric noise (for example $w_{\text{topology}} = 0.35$, $w_{\text{crs}} = 0.25$, $w_{\text{precision}} = 0.2$, $w_{\text{volume}} = 0.2$) keeps the index sensitive to silent corruption — a CRS cascade drives $Q$ down sharply even while row counts look healthy. The index is the value that an SLO target (e.g. $Q \geq 0.97$ over a rolling window) is written against.

Pipeline Integration & Configuration

Integrating the taxonomy requires explicit OpenTelemetry instrumentation with deterministic, spatially-tagged routing. The snippet below attaches the canonical spatial dimensions to a counter using the Python SDK — note that topology failures are tracked as a monotonic counter, not a ratio gauge, so the rate is computed in the backend where the denominator (total features) is also recorded:

from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.exporter.prometheus import PrometheusMetricReader

reader = PrometheusMetricReader()
provider = MeterProvider(metric_readers=[reader])
metrics.set_meter_provider(provider)
meter = metrics.get_meter("gis.etl")

# Monotonic counter — the post-join validation stage owns this metric.
invalid_geom_counter = meter.create_counter(
    "gis.spatial.topology_invalid_total",
    description="Count of invalid geometries detected per batch",
    unit="1",
)

def emit_topology_metrics(batch_id: str, srid: int, error_count: int, total_features: int):
    invalid_geom_counter.add(
        error_count,
        attributes={
            "batch.id": batch_id,
            "spatial.srid": str(srid),          # canonical dimension — never drop
            "spatial.geom_type": "polygon",     # distinct baselines per geometry class
            "pipeline.stage": "post_join_validation",
        },
    )

Structural and quality counts should be sourced from the database boundary itself, not re-derived in application code, so they reflect what PostGIS actually stored. A batch validation query that feeds the same metric keys looks like this:

-- Emit gis.spatial.topology_error_rate inputs straight from PostGIS.
SELECT
  ST_SRID(geom)                                        AS srid,
  GeometryType(geom)                                   AS geom_type,
  count(*)                                              AS feature_total,
  count(*) FILTER (WHERE NOT ST_IsValid(geom))         AS invalid_total,
  count(*) FILTER (WHERE geom IS NULL OR ST_IsEmpty(geom)) AS null_geom_total,
  ST_Extent(geom)                                      AS batch_bbox
FROM staging.parcel_ingest
WHERE batch_id = %(batch_id)s
GROUP BY ST_SRID(geom), GeometryType(geom);

Health gates must run deterministically before downstream consumers read indexed tiles. The companion guide on setting up spatial pipeline health checks in Airflow shows how a DAG pauses or reroutes when a spatial quality gate fails. A PythonSensor validates index freshness and topology thresholds against the taxonomy before triggering tile generation:

# airflow_dag_spatial_gate.yaml — documentation shape; wire the sensor via a Python DAG.
spatial_quality_gate:
  task_type: PythonSensor
  task_id: validate_spatial_integrity
  timeout: 1800
  mode: poke
  retries: 2
  retry_delay: 120
  op_kwargs:
    prometheus_url: "http://prometheus:9090/api/v1/query"
    # Block tile publish while topology error rate is above 5% for this pipeline.
    query: "gis_spatial_topology_error_rate{pipeline='etl_national_boundaries'} > 0.05"
    fail_on_match: true

Threshold Design & Alerting Logic

Thresholds must be calibrated to the spatial resolution and use-case of each dataset — high-precision cadastral pipelines need far tighter bounds than continental rasterized vector layers, so the taxonomy ships defaults but expects per-pipeline overrides. Alerting is tiered by severity and routed against the metric families directly:

1. CRITICAL (page): gis.spatial.crs_mismatch_count over its critical threshold, or gis.etl.source_sync_delta_hours > 6.0 — broken ingestion or a stalled upstream feed.
2. WARNING (ticket): gis.spatial.topology_error_rate > 5% or gis.spatial.index_build_lag_ms p95 > 5000 — transformation-worker or index-rebuild review.
3. DYNAMIC_BASELINE (dashboard): gis.spatial.feature_density_variance > 3.0 or gis.spatial.bbox_drift_degrees > 0.001° — data skew or an inefficient spatial join predicate; thresholds track a rolling baseline rather than a fixed constant.

Pre-aggregate the high-cardinality spatial counters with Prometheus recording rules so incident-time queries stay cheap, then alert on the recorded series:

# prometheus-rules.yaml
groups:
  - name: gis_spatial_recording
    rules:
      - record: gis_spatial:topology_error_rate:ratio5m
        expr: >
          sum(rate(gis_spatial_topology_invalid_total[5m])) by (pipeline)
          / sum(rate(gis_spatial_feature_total[5m])) by (pipeline)

  - name: gis_spatial_alerts
    rules:
      - alert: SpatialTopologyDegradation
        expr: gis_spatial:topology_error_rate:ratio5m > 0.05
        for: 10m
        labels:
          severity: critical
          team: gis-platform
        annotations:
          summary: "Topology error rate above 5% for {{ $labels.pipeline }}"
          description: "Check CRS alignment and snapping tolerance in the transform worker logs."

When a primary spatial API degrades and the pipeline routes to a cached geometry store, the alerting layer must keep watching the fallback chains for spatial API failures — fallback payloads have to carry identical SRID and precision metadata, or a silent quality regression hides behind a green dashboard.

Failure Modes & Edge Cases

The taxonomy earns its keep in the cases where one signal masks another. Watch for these concrete patterns:

• CRS mismatch masking freshness lag. A feed silently shipping EPSG:3857 where 4326 was expected passes row-count and freshness checks while gis.spatial.coordinate_precision_loss_meters quietly climbs. Diagnose by asserting ST_SRID(geom) against the boundary’s declared authority before trusting any temporal metric.
• Topology errors bypassing validation under sampling. Aggressive tail-sampling in the collector can drop the very spans carrying gis.spatial.self_intersection_count, so the error rate reads artificially low. Pin a 100% keep policy on spans tagged spatial.validation=failed in the spatial_sampling processor.
• bbox drift hidden by aggregation. Averaging gis.spatial.bbox_drift_degrees across a continent erases a single region that buffered geometries into the ocean. Always alert on the per-region max, not the mean.
• Index lag misattributed to CPU. A high gis.spatial.transform_queue_backlog with flat CPU usually means thread contention in the projection library, not load — confirm before scaling out, or you add workers that all block on the same lock.
• Null-geometry inflation from schema drift. A renamed source column inflates gis.spatial.null_geom_ratio while geometry stays valid; pin the schema contract at the trust boundary and reject unknown-field payloads rather than coercing nulls.

Troubleshooting Checklist

When taxonomy metrics drift from the pipeline’s actual state, isolate whether the lag is in collection, aggregation, or spatial processing — in order:

1. Verify OTel exporter flush intervals. Default batch processors buffer counters for 5–10 s. Set OTEL_BSP_MAX_EXPORT_BATCH_SIZE=50 and OTEL_BSP_SCHEDULE_DELAY=2000 for near-real-time spatial visibility.
2. Validate cross-region sync. In federated deployments, cross-AZ replication queues delay propagation; ensure Prometheus federation endpoints scrape regional aggregators with honor_timestamps: true so emission times survive, a pattern detailed in monitoring topology for multi-region GIS.
3. Profile CRS conversion bottlenecks. High gis.spatial.transform_queue_backlog with stable CPU points to lock contention in projection libraries. Profile GDAL/OGR workers with perf record -g, and disable PROJ_NETWORK in air-gapped environments to stop HTTP-timeout stalls.
4. Confirm sampling keeps error spans. Inspect the collector spatial_sampling policy and verify failed-validation spans are exempt from tail-sampling before trusting any low error rate.
5. Audit attribute cardinality. Misconfigured filter processors that drop spatial.srid or spatial.geom_type cause aggregation collisions and artificial latency spikes — assert the canonical dimensions survive the collector.
6. Correlate index lag with the database. Cross-check gis.spatial.index_build_lag_ms against pg_stat_activity to confirm index creation isn’t blocked by a long-running transaction snapshot.

Choosing dimensions that survive contact with production

A metric’s dimensions determine what questions it can answer and what it costs. Spatial platforms consistently get this wrong in the same two directions, and both are avoidable with a short checklist applied before a metric ships.

The first failure is too few dimensions to localise a fault. A validity counter without a source label tells you the platform is rejecting geometry and nothing about which provider changed. A freshness gauge without a layer label reports an average that no consumer experiences. The test is to ask what the first triage question will be — usually “which layer, which source, which region” — and to confirm the metric can answer it without a join to something else.

The second failure is too many dimensions to afford. Every dimension multiplies series count, and unbounded dimensions multiply it without limit. Feature identifiers, raw geohashes, bounding-box coordinates, free-text error strings and user identifiers are all unbounded in practice, and all of them have been attached to a spatial metric by somebody reasoning that it would be useful during an incident. It would be; it is also the fastest way to make the entire metric unusable.

The workable set for most spatial metrics is small and stable: layer always, source where the data comes from outside, region where the work is partitioned geographically, zoom for pyramid work, and a bounded reason on failure counters. Everything else belongs in a log line or a span attribute, where high cardinality is affordable because the storage model is different.

Two refinements are worth adopting. Quantise anything continuous before it becomes a label: a grid cell identifier from a coarse fixed grid is a bounded stand-in for a coordinate, and a bucketed size class is a bounded stand-in for a byte count. And derive label values from a registry rather than from the data, so that a typo in an upstream field cannot create a new series.

Metrics, logs and spans: dividing the work

The taxonomy above covers metrics, but a spatial platform emits three kinds of signal and the boundaries between them are what keep each one affordable.

Metrics answer “how much and how often”, continuously and cheaply. They are always on, they aggregate well, and every alert should derive from them. Their limitation is cardinality: a metric cannot carry the identity of an individual feature without becoming unaffordable.

Spans answer “what happened during this one run”. They carry rich, high-cardinality context — the specific batch, the predicate used, the vertex count of the feature being processed — and they are sampled, which is what makes that affordable. Their limitation is that a sampled signal cannot be the basis for an alert threshold, because the sample may not contain the event.

Logs answer “why exactly did this one thing fail”. They carry the full reason string, the offending identifier, the stack. They are cheap to write and expensive to query at scale, which makes them the right place for detail that is read only when something has already gone wrong.

The rule that follows is worth stating plainly: alert from metrics, diagnose with spans, explain with logs. A platform that alerts on log patterns has an alerting system that fails when log volume spikes. A platform that tries to carry feature-level detail in metrics has a bill that grows with its data. And a platform with no spans has a metrics system that can tell you a stage is slow and never which operation inside it.

Keeping the naming consistent across all three — the same subsystem and quantity segments in a metric name, a span name and a log field — is what lets an engineer move between them during an incident without re-learning the vocabulary each time.

A closing note on ownership: a taxonomy is a shared asset and decays without an owner. Nominate one person or rotation accountable for the namespace — approving new subsystems, reviewing new metric names, and running the periodic audit of exported names against the convention. The role is a few hours a month and it is the difference between a taxonomy and a historical document.

Finally, treat the taxonomy as documentation that consumers read rather than as an internal convention. A short reference page listing every metric, its instrument type, its unit, its dimensions and one sentence on what it captures pays for itself the first time somebody outside the owning team needs to write a query, and it makes the gaps obvious in a way that scattered instrumentation code never does.

Related

• Geospatial Observability Architecture & Fundamentals — the parent guide to instrumenting spatial pipelines end to end.
• Defining Spatial Data Trust Boundaries — the integrity zones that grant each metric its authority.
• Observability Scoping Rules for Vector Data — per-geometry baselines that set the taxonomy’s thresholds.
• OpenTelemetry Integration for GIS Pipelines — the collector and transport conventions these metrics ride on.
• Setting up Spatial Pipeline Health Checks in Airflow — gating tile publication on the metrics defined here.