Geospatial Metric Taxonomy for ETL

A geospatial metric taxonomy is the shared vocabulary that lets every stage of a spatial ETL pipeline emit comparable, machine-routable signals about geometry health, coordinate integrity, volume, and freshness. Without a canonical namespace, each ingestion job invents its own counter names, thresholds drift apart, and a self-intersecting polygon or a silently re-projected feed reaches a published tile layer before any dashboard reacts. This page is for the data engineers, GIS platform administrators, and SREs who instrument those pipelines. It sits under Geospatial Observability Architecture & Fundamentals and defines the gis.spatial.* and gis.etl.* metric families, their units and dimensions, the composite scoring that rolls them into an SLO, and the alerting and troubleshooting logic that turns the taxonomy into operational guardrails.

The diagram traces a feature from heterogeneous spatial sources through ingest, reprojection, topology validation, spatial indexing, and publication. Each stage emits a metric drawn from the taxonomy to a contrib-build OpenTelemetry collector, which samples and forwards to the observability backend where composite scores and alert rules live.

Architecture

Geospatial ETL pipelines operate across distributed compute clusters, cloud storage tiers, and heterogeneous coordinate reference systems, so the taxonomy has to capture both infrastructure state and geometric integrity in one envelope. The foundation is a clean separation between metric emission and the data plane: lightweight telemetry sidecars run alongside the spatial operators (GDAL/OGR workers, PostGIS transform pods) and route structured counters, histograms, and traces to a centralized sink over asynchronous batched gRPC, so that backpressure on the observability path never stalls feature throughput.

Where each metric is allowed to originate is governed by the spatial data trust boundaries that segment the pipeline into integrity zones. Trust boundaries dictate metric authority: raw WKB ingestion emits baseline geometry counts and SRID validation flags, while post-join stages own topology preservation ratios and ring-orientation compliance. The same boundaries decide which features get full validation versus sampled checks, a decision formalized by the observability scoping rules for vector data — point, line, and polygon datasets carry distinct baselines, so linework pipelines track segment-snapping tolerance violations while polygon pipelines monitor hole-containment ratios. The transport and attribute conventions below are shared with the broader OpenTelemetry integration for GIS pipelines, so a metric named once here is recognizable everywhere downstream.

In distributed deployments, regional edge caches, replicated tile servers, and cross-availability-zone replication introduce latency asymmetries that distort spatial freshness signals. To keep metrics comparable across regions, the telemetry envelope standardizes four spatial dimensions on every emission: spatial.srid, spatial.geom_type, spatial.bbox (as a serialized extent), and pipeline.stage. Those tags are what let an SRE correlate an infrastructure latency spike with a specific geometry class in a specific zone rather than chasing an unlabeled aggregate.

Metric Specification

The taxonomy extends beyond row-count and latency into spatial state transitions, organized into four operational dimensions — structural, volumetric, temporal, and quality. Every metric carries a canonical name under the gis.spatial.* or gis.etl.* namespace, an OpenTelemetry instrument type, a unit, and production-ready warning and critical thresholds.

To roll these into a single SLO signal, each dimension contributes a normalized breach term to a Spatial Integrity Index $Q \in [0, 1]$, where $1$ is fully conformant. For dimension metrics $m_i$ with critical thresholds $\tau_i$ and weights $w_i$ (with $\sum w_i = 1$):

$$Q = 1 - \sum_{i} w_i \cdot \min\!\left(1, \frac{m_i}{\tau_i}\right)$$

Weighting structural and quality terms above volumetric noise (for example $w_{\text{topology}} = 0.35$, $w_{\text{crs}} = 0.25$, $w_{\text{precision}} = 0.2$, $w_{\text{volume}} = 0.2$) keeps the index sensitive to silent corruption — a CRS cascade drives $Q$ down sharply even while row counts look healthy. The index is the value that an SLO target (e.g. $Q \geq 0.97$ over a rolling window) is written against.

Pipeline Integration & Configuration

Integrating the taxonomy requires explicit OpenTelemetry instrumentation with deterministic, spatially-tagged routing. The snippet below attaches the canonical spatial dimensions to a counter using the Python SDK — note that topology failures are tracked as a monotonic counter, not a ratio gauge, so the rate is computed in the backend where the denominator (total features) is also recorded:

from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.exporter.prometheus import PrometheusMetricReader

reader = PrometheusMetricReader()
provider = MeterProvider(metric_readers=[reader])
metrics.set_meter_provider(provider)
meter = metrics.get_meter("gis.etl")

# Monotonic counter — the post-join validation stage owns this metric.
invalid_geom_counter = meter.create_counter(
    "gis.spatial.topology_invalid_total",
    description="Count of invalid geometries detected per batch",
    unit="1",
)

def emit_topology_metrics(batch_id: str, srid: int, error_count: int, total_features: int):
    invalid_geom_counter.add(
        error_count,
        attributes={
            "batch.id": batch_id,
            "spatial.srid": str(srid),          # canonical dimension — never drop
            "spatial.geom_type": "polygon",     # distinct baselines per geometry class
            "pipeline.stage": "post_join_validation",
        },
    )

Structural and quality counts should be sourced from the database boundary itself, not re-derived in application code, so they reflect what PostGIS actually stored. A batch validation query that feeds the same metric keys looks like this:

-- Emit gis.spatial.topology_error_rate inputs straight from PostGIS.
SELECT
  ST_SRID(geom)                                        AS srid,
  GeometryType(geom)                                   AS geom_type,
  count(*)                                              AS feature_total,
  count(*) FILTER (WHERE NOT ST_IsValid(geom))         AS invalid_total,
  count(*) FILTER (WHERE geom IS NULL OR ST_IsEmpty(geom)) AS null_geom_total,
  ST_Extent(geom)                                      AS batch_bbox
FROM staging.parcel_ingest
WHERE batch_id = %(batch_id)s
GROUP BY ST_SRID(geom), GeometryType(geom);

Health gates must run deterministically before downstream consumers read indexed tiles. The companion guide on setting up spatial pipeline health checks in Airflow shows how a DAG pauses or reroutes when a spatial quality gate fails. A PythonSensor validates index freshness and topology thresholds against the taxonomy before triggering tile generation:

# airflow_dag_spatial_gate.yaml — documentation shape; wire the sensor via a Python DAG.
spatial_quality_gate:
  task_type: PythonSensor
  task_id: validate_spatial_integrity
  timeout: 1800
  mode: poke
  retries: 2
  retry_delay: 120
  op_kwargs:
    prometheus_url: "http://prometheus:9090/api/v1/query"
    # Block tile publish while topology error rate is above 5% for this pipeline.
    query: "gis_spatial_topology_error_rate{pipeline='etl_national_boundaries'} > 0.05"
    fail_on_match: true

Threshold Design & Alerting Logic

Thresholds must be calibrated to the spatial resolution and use-case of each dataset — high-precision cadastral pipelines need far tighter bounds than continental rasterized vector layers, so the taxonomy ships defaults but expects per-pipeline overrides. Alerting is tiered by severity and routed against the metric families directly:

1. CRITICAL (page): gis.spatial.crs_mismatch_count over its critical threshold, or gis.etl.source_sync_delta_hours > 6.0 — broken ingestion or a stalled upstream feed.
2. WARNING (ticket): gis.spatial.topology_error_rate > 5% or gis.spatial.index_build_lag_ms p95 > 5000 — transformation-worker or index-rebuild review.
3. DYNAMIC_BASELINE (dashboard): gis.spatial.feature_density_variance > 3.0 or gis.spatial.bbox_drift_degrees > 0.001° — data skew or an inefficient spatial join predicate; thresholds track a rolling baseline rather than a fixed constant.

Pre-aggregate the high-cardinality spatial counters with Prometheus recording rules so incident-time queries stay cheap, then alert on the recorded series:

# prometheus-rules.yaml
groups:
  - name: gis_spatial_recording
    rules:
      - record: gis_spatial:topology_error_rate:ratio5m
        expr: >
          sum(rate(gis_spatial_topology_invalid_total[5m])) by (pipeline)
          / sum(rate(gis_spatial_feature_total[5m])) by (pipeline)

  - name: gis_spatial_alerts
    rules:
      - alert: SpatialTopologyDegradation
        expr: gis_spatial:topology_error_rate:ratio5m > 0.05
        for: 10m
        labels:
          severity: critical
          team: gis-platform
        annotations:
          summary: "Topology error rate above 5% for {{ $labels.pipeline }}"
          description: "Check CRS alignment and snapping tolerance in the transform worker logs."

When a primary spatial API degrades and the pipeline routes to a cached geometry store, the alerting layer must keep watching the fallback chains for spatial API failures — fallback payloads have to carry identical SRID and precision metadata, or a silent quality regression hides behind a green dashboard.

Failure Modes & Edge Cases

The taxonomy earns its keep in the cases where one signal masks another. Watch for these concrete patterns:

• CRS mismatch masking freshness lag. A feed silently shipping EPSG:3857 where 4326 was expected passes row-count and freshness checks while gis.spatial.coordinate_precision_loss_meters quietly climbs. Diagnose by asserting ST_SRID(geom) against the boundary’s declared authority before trusting any temporal metric.
• Topology errors bypassing validation under sampling. Aggressive tail-sampling in the collector can drop the very spans carrying gis.spatial.self_intersection_count, so the error rate reads artificially low. Pin a 100% keep policy on spans tagged spatial.validation=failed in the spatial_sampling processor.
• bbox drift hidden by aggregation. Averaging gis.spatial.bbox_drift_degrees across a continent erases a single region that buffered geometries into the ocean. Always alert on the per-region max, not the mean.
• Index lag misattributed to CPU. A high gis.spatial.transform_queue_backlog with flat CPU usually means thread contention in the projection library, not load — confirm before scaling out, or you add workers that all block on the same lock.
• Null-geometry inflation from schema drift. A renamed source column inflates gis.spatial.null_geom_ratio while geometry stays valid; pin the schema contract at the trust boundary and reject unknown-field payloads rather than coercing nulls.

Troubleshooting Checklist

When taxonomy metrics drift from the pipeline’s actual state, isolate whether the lag is in collection, aggregation, or spatial processing — in order:

1. Verify OTel exporter flush intervals. Default batch processors buffer counters for 5–10 s. Set OTEL_BSP_MAX_EXPORT_BATCH_SIZE=50 and OTEL_BSP_SCHEDULE_DELAY=2000 for near-real-time spatial visibility.
2. Validate cross-region sync. In federated deployments, cross-AZ replication queues delay propagation; ensure Prometheus federation endpoints scrape regional aggregators with honor_timestamps: true so emission times survive, a pattern detailed in monitoring topology for multi-region GIS.
3. Profile CRS conversion bottlenecks. High gis.spatial.transform_queue_backlog with stable CPU points to lock contention in projection libraries. Profile GDAL/OGR workers with perf record -g, and disable PROJ_NETWORK in air-gapped environments to stop HTTP-timeout stalls.
4. Confirm sampling keeps error spans. Inspect the collector spatial_sampling policy and verify failed-validation spans are exempt from tail-sampling before trusting any low error rate.
5. Audit attribute cardinality. Misconfigured filter processors that drop spatial.srid or spatial.geom_type cause aggregation collisions and artificial latency spikes — assert the canonical dimensions survive the collector.
6. Correlate index lag with the database. Cross-check gis.spatial.index_build_lag_ms against pg_stat_activity to confirm index creation isn’t blocked by a long-running transaction snapshot.

Related

• Geospatial Observability Architecture & Fundamentals — the parent guide to instrumenting spatial pipelines end to end.
• Defining Spatial Data Trust Boundaries — the integrity zones that grant each metric its authority.
• Observability Scoping Rules for Vector Data — per-geometry baselines that set the taxonomy’s thresholds.
• OpenTelemetry Integration for GIS Pipelines — the collector and transport conventions these metrics ride on.
• Setting up Spatial Pipeline Health Checks in Airflow — gating tile publication on the metrics defined here.