How to Map Geospatial Data Lineage for Observability

Mapping geospatial data lineage means turning every coordinate reference system (CRS) conversion, topology validation, and feature generalization into a first-class telemetry event, so that when a published tile layer renders garbage you can trace the exact ETL node, transform function, and source dataset version that produced it. Traditional APM stacks see a slow span; they do not see that an EPSG:4326 -> EPSG:3857 reprojection silently shifted vertices, or that a self-intersecting polygon slipped a validation gate. This page is for the data engineers and SREs who own that gap. It sits under Defining Spatial Data Trust Boundaries within Geospatial Observability Architecture & Fundamentals, and gives an execution-ready procedure for instrumenting lineage, anchoring it to trust gates, and querying it back to the originating event to cut mean time to resolution (MTTR) from hours to minutes.

Geospatial lineage instrumentation across the ETL pipeline Spatial sources (WFS, GDAL, PostGIS) enter a left-to-right pipeline of four OpenTelemetry-instrumented ETL stages: ingestion, crs_transform, topology_validate, and generalization. Each stage emits a span carrying geo.* attributes — geo.crs, geo.feature_count, geo.topology_rule, and geo.pipeline_stage — downward into a dedicated 30-day spatial lineage index that is queryable by trace_id back to the originating source URI and dataset version. At the topology_validate trust gate a dashed breach branch routes any batch stamped lineage.breach=true to a quarantine incident queue, halting downstream propagation. Incident queue (quarantine) halts on lineage.breach=true Spatial sources WFS · GDAL · PostGIS ingestion crs_transform topology_validate trust gate generalization breach geo.* span ↓ Lineage index — dedicated 30-day spatial trace store geo.crs · geo.feature_count · geo.topology_rule · geo.pipeline_stage queryable by trace_id → source_uri + dataset version

Problem Framing

Spatial lineage blindness arises because most geospatial pipelines are stitched together from heterogeneous services — a Python reprojection worker, a PostGIS validation job, a tile generalizer — that each log locally and never propagate a shared trace context. The signals that something is wrong appear downstream: a rendering artifact in a web map, a query timeout on a fragmented GiST index, a freshness SLA breach. By then the corrupting transformation is many hops away, and without lineage you are reduced to bisecting deploy logs by hand.

The stage most often implicated is the crs_transform boundary. A reprojection that loses precision does not throw — it produces valid-looking geometry that is subtly wrong, so the failure surfaces only when a spatial join or distance calculation returns nonsense. The remedy is to treat the four canonical pipeline stages — ingestion, crs_transform, topology_validate, and generalization — as instrumented checkpoints that emit a span carrying the spatial context needed to reconstruct provenance. Those attributes are not ad hoc: they draw their names from the Geospatial Metric Taxonomy for ETL, which defines the canonical geo.* / gis.spatial.* namespace every stage must use so that spans from different services remain joinable.

Implementation

Lineage mapping begins at ingestion. Every spatial transformation must emit structured telemetry with mandatory geospatial attributes, and w3c.traceparent headers must be attached to all internal RPCs so a single trace stitches the whole pipeline together. The collector configuration below — built on the OpenTelemetry contrib collector, the same foundation described in OpenTelemetry Integration for GIS Pipelines — disables probabilistic sampling for lineage-critical paths and routes spatial spans to a dedicated index.

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024
  tail_sampling:
    decision_wait: 10s
    num_traces: 10000
    policies:
      - name: spatial_lineage_policy
        type: and
        and:
          policies:
            - name: geo_attribute_filter
              type: string_attribute
              string_attribute:
                key: geo.pipeline_stage
                values: ["ingestion", "crs_transform", "topology_validate", "generalization"]
            - name: force_sample          # never drop a lineage span to sampling
              type: probabilistic
              probabilistic:
                sampling_percentage: 100

exporters:
  otlp/lineage_index:
    endpoint: "telemetry-cluster.internal:4317"
    tls:
      insecure: true
    sending_queue:
      num_consumers: 4
      queue_size: 1000
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s

service:
  pipelines:
    traces/spatial:
      receivers: [otlp]
      processors: [batch, tail_sampling]
      exporters: [otlp/lineage_index]

Enforce four mandatory attributes — geo.crs, geo.feature_count, geo.topology_rule, and geo.pipeline_stage — in your SDK initialization, cap a 30-day retention policy on the spatial index, and limit trace payloads to 64 KB to prevent telemetry bloat.

A lineage graph is only trustworthy when anchored to explicit validation thresholds, so every dataset crossing into the observability stack passes a synchronous gate that verifies coordinate precision, topology, and schema completeness before emitting a lineage event. The decision flow is:

Trust-gate decision flow for emitting a lineage event An incoming feature passes three sequential checks. First, is coordinate round-trip drift within 0.5 metres? If no, route to a lineage breach. Second, is the topology valid? If no, route to a lineage breach. Third, is attribute cardinality at least 99.2 percent of the source schema? If no, route to a lineage breach. The breach node halts propagation, stamps lineage.breach=true, and routes the batch to the incident queue. Only when all three checks pass does the gate emit a lineage event and propagate the feature downstream. Incoming feature Coordinate drift within 0.5 m? Topology valid? Cardinality ≥ 99.2%? Lineage breach halt + incident queue lineage.breach=true Emit lineage event propagate downstream yes yes yes no no no 
# spatial_validation_gate.py
from opentelemetry import trace
from shapely.ops import transform
from pyproj import Transformer

def validate_and_emit_lineage(features, source_crs: str, target_crs: str):
    tracer = trace.get_tracer("spatial.etl")
    transformer_fwd = Transformer.from_crs(source_crs, target_crs, always_xy=True)
    transformer_inv = Transformer.from_crs(target_crs, source_crs, always_xy=True)

    with tracer.start_as_current_span("trust_boundary_validation") as span:
        span.set_attribute("geo.crs", f"{source_crs}->{target_crs}")
        span.set_attribute("geo.pipeline_stage", "topology_validate")

        total_features = len(features)
        topology_failures = 0
        cardinality_loss = 0

        for feat in features:
            original_geom = feat.geometry

            # 1. CRS transform + round-trip drift check (only meaningful on projected source)
            transformed_geom = transform(transformer_fwd.transform, original_geom)
            roundtrip_geom = transform(transformer_inv.transform, transformed_geom)
            drift = original_geom.distance(roundtrip_geom)
            if drift > 0.5:                     # meters
                topology_failures += 1

            # 2. Topology integrity check
            if not transformed_geom.is_valid:
                topology_failures += 1

            # 3. Schema cardinality check
            expected_attrs = 12
            actual_attrs = sum(1 for v in feat.properties.values() if v is not None)
            if expected_attrs > 0 and (actual_attrs / expected_attrs) < 0.992:
                cardinality_loss += 1

        # Threshold enforcement
        error_rate = topology_failures / total_features if total_features > 0 else 0.0
        if error_rate > 0.001 or cardinality_loss > 0:
            span.set_attribute("lineage.breach", True)
            span.set_attribute("breach.reason", "topology_drift_or_cardinality_loss")
            span.set_status(trace.StatusCode.ERROR, "Trust boundary violated")
            raise RuntimeError("Spatial lineage breach detected. Trace routed to incident queue.")

        span.set_attribute("geo.feature_count", total_features)
        return features

The gate enforces three rules: reject features whose round-trip drift exceeds 0.5m, flag topology when the failure rate exceeds 0.1% of features, and alert when attribute cardinality drops below 99.2% of the source schema. On breach it stamps lineage.breach=true on the active trace, halts propagation, and routes the batch to a dedicated incident queue rather than letting a corrupted geometry degrade the spatial index. Halting is the safe default here; where availability must be preserved instead, hand the breached payload to the degradation primitives in Fallback Chains for Spatial API Failures so downstream consumers receive a simplified-geometry response rather than a hard error.

Pair the gate with metric-based alerting so degradation is caught before it reaches consumers:

# spatial-lineage-alerts.yaml
groups:
  - name: spatial_lineage_slos
    rules:
      - alert: HighTopologyFailureRate
        expr: rate(spatial_topology_errors_total{severity="critical"}[5m]) / rate(spatial_features_processed_total[5m]) > 0.001
        for: 2m
        labels:
          severity: critical
          team: gis-platform
        annotations:
          summary: "Spatial topology failure rate exceeds 0.1%"
          description: "Pipeline {{ $labels.pipeline }} is emitting invalid geometries. Check lineage.breach traces."
      - alert: CRSConversionDrift
        expr: histogram_quantile(0.95, rate(spatial_crs_drift_meters_bucket[10m])) > 0.5
        for: 5m
        labels:
          severity: warning
          team: data-engineering
        annotations:
          summary: "p95 coordinate drift exceeds 0.5m threshold"
          description: "CRS transform {{ $labels.source_crs }}->{{ $labels.target_crs }} is introducing spatial drift."

Verification & Testing

Confirm the lineage graph is wired correctly by reconstructing provenance from stored spans before you depend on it during an incident. The first query proves that breach spans are reachable; it targets backends that store OTel spans in SQL-queryable tables (Jaeger on a SQL backend, or ClickHouse):

SELECT
    trace_id,
    span_name,
    SpanAttributes['geo.crs']            AS crs_transition,
    SpanAttributes['geo.feature_count']  AS processed_features,
    SpanAttributes['geo.pipeline_stage'] AS stage,
    SpanAttributes['lineage.breach']     AS breach_detected,
    Timestamp,
    Duration
FROM otel_traces
WHERE SpanAttributes['geo.pipeline_stage'] IN ('crs_transform', 'topology_validate')
  AND SpanAttributes['lineage.breach'] = 'true'
  AND Timestamp > NOW() - INTERVAL 24 HOUR
ORDER BY Timestamp DESC;

To prove the end-to-end graph reconstructs, join spans to dataset version-control metadata so a slow ST_IsValid span resolves to a concrete source URI and version:

SELECT
    t.TraceId,
    t.SpanAttributes['geo.crs']          AS crs,
    d.version                            AS dataset_version,
    d.source_uri,
    t.Duration / 1000000                 AS duration_ms
FROM otel_traces t
JOIN dataset_registry d
  ON t.SpanAttributes['dataset.id'] = d.id
WHERE t.SpanAttributes['geo.topology_rule'] = 'ST_IsValid'
  AND t.Duration / 1000000 > 5000;

Before trusting the gate in production, run a synthetic breach: feed validate_and_emit_lineage a deliberately self-intersecting polygon (a bowtie quad) and a feature stripped of two attributes, then assert that a span with lineage.breach=true appears in the first query within the collector’s decision_wait window (10s). If it does not surface, your tail-sampling policy is dropping it — see the gotchas below.

Gotchas & Failure Modes

  • Adaptive sampling drops the very spans you need. A probabilistic or tail-sampling policy tuned for cost will silently discard error spans, and breaches are rare by definition. The force_sample clause above pins lineage-stage spans to 100%; if you inherit a shared collector, verify the geo_attribute_filter matches before any drop policy in the processor chain, or breach traces vanish exactly when an incident starts.
  • Round-trip drift is meaningless on a geographic source CRS. The 0.5m threshold assumes a projected CRS in meters. If source_crs is EPSG:4326, original_geom.distance() returns degrees, so a real 50 km error reads as ~0.0004 and passes. Gate the drift check on the source SRID’s unit, or compute drift after projecting both geometries to a metric CRS.
  • SRID mismatch passes a vertex-count check. Comparing ST_NPoints before and after a transform confirms no vertices were lost but says nothing about whether they landed in the right place. Always pair cardinality checks with a coordinate-level assertion, and keep geo.crs on every span so cross-region drift stays diagnosable — the same discipline that Monitoring Topology for Multi-Region GIS relies on to keep boundary contracts identical across regions.

Frequently Asked Questions

Where in the pipeline should I emit the lineage span?

Emit one span per canonical stage — ingestion, crs_transform, topology_validate, generalization — and propagate the same traceparent between them so a single trace covers the whole journey. Emitting a single coarse span for the entire job destroys lineage granularity; you lose the ability to pinpoint which transform corrupted the geometry.

How do I keep lineage spans from overwhelming my trace backend?

Route them to a dedicated spatial index with a 30-day retention policy and a 64 KB payload cap, and never serialize full geometries into span attributes — store a dataset.id and geo.feature_count instead, then join to the dataset registry at query time. Pin sampling to 100% only for the lineage stages, not the whole service.

Why use trace context instead of a separate lineage database?

Trace context already carries the causal parent_span_id chain across service boundaries for free, so provenance reconstruction is a join rather than a bespoke graph-write on every hop. A separate lineage store adds a second write path that can diverge from reality; deriving lineage from the same spans that drive your alerts guarantees the two never disagree.

What is a safe drift threshold for the CRS gate?

For projected CRSs in meters, 0.5m round-trip drift is a conservative default that catches precision loss without flagging normal floating-point noise. Tighten it for survey-grade data and loosen it for web-scale generalized layers — but always express it in the source CRS’s linear unit, never in raw degrees.

My breach alert fires but the trace is missing — what happened?

Almost always a sampling policy ordering bug: a drop policy ran before the geo_attribute_filter matched, so the error span was discarded before export. Confirm the spatial filter precedes any probabilistic drop in the collector’s processor chain, then replay the synthetic breach from the Verification section to confirm the span now lands in the index.