OTel Tail-Sampling Policy for Topology Spans

A topology-validation error is the rarest and most valuable span a geospatial pipeline emits, and a naive probabilistic sampler will discard almost all of them while faithfully retaining thousands of identical successful tile renders. This page designs a tail_sampling policy for the contrib OpenTelemetry collector that keeps every topology-error trace with certainty, down-samples the heavy high-volume transform spans that would otherwise dominate storage, and tunes decision_wait so the sampler never rules on a trace before a re-sharding spatial join has finished emitting its children. It builds on OpenTelemetry integration for GIS pipelines within the geospatial observability architecture, for the data engineers and SREs who own trace fidelity.

Tail-sampling decision_wait buffer and OR policy evaluation Incoming spans of one trace, a root plus join children, arrive over time and are held in the decision_wait buffer until the trace is complete. The tail_sampling processor then evaluates four policies with OR semantics: a topology-error keep, a status equals error keep, a latency over 1200 milliseconds keep, and a probabilistic five percent. If any policy matches the trace is kept; if none match it is dropped. incoming spans root + join children decision_wait 30s hold whole trace tail_sampling policies (OR) 1 · topology error → keep 2 · status = ERROR → keep 3 · latency > 1200ms → keep 4 · probabilistic 5% any match wins KEEP any policy true DROP no match

Why tail sampling, not head sampling

Head sampling decides a trace’s fate at its root span, before any child span exists — which means it decides before the topology validator has run. For a pipeline whose most important signal is a validity failure discovered mid-trace, that ordering is fatal: the sampler has already committed to keep or drop by the time the error appears, so the keep decision cannot depend on it. Tail sampling inverts the timing. The collector buffers all spans of a trace, waits until the trace is complete, and only then evaluates policies against the whole trace — including the child span that carries spatial.validation=failed. That is the only place a “keep every topology error” guarantee can actually be enforced.

The cost of that guarantee is memory and latency: the collector holds every in-flight trace for the length of decision_wait, and it must hold enough of them to cover the peak trace rate. Those two knobs — how long to wait and how many traces to buffer — are what the rest of this page tunes, alongside the policy list itself. The attribute that the keep policy keys on, spatial.validation, is stamped at the point of work as shown in the parent OpenTelemetry integration for GIS pipelines, and the metric families the trace correlates with come from the geospatial metric taxonomy for ETL.

Policy anatomy and OR semantics

The single fact that governs every tail-sampling design is this: the contrib processor evaluates all policies and keeps the trace if any policy returns sampled. Policies are OR-combined, not sequential filters. That is exactly the property that makes the pattern work — a deterministic keep policy for topology errors and a probabilistic policy for the rest coexist without conflict, because the error policy keeps the rare failures and the probabilistic policy independently keeps a fraction of everything else.

A common misconception is that a later probabilistic policy can “sample away” a trace an earlier policy kept. It cannot: OR means a single sampled verdict is decisive regardless of what the other policies say. Ordering does still matter, but for two narrower reasons. First, the and and composite policy types combine sub-policies whose order changes their meaning. Second, the processor emits a decision cache and debug telemetry in policy order, so putting the cheap, deterministic keeps first makes the decision log readable. For a plain OR of independent policies, correctness does not depend on order — but clarity does.

Which stage of the pipeline is even allowed to raise a topology-error span is set upstream by the spatial data trust boundaries; the sampler’s job is only to never lose the spans those boundaries produce.

The full policy, annotated

The configuration below keeps three classes of trace unconditionally — topology-operation spans, any error-status trace, and the slow transforms worth investigating — then probabilistically down-samples the high-volume success path. Because the first three are OR-combined with the fourth, the topology errors are retained with certainty while roughly 95% of successful tile renders are discarded:

# otel-collector-config.yaml  (contrib build — tail_sampling processor)
processors:
  tail_sampling:
    decision_wait: 30s                 # must exceed the slowest trace (tile-pyramid build)
    num_traces: 200000                 # in-flight buffer; size to peak trace rate x decision_wait
    expected_new_traces_per_sec: 4000  # pre-sizes the internal map to avoid rehash churn
    policies:
      # 1. Deterministic keep: any span from the topology validator.
      - name: keep-topology-ops
        type: string_attribute
        string_attribute:
          key: spatial.operation
          values: [validate_topology]

      # 2. Deterministic keep: any trace whose validator marked a span failed/errored.
      - name: keep-error-status
        type: status_code
        status_code:
          status_codes: [ERROR]

      # 3. Deterministic keep: heavy transforms worth a full trace for debugging.
      - name: keep-slow-transforms
        type: latency
        latency:
          threshold_ms: 1200

      # 4. Probabilistic: 5% of everything else (valid reprojections, tile renders).
      - name: sample-success-path
        type: probabilistic
        probabilistic:
          sampling_percentage: 5

Policies 1 and 2 overlap deliberately: spatial.operation=validate_topology catches the validator’s spans even when the pipeline records a soft failure without setting the OTel status to ERROR, and the status_code policy catches genuine exceptions anywhere in the trace. Belt and braces is correct here, because the whole design exists so a topology error is never the trace you sampled away.

decision_wait and late child spans

decision_wait is the interval the collector holds a trace after first seeing it, waiting for stragglers before ruling. Its floor is the duration of your slowest trace. Spatial pipelines have unusually long traces — a tile-pyramid build or a large spatial join fans one input feature into many work units, and the child spans can arrive seconds after the root. If decision_wait is shorter than that span, the processor decides while children are still in flight, evaluates an incomplete trace, and can drop it before the failed-validation child ever arrives. The guarantee silently breaks.

Size the trace buffer against the same window. The number of in-flight traces the collector must hold is at least the arrival rate times the wait:

Nbuffer≥λtrace×tdecision_waitN_{\text{buffer}} \ge \lambda_{\text{trace}} \times t_{\text{decision\_wait}}

At 4,000 new traces per second and a 30-second wait, that is 120,000 concurrent traces, so num_traces: 200000 leaves headroom for bursts. Undersize it and the processor evicts the oldest traces early — the same too-early decision, reached by a different route. The contrib collector exposes this directly: otelcol_processor_tail_sampling_sampling_trace_dropped_too_early counts traces evicted before decision_wait elapsed, and any non-zero rate means the buffer or the wait is too small.

One ordering constraint is inherited from the collector topology: tail sampling runs on already-received spans, so it cannot recover spans the memory_limiter shed under a raster-tiling burst. The keep guarantee is only as strong as the weakest upstream stage — size limit_mib against the worst-case tiling burst, a point the sibling guide on configuring spatial metric collection in Kubernetes covers when it sets the collector’s resource envelope.

Verifying the keep guarantee

Do not trust the policy until you have tried to defeat it. Inject a feature with a known self-intersection so the validator emits a real validate_topology error span, flood the same window with a high volume of successful tile-render traces, and confirm the error trace survives end to end while the success path is thinned to roughly the probabilistic rate:

# The error trace must always survive; the success path should thin to ~5%.
sum(rate(otelcol_processor_tail_sampling_count_traces_sampled{
  sampled="true", policy="keep-topology-ops"}[5m]))

# Any non-zero value here means decision_wait is too short OR num_traces too small.
sum(rate(otelcol_processor_tail_sampling_sampling_trace_dropped_too_early[5m]))

A passing run is three conditions together: the injected topology error appears in the backend as a complete trace with its validator child span, the too-early-drop counter stays flat at zero, and the success-path sample rate lands near 5%. If the error trace is missing but the drop counter is zero, the keep policy is not matching — check that the validator actually stamps spatial.operation=validate_topology. If the drop counter is rising, lengthen decision_wait or raise num_traces before anything else. The full topology-rule semantics behind what counts as a validation error live in the geometry validity and topology checks reference.

FAQ

Does policy order change which traces are kept?

For a plain list of independent policies, no — tail_sampling OR-combines them, so a trace is kept if any policy returns sampled, regardless of order. A later probabilistic policy cannot undo an earlier keep. Order matters only inside and/composite policies, whose sub-policy sequence changes their meaning, and for decision-log readability. Put deterministic keeps first for clarity, not correctness.

How long should decision_wait be for a tile-pyramid build?

Longer than the slowest complete trace. Tile-pyramid builds and large spatial joins emit child spans seconds after the root, so a 5-second wait that suits an HTTP service will decide before those children arrive and can drop the trace mid-flight. Measure your p99 trace duration and set decision_wait above it — 30 seconds is a common floor for spatial pipelines. Watch sampling_trace_dropped_too_early to confirm the wait is sufficient.

Why are my topology errors still missing after adding a keep policy?

Three usual causes. The producing stage never stamps the attribute the policy keys on, so spatial.operation=validate_topology matches nothing — verify the span attribute exists. Or decision_wait is shorter than the trace, so the processor rules before the error child arrives. Or the memory_limiter shed the span upstream of tail sampling, which cannot recover an already-dropped span. Check the too-early and span-drop counters to distinguish them.

Head sampling or tail sampling for topology spans?

Tail sampling, without exception. Head sampling commits at the root span, before the validator has run, so its keep decision cannot depend on a topology error discovered mid-trace. Only tail sampling buffers the whole trace and evaluates policies after the error span exists. Head sampling is fine for uniformly cheap, uniformly interesting traffic — spatial validation is neither.

How do I size num_traces?

Multiply your peak new-trace arrival rate by decision_wait and add burst headroom. At 4,000 traces per second and a 30-second wait, that is 120,000 concurrent traces, so a num_traces of 200,000 is comfortable. Undersizing it evicts the oldest in-flight traces before their decision window elapses, producing the same early-drop failure as too-short a wait. Confirm with the too-early-drop counter under load.