Budgeting Collector CPU for Vector Telemetry

An observability agent that competes with the pipeline it observes is a liability. On vector workloads that competition is unusually easy to trigger, because the expensive telemetry arrives in exactly the bursts where the pipeline is already saturated — a bulk load of dense boundary geometries generates the largest spans, the most attribute serialisation and the highest export volume at the precise moment the worker needs its cores for reprojection. This guide covers how to set a defensible CPU and memory budget for a spatial telemetry collector, how to measure whether you are inside it, and what to shed first when you are not. It belongs to observability scoping rules for vector data under geospatial observability architecture fundamentals.

Pipeline and collector CPU demand over a bulk load, showing correlated peaks Two stacked series run across a bulk load window. The pipeline's own CPU demand rises steeply as dense geometries are reprojected. The collector's demand rises at the same time and in proportion, because the same features generate the largest telemetry payloads. A dashed budget line marks fifteen percent of the worker's cores. During the peak the collector exceeds the budget and the combined demand exceeds capacity, marked as the contention window where the collector becomes back-pressure on the pipeline. Collector demand peaks exactly where pipeline demand peaks 100% 0 worker capacity pipeline work + collector contention window the collector is now back-pressure on the load it is observing

Problem framing: budget the peak, not the mean

Average collector utilisation is a comfortable and misleading number. A collector sitting at three percent of a core across the day and thirty percent during the nightly load is over budget where it matters, and averaging hides it completely.

The workload has three properties that force this.

Cost per feature scales with vertex count. Serialising a span carrying a geometry envelope, vertex count and validity result costs roughly linearly in the geometry’s complexity, and vertex counts in boundary and parcel data are heavily right-skewed. A handful of features dominate the collector’s work.

Arrival is bursty and correlated with pipeline load. Telemetry volume is a function of throughput, so the collector’s busiest minute is the pipeline’s busiest minute by construction.

Backpressure propagates the wrong way. When the export buffer fills, a collector configured to block will slow the emitting worker; one configured to drop will shed exactly the telemetry generated during the incident you most want to observe. Neither is acceptable as an unplanned outcome, so the shedding policy has to be chosen deliberately.

A workable budget is therefore expressed at the peak: the collector may use no more than a stated fraction of the worker’s cores and a stated resident memory ceiling during the busiest five minutes of the day, not on average.

Implementation: measure, then bound

Start by measuring what the collector actually costs on a representative load, broken down by pipeline stage so the expensive telemetry is identifiable.

# Collector CPU as a fraction of the worker's cores, at the peak rather than the mean.
max_over_time(
  (
    rate(process_cpu_seconds_total{job="otel-collector"}[1m])
      /
    on (instance) group_left() machine_cpu_cores{job="node"}
  )[24h:1m]
)

# Which spans dominate? Export payload bytes by pipeline stage.
topk(5,
  sum by (stage) (rate(otelcol_exporter_sent_spans_bytes_total[5m]))
)

Then bound it explicitly. Container limits are the blunt instrument that guarantees the pipeline keeps its cores; the collector’s own queue and batch settings are the fine control that determines what happens when the limit binds.

# otel-collector-contrib.yaml — bounded so the collector can never become
# back-pressure on the spatial worker it shares a node with.
processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 256              # hard ceiling; matches the container limit
    spike_limit_mib: 64         # start refusing before the ceiling is hit

  # Complexity-aware sampling: cost per span rises with vertex count, so the
  # keep-probability falls with it. The aggregate histogram is emitted
  # unconditionally elsewhere, so the tail stays observable.
  probabilistic_sampler/high_vertex:
    sampling_percentage: 1
  probabilistic_sampler/medium_vertex:
    sampling_percentage: 10

  batch:
    send_batch_size: 512
    timeout: 5s                 # bounded latency so bursts do not accumulate

exporters:
  otlp:
    endpoint: telemetry-gateway:4317
    sending_queue:
      enabled: true
      queue_size: 2000          # bounded: a full queue drops, never blocks
      num_consumers: 4
    retry_on_failure:
      enabled: true
      max_elapsed_time: 120s    # give up rather than retry into a saturated link

The two settings that matter most are queue_size and the absence of a blocking mode. A bounded queue that drops on overflow keeps the collector’s failure contained; an unbounded or blocking queue converts a telemetry problem into a pipeline problem, which is the outcome the whole budget exists to prevent.

What to shed, in order

When the budget binds, shed in an order that preserves the ability to diagnose. The ordering below reflects what each signal costs and what it would cost you to lose.

Order Shed Why it goes first
1 Per-feature spans for high-vertex geometries Highest cost per unit of information; the aggregate histogram retains the shape
2 Per-feature spans for medium-vertex geometries Same argument, lower cost saving
3 Debug-level attributes on retained spans Large payload, rarely queried
4 Successful-path spans, keeping error spans Errors are what diagnosis needs
5 Aggregate histograms Only under extreme pressure — losing these blinds you entirely

Counters and gauges for correctness signals — projection mismatches, validity failures, freshness age — should never be shed. They are tiny, they are what alerting depends on, and dropping them converts a resource problem into an unmonitored platform.

Shedding ladder ordered by cost saved against diagnostic value lost Five telemetry classes are plotted as bars against two measures. High-vertex per-feature spans save the most cost and lose the least diagnostic value, so they are shed first. Medium-vertex spans follow. Debug attributes on retained spans come third. Successful-path spans come fourth, retaining error spans. Aggregate histograms come last because they lose substantial diagnostic value. A separate marked band shows correctness counters and gauges as never shed, with a note that they are small and are what alerting depends on. Shed by cost-saved per unit of diagnostic value lost high-vertex spans medium-vertex spans debug attributes successful-path spans aggregate histograms cost saved diagnostic value lost never shed · correctness counters and gauges projection mismatches, validity failures, freshness age — tiny, and everything alerting depends on Collector peak CPU as a share of worker cores, by pipeline class Five pipeline classes are shown with the collector's measured peak CPU share. A streaming feed sits lowest because its telemetry rate is steady. A tile publisher sits slightly higher. A nightly batch loader sits near the budget line. A raster warp fleet sits above it. A bulk reprojection job sits far above and is marked as needing harder sampling rather than a raised limit. Peak collector CPU by pipeline class, against a 15% budget streaming feed 3% of cores tile publisher 6% nightly batch loader 14% · at budget raster warp fleet 19% bulk reprojection job 31% · sample harder

Verification

Run a load test at the ninety-fifth percentile of your real ingest rate and confirm three facts: the collector’s peak CPU stays inside its budget, its resident memory stays below the limiter’s ceiling, and the pipeline’s own throughput is unchanged compared with a run where telemetry is disabled entirely. The third is the one that matters — a collector inside its budget that still slows the pipeline is contending for something other than CPU, usually the export link or the same disk.

Then force overflow deliberately. Point the exporter at a black-holed endpoint and confirm the queue fills, drops, and reports the drop — and that the emitting worker’s throughput does not change. A worker that slows when telemetry export fails has a blocking path somewhere, and that is the single most dangerous configuration in the whole stack.

Gotchas

Budgeting on the mean. Hides the only window where contention occurs. Budget the peak.

Unbounded export queue. Converts a downstream telemetry outage into a pipeline outage. Bound it and accept drops.

Sampling applied after attribute construction. If the expensive geometry attributes are built before the sampling decision, the cost is already paid. Decide early, in the producer, as the parent topic’s stratified sampling describes.

No memory limiter. A collector that grows until the container is killed takes its buffered telemetry with it, losing exactly the window of the incident.

Shedding correctness counters under pressure. Cheapest signals, highest value; excluding them from any shedding rule should be explicit rather than assumed.

FAQ

What fraction of a worker’s cores is reasonable?

Around fifteen percent at peak is a defensible starting point for a sidecar collector on a spatial worker, with a hard container limit at twenty. If the collector needs more than that, the answer is almost always to sample harder in the producer rather than to raise the limit, because the cost is dominated by a small number of very large features.

Sidecar or shared node collector?

A sidecar gives clean per-worker budgeting and isolates failure; a shared collector amortises overhead but makes one noisy worker everyone’s problem. For spatial workloads with bursty, complexity-correlated telemetry, the isolation of a sidecar is usually worth the extra overhead — and the budget is far easier to reason about.

How does this interact with tail sampling?

Tail sampling needs to hold spans until a trace completes, so it costs memory in proportion to trace duration and rate. On spatial pipelines with long-running batch traces that can be substantial, which is why the policy design in OTel tail-sampling policy for topology spans belongs in the same budget conversation.

Should the budget differ by pipeline class?

Yes. A streaming feed’s collector runs at a steady low rate and can afford richer per-event telemetry; a nightly bulk loader needs a much tighter peak budget because its burst is enormous and brief. Deriving the budget from the layer’s pipeline class keeps one configuration pattern across both.

How do I set the budget for a worker whose load varies seasonally?

Budget against the seasonal peak rather than the current one, and re-derive it after each peak passes. Spatial platforms frequently have a pronounced annual shape — a cadastral refresh cycle, a growing-season imagery run, a census update — and a budget set during the quiet months will bind exactly when the platform is busiest and least able to absorb contention. Keeping the previous peak’s measurements alongside the current budget makes the annual review a five-minute check rather than a fresh investigation.

What should page when the budget is exceeded?

Nothing overnight. A collector over budget is a capacity finding for the daily queue unless it is actually degrading the pipeline, in which case the pipeline’s own throughput alert is the one that should fire. Paging on telemetry resource usage trains people to ignore telemetry alerts.