Sampling in Distributed Tracing: Head and Tail Sampling, Routing a Trace to One Collector, and Skewed Metrics

Teams that turn on distributed tracing tend to hit the same wall within a few weeks: keeping every span of every request produces a storage bill larger than the application itself. Most of that traffic is fast, successful and indistinguishable from the request before it. Sampling is unavoidable. The real question is where the decision is made and what it gets to look at, because a decision made in the wrong place either throws away the trace you needed or cuts traces in half.

Two places to decide

Head sampling decides at the very start of a trace, when the root span is created. The decision travels downstream in the sampled flag of the W3C traceparent header and every service follows its parent. It is nearly free: unsampled spans are never exported, so they cost nothing on the network or in the collector. The weakness is that nobody knows at decision time how the request will end. With a 10 percent ratio, roughly nine out of ten traces of a rare error that happens fifty times a day are simply never recorded.

Tail sampling decides after the whole trace has been collected. The collector holds a trace's spans in memory for a while, then applies rules: keep traces with errors, keep traces above a latency threshold, keep traces that touched a specific endpoint, and keep a small fraction of everything else. The interesting outliers are never lost. The price is that every span has to leave the application and reach the collector, sit in memory there, and every part of the same trace has to land on the same collector instance.

Taking a side: for any system with more than one service and moderate traffic, tail sampling should be the default. Almost all of the value in trace data sits in the exceptions, and head sampling discards exceptions at exactly the same rate as everything else. Head sampling earns its place at very high volume, where even shipping spans to a collector is expensive, and as a coarse first filter in front of tail sampling.

Head sampling: keep it consistent

OpenTelemetry SDKs pick the sampler from environment variables:

OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1

The parentbased_ prefix matters. This sampler applies the ratio only to root spans and simply follows the incoming decision when there is a parent context. A service using plain traceidratio makes its own decision even if the caller sampled the trace. Because the ratio is derived deterministically from the trace ID, services agree as long as they share the same ratio, but the moment one service is configured differently some traces break in the middle and the UI shows fragments with missing parents.

The default is parentbased_always_on, so with no configuration everything is sampled. If you plan to tail sample, that is the right starting point: keep everything in the application and decide in the collector.

Tail sampling with the OpenTelemetry Collector

The tail_sampling processor in the Collector's contrib distribution does the work. Policies are evaluated and the trace is kept if any of them samples it:

receivers:
  otlp:
    protocols:
      grpc:

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  tail_sampling:
    decision_wait: 10s
    num_traces: 100000
    expected_new_traces_per_sec: 1000
    policies:
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: slow
        type: latency
        latency:
          threshold_ms: 1500
      - name: baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 5
  batch: {}

exporters:
  otlp:
    endpoint: trace-backend:4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlp]

decision_wait is how long the processor waits after the first span before deciding. It has to be longer than your slowest normal request, otherwise half the trace has not arrived when the decision is made. num_traces is how many traces are held in memory at once and directly drives memory use. Roughly, new traces per second times the wait is what sits in memory; leave two to three times that as headroom. Put memory_limiter first in the pipeline so that a spike gets data refused instead of the collector being killed for running out of memory.

Same trace, same collector

This is the requirement people skip most often. Scale the collector horizontally behind an ordinary load balancer and the spans of one trace get spread across instances. Each instance sees only part of the trace: the one holding the failing span keeps its fragment, the others think the trace is boring and drop theirs. You end up with half traces.

The fix is two tiers. The first tier receives data from applications and uses the loadbalancing exporter to route by trace ID to a second tier, which does the tail sampling:

receivers:
  otlp:
    protocols:
      grpc:

connectors:
  spanmetrics: {}

exporters:
  loadbalancing:
    routing_key: traceID
    protocol:
      otlp:
        tls:
          insecure: true
    resolver:
      dns:
        hostname: sampling-collector-headless.observability.svc.cluster.local
  otlphttp/metrics:
    endpoint: http://metrics-backend:4318

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [spanmetrics, loadbalancing]
    metrics:
      receivers: [spanmetrics]
      exporters: [otlphttp/metrics]

The DNS resolver needs a name that returns the address of every second tier instance; in Kubernetes that is a headless service. A regular service name resolves to a single virtual IP and you are back to random distribution. There is also a k8s resolver that watches endpoints through the API.

Produce metrics before sampling

In the configuration above the spanmetrics connector sits in the first tier, before any sampling. That is deliberate. If you derive request counts, error rates and latency distributions from spans and do it on sampled data, your dashboards will show an error rate many times higher than reality, because the kept traces are heavily weighted toward errors and slow requests. Request volume will also drop to about five percent.

With head sampling the problem goes deeper: unsampled spans never reach the collector at all, so any metric derived from spans is wrong from the start. In that case take request metrics from the SDK's metrics instrumentation, not from traces.

Pitfalls

Errors are whatever the instrumentation says they are. The status_code policy looks at span status, not the HTTP code. Under the semantic conventions, 4xx responses on server spans are not errors and the status stays unset. Exceptions that are caught and turned into a friendly response often do not mark the span either. If a failure path matters to you, set the span status explicitly in code, or write a separate policy on the attribute that identifies it.

Late spans. A span that arrives after the decision is treated as a new trace if the collector no longer remembers the decision, and usually gets dropped. Long background jobs and messages consumed late from a queue show up as missing pieces for this reason. Check whether your version has decision cache settings and what their defaults are; otherwise, starting very long jobs as their own trace and connecting them with a span link gives cleaner results.

Rebalancing during scaling. When a second tier instance is added or restarts, the mapping from trace ID to instance changes and traces in flight at that moment are split across two instances. This is not a permanent fault, but an overly twitchy autoscaler turns it into a constant stream of half traces. Keep the second tier at a fixed or slowly changing size.

Adding up the policies. Policies are evaluated independently. Estimate stored volume as the sum of what each policy keeps. A sudden wave of errors multiplies stored volume through the error policy alone. If your backend cannot absorb that, consider a rate_limiting policy as a ceiling next to the error policy.

Forgetting the ratio during analysis. Answering "how many times per hour was this endpoint called" from stored traces is misleading when every trace was kept for a different reason at a different rate. Ask counting questions of metrics and ask traces for the story of a single request.

When not to sample

For a system handling a few thousand requests a day, keeping everything is a trivial cost on most backends and the operational overhead of a sampling setup outweighs the savings. For a single service application, a two tier collector for tail sampling adds more moving parts than it removes. In those cases send everything with parentbased_always_on and keep retention short. Move to sampling when trace storage or collector load becomes a real line item, and on the day you do, verify that your metrics are still produced from unsampled data.