Trace Sampling: Head Sampling vs Tail Sampling

Last verified on Nov 8, 2026.

Trace sampling is the practice of keeping only a share of your traces so you can afford to store and search them. OpenTelemetry's position is that if most of your requests succeed with acceptable latency, you do not need every trace to understand your system. You need the right ones. A trace or span is "sampled" when it is processed and exported, and "not sampled" when it is dropped.

Head Sampling

Head sampling decides as early as possible, usually when the root span starts. The decision is not made by looking at the trace as a whole. The most common form is consistent probability sampling: the decision comes from the trace ID and a target percentage, so every span in a trace gets the same answer and you get whole traces at a steady rate such as 5%.

• Good: easy to understand, easy to configure, efficient, and doable anywhere in the pipeline.
• Bad: you cannot guarantee that traces containing an error are kept, because the error has not happened yet.

In an SDK, set the sampler with environment variables. The default is parentbased_always_on, which keeps everything:

export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.1

parentbased_traceidratio samples 10% of new traces and follows the parent's decision when one exists. In a Collector, the probabilistic sampler processor uses a hash of the trace ID.

Tail Sampling

Tail sampling decides after all or most spans of the trace are in, so you can keep traces by what they contain: any error, total latency, or attribute values. OpenTelemetry gives these examples: always keep traces with an error, sample by overall latency, sample more from a newly deployed service, and apply different rates to low- and high-volume services.

This Collector configuration keeps errors and slow traces, then 5% of the rest. It passed otelcol-contrib validate on version 0.165.0:

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    policies:
      - name: keep-errors
        type: status_code
        status_code:
          status_codes:
            - ERROR
      - name: keep-slow
        type: latency
        latency: { threshold_ms: 2000 }
      - name: sample-the-rest
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }

The costs are real. OpenTelemetry names three: tail sampling is hard to implement, hard to operate because it needs a stateful component holding a large amount of data, and often vendor-specific. The processor's README adds that all spans of a trace must reach the same Collector instance, and suggests two layers of Collectors, one with the load balancing exporter and one with the tail sampling processor. The decision_wait setting is how long the processor waits for spans before it decides.

Head vs Tail at a Glance

OpenTelemetry notes that some systems use both, for example head sampling to cut an extremely high-volume service before tail sampling the rest.

The sampled Flag and Broken Traces

W3C Trace Context carries a sampled bit in traceparent. It is a recommendation from the caller, and a callee may make its own choice, including downsampling. That is where broken traces come from: if service A keeps a trace and service B drops its part, you see a trace with a gap. Use parent-based samplers so downstream services follow the decision, and make sure all services use the same rules. Note that a dropped trace's ID still shows up in logs, so a trace ID in a log line does not promise stored spans.

When Not to Sample

OpenTelemetry suggests considering sampling at about 1000 traces per second or more, and avoiding it when you produce very little data, only use data in aggregate, or are bound by rules that forbid dropping data. Weigh the compute and engineering costs against simply storing more.

Sampling and Sazabi

Sazabi, an AI-native observability and security platform, ingests the spans your exporter sends. Sazabi ingests OpenTelemetry metrics (Prometheus metrics through an OpenTelemetry Collector) and OTLP traces through the same intake as your logs. It stores each metric data point and each span as a log record that its agent queries alongside your logs, so there is no separate metrics store to run. Whatever you drop at the SDK or Collector never arrives, so decide with your own latency and error goals in mind. Setup is in OpenTelemetry traces and the Collector guide. Rising trace volume can also drive spend, which cost anomalies is built to catch, and slow outliers are the target of tail latency detection.