Log Sampling: Cut Log Volume Without Losing Errors

Last verified on Nov 8, 2026.

Log sampling means keeping a fraction of log events and dropping the rest, so that volume and cost go down while the data still tells you what happened. Done well, you sample the repetitive healthy traffic and keep every error, warning, and audit event. Done badly, you drop the one line you needed. This guide explains the three approaches, then gives configurations for nginx, application code, and the OpenTelemetry Collector that were each run and counted.

Choose What to Sample

Sampling is a trade of detail for cost. The OpenTelemetry docs frame it as one of the most effective ways to reduce the cost of observability without losing visibility, because a smaller sample can represent a larger population, and at high volume a rate of 1% or lower can represent the rest. Those docs are written for traces, but the reasoning applies to logs.

They also list when not to sample: you produce very little data, you only use the data in aggregate, or regulation prohibits dropping it. So begin by sorting your logs:

See audit logs and security logging for what falls into the never-sample group.

Head Sampling, Tail Sampling, and Rate Limiting

Head sampling decides as early as possible, without inspecting the outcome. The common form is consistent probability sampling: hash an ID and keep the record if the hash falls under the target percentage. It is cheap and easy, and its downside is that it can drop an error because it never looks at one. For traces, the OpenTelemetry docs note that this keeps whole traces rather than fragments. For logs, hash a trace or request ID, so all lines for one request stay or go together.

Tail sampling decides after the outcome is known, for example always keep traces with an error or high latency. The OpenTelemetry tail sampling processor works on traces. For logs there is no whole-trace view at the point of emission, so the practical equivalent is a severity rule: always keep warnings and above, and sample the rest. That is what the examples below do.

Rate limiting caps events per interval regardless of content. It protects a pipeline from a runaway loop but does not preserve representativeness, because during a burst it drops everything over the cap, including the errors. journald does this by default: each service is limited to 10000 messages in 30 seconds, and after that it drops messages until the interval ends and writes a message saying how many it dropped. Treat a rate limit as a safety net on top of sampling, not a replacement, and alert on the "dropped" notices.

Sample in nginx

Access logs for successful requests are usually the largest log stream. nginx's access_log accepts if=, and a request is not logged when the condition is 0 or empty. split_clients hashes a string with MurmurHash2 and assigns a value by percentage of the hash range. Combine them to keep all 4xx and 5xx responses and 10% of the rest:

map $status $is_error {
    ~^[45]  1;
    default 0;
}

split_clients $request_id $in_sample {
    10%  1;
    *    0;
}

map "$is_error$in_sample" $loggable {
    "00"    0;
    default 1;
}

access_log /var/log/nginx/access.log combined if=$loggable;

Run against nginx 1.28 with 1,000 requests returning 200 and 20 returning 500, the log held 108 lines: all 20 of the 500 responses and 88 of the 1,000 successes. That is about 9%, close to the 10% target, and the variation is expected for a hash on a random $request_id. Because $request_id is random per request, this is not consistent per client. To keep or drop by client, hash $remote_addr instead. Lower-effort variants are in the nginx logs guide, including access_log off in a health check location.

Sample in Application Code

Sampling in the application saves the most, because the line is never formatted or written. A logging filter that always passes warnings and above and samples the rest:

import logging, random, sys

class SampleFilter(logging.Filter):
    """Keep every WARNING and above, and every audit record. Sample the rest."""
    def __init__(self, rate):
        super().__init__()
        self.rate = rate

    def filter(self, record):
        if record.levelno >= logging.WARNING or getattr(record, 'audit', False):
            record.sample_rate = 1.0
            return True
        if random.random() < self.rate:
            record.sample_rate = self.rate
            return True
        return False

handler = logging.StreamHandler(sys.stdout)
handler.addFilter(SampleFilter(0.1))

# Audit and security events are often INFO, so mark them to bypass sampling
logging.getLogger('auth').info('sign-in succeeded', extra={'audit': True})

For 1,000 INFO calls and one error, this wrote 100 INFO lines and the error. The sample_rate attribute is the important part: record it on every kept line, so that when you count events you can multiply by 1 / sample_rate and get an estimate of the true total. A sample without its rate gives numbers that quietly understate reality. Put it in a field if you use JSON logging.

Databases offer the same control. PostgreSQL's log_min_duration_sample and log_statement_sample_rate log a fraction of slow statements, and statements above log_min_duration_statement are always logged, so the worst queries are never sampled away. See the Postgres logs guide.

Sample in the OpenTelemetry Collector

The Collector's probabilistic_sampler processor supports logs, with the processor's stability listed as alpha for logs. By default it hashes the trace ID, or an attribute you name, against sampling_percentage. Its sampling_priority option reads a per-record attribute from 0 to 100, where 0 means never keep and 100 or more means always keep. Use a transform processor to set that attribute on important records:

processors:
  transform/keep_errors:
    log_statements:
      - set(log.attributes["sampling.keep"], 100) where log.severity_number >= SEVERITY_NUMBER_WARN
      - set(log.attributes["sampling.keep"], 100) where log.attributes["audit"] == true
  probabilistic_sampler/logs:
    sampling_percentage: 10
    sampling_priority: sampling.keep
  batch: {}

service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [transform/keep_errors, probabilistic_sampler/logs, batch]
      exporters: [otlphttp]

On Collector v0.165.0 with 2,000 JSON lines in a filelog receiver (5 errors and 1,995 info lines), the output was 181 records: all 5 errors and 176 of the info lines, about 8.8%. Records without the attribute use the default percentage, so give audit and security events (often logged at INFO) the same keep attribute, as the second statement does, or send them through a separate pipeline with no sampler. Order matters: the transform step must run before the sampler. Because the hash uses trace IDs, lines from one trace are kept or dropped together when they carry a trace ID, so parse it from the log line as shown in the OpenTelemetry logs guide. For trace sampling concepts, see trace sampling.

Check That Sampling Is Safe

• Errors before and after. Count level >= warn at the source and at the destination. They should match.
• Alerts still fire. An alert on an error rate must not read from a sampled stream unless it divides by sample_rate.
• Incident lookups. Pick a past incident and confirm the lines you used are in the kept set. If they were info-level, lower the sampling rate for that service.
• Dropped counts. Watch for journald or agent notices about dropped messages.
• Retention. If you cannot sample at all because of regulation, shorten retention or route unsampled data to cheaper storage instead. See log retention and logging best practices.

Sending Sampled Logs to Sazabi

Sazabi, an AI-native observability and security platform, bills log ingestion as metered usage, per its production checklist, which says scaling traffic up raises metered ingestion and tells you to estimate each source's volume and check limits before cutover. That makes the source or the Collector the right place to shape volume, using the sampler above on the OpenTelemetry Collector path before records reach Sazabi's intake. Keep warnings, errors, and security events at 100%. After you turn sampling on, watch each source's ingestion volume on the Intake page to confirm the reduction you expected.