Logging Best Practices for Production Services

Last verified on Nov 8, 2026.

Good logging answers questions during an incident: what failed, for whom, in which request, and what changed just before. The practices below are what make that possible, and each one has its own guide if you want the details. The short version is to log events that matter, in a consistent structure, with an ID that ties them to a request, without secrets, at a volume you can afford to keep.

Start From the Questions

Decide what you will need to answer before you decide what to log. OWASP's logging guidance splits the purpose into operational use, such as troubleshooting, and security use, such as spotting attacks and supporting an investigation. It also warns that a blind checklist leads to "alarm fog" where real problems go unnoticed, so tie each log line to a question. Typical ones:

• Did this request succeed, and how long did it take?
• Which user, tenant, or order was involved?
• What did we call downstream, and what came back?
• What changed in configuration or deployment before the failure?
• Did someone try something they should not have?

If you cannot name the question a line answers, it is a candidate to drop or demote to debug.

What to Log

OWASP lists events to always log. Adapting that list for a service:

OWASP also says never to exclude events from "known" users such as internal systems, search engine robots, monitoring, or auditors. Add a classification flag instead, so you can filter them out when reading. For each event, record when, where, who, and what: a time, the application and host, the actor, and the event type with a severity and description.

Use Levels Consistently

A level is only useful if every service means the same thing by it. The Python documentation defines the standard set, and it is a good shared vocabulary:

Syslog numbers its severities 0 (Emergency) through 7 (Debug), and OpenTelemetry maps levels from any source into numeric ranges so they compare across systems. Python's default threshold is WARNING, so INFO lines disappear until you set a level. Write down your own rule, for example "ERROR means a request failed and a person may need to act", and apply it. Overusing ERROR for expected conditions teaches everyone to ignore it. See log levels.

OWASP adds a rule about changing levels: the default level must provide enough detail for business needs, and it should not be possible to switch off logging of events required for compliance. Changes to the level should follow change management or an approved algorithm.

Write Structured Events

Log one event per line with named fields, not a sentence with the data embedded. Keep the message constant and move variable parts to fields:

Bad:   order 4812 for jane@example.com failed after 3 retries: card declined
Good:  {"level":"ERROR","message":"payment failed","order_id":"4812","retries":3,"reason":"card_declined"}

The good line can be counted, grouped, and filtered, and it does not drop an email address into free text. OWASP also recommends structured formats such as JSON over unstructured text to resist log injection, since a carriage return or line feed in user input can otherwise forge an entry. See structured logging and the JSON logging guide for schemas and examples in several languages.

Add Context IDs

Every line should carry an ID that ties it to a request: the trace ID when you run tracing, or a request ID generated at the edge. OWASP's term is an interaction identifier, and it says to record it rather than make later analysis reconstruct which events belonged together. Add the service name, environment, and version as well. In a system with more than one service, this is what turns logs into a story. See the microservices logging guide for propagation, and the OpenTelemetry logs guide for how a bridge adds trace and span IDs automatically.

Keep Secrets and Personal Data Out

OWASP lists data that should not be recorded directly in logs and should be removed, masked, sanitized, hashed, or encrypted: access tokens, session identifiers, passwords, encryption keys, database connection strings, bank account or payment card data, sensitive personal data, and source code. File paths, internal addresses, and some personal data may need special handling too.

Practical steps:

1. Allow-list fields. Log the specific fields you need from a request, not the whole object or body.
2. Redact in the logger. Most libraries have a hook for it, such as pino's redact paths. Redact in the application or collector, where you own the data.
3. Test for it. Grep a day of staging logs for password, token, Authorization, and email-shaped strings.
4. Limit user input size. OWASP advises limiting the size of untrusted values that go into log messages.

If something slips through, treat it as an exposure and follow the secrets in logs guide.

Do Not Let Logging Hurt the Service

OWASP's verification list is a useful set of failure tests: ensure failures in logging do not stop the application, that logging cannot fill the disk and cause a denial of service, and that the effect of a full disk, missing permissions, or lost network on the logging path is understood. In practice:

• Write to standard output in containers and let the platform collect it. Kubernetes describes containers writing to stdout and stderr with a node-level agent forwarding them. OWASP also suggests considering sending the event stream, unbuffered, to stdout for the execution environment to manage.
• Cap file logs with rotation. See logrotate and journalctl for SystemMaxUse=.
• Avoid logging inside tight loops. One line per item in a million-row job is a volume incident waiting to happen.

Control Volume With Sampling

Volume is the main cost driver, and most of it is healthy, repetitive traffic. Reduce it by sampling the high-volume, low-value lines and keeping every warning, error, and audit event. Keep a sample_rate on sampled lines so counts can be scaled. See the log sampling guide for tested nginx, application, and Collector examples. Rate limits are a safety net, not a policy, because they drop errors in a burst along with everything else.

Set Retention Deliberately

OWASP is explicit on disposal: log data, temporary debug logs, and their backups and copies must not be destroyed before the required retention period and must not be kept beyond it, and legal, regulatory, and contractual obligations may set those periods. Decide per log type. Access logs for troubleshooting might need days. Security and audit logs often need far longer. Then make local rotation and central log retention agree, and copy logs off the host so they survive it.

Protect and Review the Logs

Logs hold sensitive information and are a target. OWASP's protection list includes tamper detection, copying log data to read-only media as soon as possible, recording and monitoring all access to logs, and reviewing who can read them periodically. Include logging in code review and testing: check that events are classified consistently, that field names, types, and lengths follow an agreed standard, and that logging is not open to injection. For the security side, see security logging.

A Short Checklist

• Every line is one JSON object with time, level, message, and service.
• Every request line has a trace or request ID.
• Levels follow a written rule, and the production default is INFO or WARNING.
• Errors, authentication events, and access failures are always logged and never sampled.
• No secrets, tokens, or personal data, and a redaction test in CI or staging.
• Rotation and retention are set, and the two agree.
• Logs leave the host, and someone owns the alerts that read them.

Sending Production Logs to Sazabi

Sazabi, an AI-native observability and security platform, has a production checklist that matches several points above: redact secrets and sensitive fields at your log source or collector rather than after ingestion, estimate each source's production volume, and confirm that the retention window fits your needs. It also offers a project setting to redact emails, phone numbers, SSNs, card numbers, and secrets with placeholders before ingested logs are stored, as an extra layer. Send logs through the OpenTelemetry Collector or the OpenTelemetry integration.