Log Anomaly Detection: Catch New Errors After Deploys

Last verified on Nov 8, 2026.

A novel event is a log message your services have never emitted before. It usually shows up right after a deploy, a dependency upgrade or a configuration change, because new code paths fail in new ways. Alerts keyed on known errors or on volume thresholds stay quiet, since nothing you already track has changed.

This guide covers how to detect first-seen messages with any log store, how to keep the result usable and what to do when one appears.

Why new messages matter

Most log anomaly detection watches counts: errors per minute, lines per second. That finds spikes in things you already know about. A novel event is a change in vocabulary, not volume. One line, logged twice an hour, can be the only trace of a save path that now silently skips writes.

An example of what you are looking for, from a service that shipped at 14:02:

2026-11-08T14:06:41Z level=warn service=docs-api version=4.18.0 msg="skipping save, draft locked" doc_id=8812 user_id=317

Nothing in your dashboards changed. The message simply did not exist in the 30 days before version 4.18.0.

Novelty is not tied to severity. The OpenTelemetry log data model places INFO at severity numbers 9 to 12, and failure wording is often logged at that level or lower. A new "falling back to default config" at info level deserves the same attention as a new error.

What to log so novelty is detectable

Detection is only as good as the message text. Two habits make it work:

• Keep the message constant and move variables into fields. msg="payment declined" reason=insufficient_funds is comparable across requests. msg="payment 4412 for user 317 declined" is different every time.
• Record the version that emitted the line. OpenTelemetry defines service.version as a resource attribute for this, and the format is yours to choose, so a commit SHA works.

Detect first-seen messages step by step

1. Reduce each message to a stable key. If you log structured events, use the event name or the constant msg field. For free text, replace numbers, UUIDs, hex strings and quoted values with a placeholder and hash what remains.
2. Store the key with the service, the environment and a first-seen timestamp. A small table updated at ingest is enough.
3. Query for keys whose first-seen time falls inside the window you care about:

-- Example: message keys first seen in the last hour, production only
SELECT msg_key, service, min(ts) AS first_seen, count(*) AS hits,
       any_value(message) AS sample
FROM logs
WHERE env = 'production'
  AND ts >= now() - interval '30 days'
GROUP BY msg_key, service
HAVING min(ts) >= now() - interval '1 hour'
ORDER BY hits DESC;

4. Join the result against your deploy events. A new key that appears within minutes of a release is the high-value case, and the version field tells you which build introduced it.
5. Route only the useful subset: warn and above, or any severity with more than a few hits, in a service that just deployed.

Thresholds and pitfalls

Look-back length decides your false positive rate. A weekly job logs its summary line once a week, so a seven-day history flags it as new every Monday. Keep a history longer than your longest business cycle, such as a month-end or quarterly job.

Poor normalization floods you. If user names or request IDs stay inside the message text, every line looks novel. Watch the ratio of distinct keys to total lines. When it climbs with traffic, fix the key, not the alert.

Other sources of noise:

• A brand new service or environment has nothing in its history, so everything is novel. Suppress the first day or seed the history from staging.
• Multi-line stack traces should be keyed on the exception class and top frame, not the full text.
• Debug-level lines enabled temporarily during an investigation will fire once and should be dismissed, not tuned away.
• Novelty says nothing about impact. Pair it with the error rate of the affected route before you page anyone.

What to do when it fires

Read the sample line and find the code that emits it. Check the version and the deploy time to confirm the link to a release. Then look at the neighboring signals: did the error rate on that route move, did a downstream write stop (see the lost writes guide), did client errors rise (see the frontend breakage guide)?

If the message is harmless, mark the key as known so it does not fire again. If it is a failure path, roll back or flag off the change first and fix second.

Where Sazabi fits

Sazabi detects novel events for every production project with no setup, and the result feeds regression issues. The novel events detection page describes what you see when one opens. To send logs from any stack, start with OpenTelemetry or the Vector integration.