Data Drift Detection: Catch Field Changes in Event Logs

Last verified on Nov 8, 2026.

Field drift is a change in what your events contain, not whether they arrive. A field that was always a number becomes a string. A column that was never null starts arriving empty on a few percent of writes. A status field gains a value no consumer knows about. Every request still returns 200, so error monitoring stays quiet while downstream reports, billing jobs and models quietly absorb bad data.

Two things drift. The shape changes when keys appear, disappear or change type. The values change when a field keeps its type but its distribution moves: more nulls, more empty strings, new categories or payloads that are suddenly ten times larger. This guide covers both. Most teams start with schema checks and find they miss the second kind.

What the evidence looks like

The clearest evidence is a field-level rate that changes at a specific time, often a deploy.

{"ts":"2026-11-03T14:22:09Z","event":"order.write","order_id":"o_88172","company_id":null,"currency":"","items":3}
{"ts":"2026-11-03T14:22:11Z","event":"order.write","order_id":"o_88173","company_id":"c_204","currency":"USD","items":1}

This is an example. Taken alone, the first line looks like one bad record. As a rate, company_id going from 0% null to 3% null on order.write after a 14

Other signs in logs and events: a new key in the payload, a key that vanishes, a value type change such as "items":"3", a category not seen in the previous 30 days, and a size field (bytes, array length) whose upper percentiles jump.

How to detect it step by step

1. Pick the fields that matter. Drift on every field is noise. Start with fields that are joined on, used in billing or read by another team: identifiers, amounts, currencies, status and tenant keys.

2. Track a few numbers per field per hour. Null rate, empty-string rate, distinct count, and the 50th and 99th percentile of length or size:

SELECT date_trunc('hour', ts) AS hour,
       count(*) AS writes,
       avg((company_id IS NULL)::int) AS null_rate,
       avg((currency = '')::int) AS empty_rate,
       count(DISTINCT status) AS distinct_status,
       percentile_cont(0.99) WITHIN GROUP (ORDER BY octet_length(payload)) AS p99_bytes
FROM events
WHERE event = 'order.write'
GROUP BY 1
ORDER BY 1

3. Find new categories directly. For low-cardinality fields, compare today's values to a trailing set:

SELECT status, count(*) FROM events
WHERE event = 'order.write' AND ts > now() - INTERVAL '1 day'
  AND status IS NOT NULL
  AND status NOT IN (
    SELECT DISTINCT status FROM events
    WHERE event = 'order.write' AND status IS NOT NULL
      AND ts BETWEEN now() - INTERVAL '31 days' AND now() - INTERVAL '1 day')
GROUP BY status

4. Check shape against a contract. Write a JSON Schema for your key events. The additionalProperties keyword controls how extra properties are handled, so you can reject unlisted keys instead of accepting them. Validate a sample of production events in CI or in the pipeline and count failures per field.

5. Use a statistical test for distributions. The two-sample Kolmogorov-Smirnov test compares two samples of a numeric field, and SciPy provides it as scipy.stats.ks_2samp. The Evidently documentation describes its default drift logic: a Kolmogorov-Smirnov test for numeric columns with more than five unique values and a chi-squared test for categorical columns, on small reference datasets. Use these when a simple rate is not enough, for example on an amount or a latency-like field.

Thresholds and pitfalls

A fixed null-rate limit works for fields that are required, where the right answer is zero. For everything else, compare against the same hour one week earlier so weekday patterns do not trigger alerts. A move from 0.5% to 1.5% empty is a stronger signal than a swing from 40% to 45% on a field that always varies.

Common false positives:

• Identifiers. IDs, timestamps and free text always differ between samples, so statistical tests flag them. Exclude them.
• Small samples. Tests on a few hundred rows fire on chance. Require a minimum count per window, such as 500 events, before alerting.
• Mix shift. A new large customer changes the category distribution without any bug. Break rates out by tenant or client version before you decide.
• Planned changes. A migration that adds a currency is drift by definition. Link alerts to the release that caused them.

Statistical tests report that something changed, not that it is wrong. A significant result with a tiny effect size is usually not worth a page.

What to do when it fires

Find the first affected event and the deploy or upstream change that precedes it, then look at which clients, versions or tenants produce the bad values. Fix at the producer when you can. If the data already landed, quarantine the affected window so jobs that read it can skip or reprocess it.

Then turn the incident into a contract: a schema check for the field, plus the hourly rate query above as a standing alert. Related: cardinality jumps covers drift in the number of distinct values, reconciliation mismatches catches the same problem downstream and lost writes covers records that never land. If your events flow through a pipeline, the Vector and OpenTelemetry pages show where checks can sit. The sibling guide on clock skew covers drift in timestamps specifically.