Configuration Drift Detection: Find Unplanned Changes

Last verified on Nov 8, 2026.

Configuration drift is the gap between what you intended to run and what is running. It appears when someone edits a setting in a console during an incident, when a feature flag is flipped for one environment, when an autoscaler or an operator changes a value or when staging and production were never quite the same.

Drift is quiet. Nothing fails at the moment of change, and the symptom shows up later as behavior no deploy explains.

Where drift comes from

• Manual changes to cloud resources that your infrastructure code does not know about.
• Environment variables and flags that differ between environments or between instances of one service.
• Cluster objects edited with a direct command instead of through the manifest in version control.
• Rollouts that stall halfway, leaving some instances on the old settings.
• Defaults that changed in a dependency or platform upgrade.

Evidence in your telemetry

You rarely get a log line that says "drift". You get an odd distribution. One example is a startup line that includes a hash of the effective configuration:

2026-11-08T10:02:17Z msg="service started" service=checkout version=4.18.0 config_hash=9f3a1c env=production host=web-04
2026-11-08T10:02:19Z msg="service started" service=checkout version=4.18.0 config_hash=b71e50 env=production host=web-05

Same version, same environment, two config hashes. Either a rollout is in progress or two hosts drifted.

Other evidence: errors concentrated on one host or one region, a behavior change with no deploy near it and audit events showing a change made outside your pipeline.

Detect it step by step

1. Compare declared state with real state on a schedule. Terraform plans against real infrastructure by default, and terraform plan -refresh-only previews the state changes a refresh would make without applying them. With -detailed-exitcode, exit code 0 means no changes, 1 means error and 2 means changes are present, so a scheduled job can fail on 2.

# Example scheduled check
terraform plan -refresh-only -detailed-exitcode -input=false
echo "exit=$?"   # alert when exit=2

2. Use the platform's own drift checks where they exist. AWS CloudFormation can detect whether a stack's actual configuration differs from its expected configuration, and reports statuses such as IN_SYNC and DRIFTED. A resource counts as drifted if any actual property value differs from the expected one. Resources that do not support drift detection get NOT_CHECKED, so watch for that status as a blind spot.
3. For Kubernetes manifests, kubectl diff compares a file with the live object. It exits 0 when there are no differences, 1 when there are, and above 1 when the command itself fails. Keep those apart or a failing job will look like drift.
4. Emit a configuration hash at startup, as in the example above, and query for services with more than one hash in the same environment:

-- Example: services running more than one configuration at the same time
SELECT service, env, count(DISTINCT config_hash) AS configs, count(DISTINCT host) AS hosts
FROM logs
WHERE msg = 'service started' AND ts >= now() - interval '1 day'
GROUP BY service, env
HAVING count(DISTINCT config_hash) > 1;

5. Log every flag and setting change with who, what, old value and new value, then join those events to error rates. A flag rolled out to 10 percent of users that lines up with 10 percent of failures is a cause, not a coincidence.

Thresholds and pitfalls

Any difference counts as a finding, but not every finding is a problem. Rank them.

• A rolling deploy creates two hashes for a few minutes. Ignore it until the rollout window ends.
• Fields that tools manage for you, such as replica counts under an autoscaler or computed timestamps, produce permanent diffs. Exclude them explicitly and keep the exclusion list short and reviewed.
• Intended differences between staging and production, such as endpoints and sizes, belong in an allowlist. Compare shape and keys, not values, for those.
• A drift check that errors out is not the same as a clean result. Alert on the check failing too.
• Credentials in plan output are sensitive, so keep plan logs access controlled. See the exposure guide.

What to do when it fires

Decide which side is right. If the manual change was a good fix, write it into code and merge it. If it was a mistake or unreviewed, apply the declared state again. Record who changed it and why, then close the path that allowed it, for example by limiting console write access.

For permission-related changes specifically, see privilege changes. Related reading: capacity risks for limits that drift toward exhaustion, and the Terraform and Kubernetes integration pages for tooling around the checks above.