Stale Data: How to Set Up Data Freshness Monitoring

Last verified on Nov 8, 2026.

Stale data is the quietest failure there is. A cache stops refreshing, a nightly sync fails after its first step or an upstream feed goes silent. Every request still returns 200 and every dashboard still loads. The only thing wrong is that the values are three days old, and nobody notices until a customer does.

Because nothing errors, you cannot detect staleness from error rates. You need a direct measure: how old is the newest piece of data, and how long since the job that refreshes it last succeeded?

What the evidence looks like

For a table, the evidence is a latest timestamp that has stopped moving. This example shows a freshness query result for a directory table:

table=directory_people  newest_updated_at=2026-11-05T02:00:14Z  age_hours=71.4  expected_every_hours=24

For a job, it is a last-success time far in the past while the scheduler still reports runs. This example shows a scheduler log that looks healthy at a glance:

2026-11-08T02:00:01Z sync-directory started
2026-11-08T02:00:09Z sync-directory fetch status=401 upstream=hr-api
2026-11-08T02:00:09Z sync-directory finished rows_written=0 exit=0

The job ran and exited zero, but it wrote nothing. A check based on "did the job run" would pass.

How to detect it

1. Pick a freshness timestamp for each important dataset: updated_at, loaded_at or an event time from the source. Prefer a value written by the loader over one copied from the source, so the check reflects your pipeline.
2. Write down the expected cadence for each dataset.
3. Measure age and compare it with a multiple of the cadence.

SELECT 'directory_people' AS dataset,
       now() - max(loaded_at) AS age,
       interval '36 hours'   AS max_age,
       now() - max(loaded_at) > interval '36 hours' AS stale
FROM directory_people;

dbt formalizes this for warehouse tables. A source declares a loaded_at_field and warn_after and error_after thresholds, and dbt reports a warning or an error when the newest row is older than the threshold.

For jobs, emit a last-success timestamp as a metric when a run finishes with data, and alert on its age.

# Last successful sync older than 36 hours
time() - sync_last_success_timestamp_seconds{job="sync-directory"} > 36 * 3600

# Job has never reported at all
absent(sync_last_success_timestamp_seconds{job="sync-directory"})

If your jobs push to a Prometheus Pushgateway, it records push_time_seconds and push_failure_time_seconds for each group so you can alert on pushers that failed or have not run recently. Use absent() as well, since a missing series is a failure too, as the Prometheus docs note.

For caches, compare the object's age with its intended lifetime. HTTP defines this clearly: a response is fresh until its age exceeds its freshness lifetime and stale afterward. A cache that serves content far past its lifetime, because revalidation is failing or the origin is down, is a staleness bug even though it is working as designed.

Thresholds and pitfalls

Alert at about twice the cadence for a daily job and tighter for frequent ones. A 24-hour job checked at 25 hours pages you for every slow run. A 36 to 48 hour limit leaves room for a delayed run and still catches a stuck one within two days. Use a warning level at the first threshold and a page at a higher one.

Pitfalls:

• Timestamps without data. A job can update loaded_at and load zero rows, as in the log above. Add a row-count or source-watermark check next to the age check.
• Partial staleness. The newest row can be fresh while one partition, tenant or source stopped. Check the freshness of each partition, not only the global maximum.
• Weekends and holidays. A feed that legitimately pauses needs a calendar-aware limit or a longer window.
• Clock and time zone mismatch. Store timestamps in UTC and compare with the same clock.
• Missing metrics. If the job never emitted its metric, the age rule has nothing to evaluate. Pair it with an absence rule.

What to do when it fires

Start at the source and work forward. Did the upstream publish? Did the fetch authenticate? Did the load write rows? In the example above the answer was in the log: a 401 from the upstream. An expired credential is one cause worth ruling out early. See expiring certificates and tokens for catching those before they lapse.

Make jobs fail loudly. Exit non-zero when a fetch fails or zero rows come back where rows are expected. Then backfill the gap and confirm the age metric falls.

Related guides: missed schedules covers jobs that never start, reconciliation mismatches covers two systems that disagree and webhook gaps covers missing deliveries. The Airbyte and Dagster pages cover two pipeline tools you may be checking.