Cron Job Monitoring: Detect Missed and Duplicate Runs

Last verified on Nov 8, 2026.

Scheduled jobs fail without making noise. A nightly export that never starts produces no error, no stack trace and no failed request. The first symptom is a customer asking why the report is empty. Jobs also fail in two other ways: they run twice and double-charge or double-send, or they run long and hold a lock until the next run collides with the first.

The scheduler itself is part of the problem. The Kubernetes documentation says a CronJob creates a Job approximately once per execution time, and that in some circumstances two Jobs might be created or no Job might be created. Managed schedulers, in-process timers and plain crontab each have their own version of that caveat. You cannot treat "the schedule exists" as "the job ran".

What the evidence looks like

A healthy job leaves two events per run: one when it starts and one when it finishes. Log both with a run ID and the time the run was meant to start.

{"ts":"2026-11-01T03:00:02Z","job":"nightly-export","event":"started","run_id":"8f2c1a","scheduled_for":"2026-11-01T03:00:00Z"}
{"ts":"2026-11-01T03:06:55Z","job":"nightly-export","event":"finished","run_id":"8f2c1a","rows":1204331,"duration_ms":413000}

This is an example. A missed run is the absence of that pair for one scheduled_for value, which is why the absence has to be computed against a list of expected times rather than seen in the logs directly. A duplicate is two started events with the same scheduled_for and different run IDs. A long run is a finished event with a duration far above the job's own history, or no finished event before the next started.

Platform signals help. The Kubernetes docs describe a TooManyMissedTimes warning event when a CronJob has more than 100 missed schedules, and a startingDeadlineSeconds field after which a late Job is skipped and counted as failed. kube-state-metrics exposes kube_cronjob_status_last_schedule_time and kube_cronjob_status_last_successful_time for each CronJob.

How to detect it step by step

1. Alert on time since last success. The Prometheus instrumentation guide says the key metric of a batch job is the last time it succeeded. With kube-state-metrics, a daily job that should finish every 24 hours can use a grace period of two hours:

time() - kube_cronjob_status_last_successful_time{cronjob="nightly-export"} > 26 * 3600

For jobs outside Kubernetes, set a gauge such as job_last_success_timestamp_seconds at the end of each successful run and use the same expression. A metric that stops updating is covered by absent_over_time, which returns 1 when a series has no samples in the range you give it.

2. Compute expected runs and look for holes. If your jobs only emit logs, keep a small table of expected run times and join it to the finish events:

SELECT e.expected_at
FROM expected_runs e
LEFT JOIN job_events f
  ON f.job = e.job AND f.event = 'finished'
 AND f.ts BETWEEN e.expected_at AND e.expected_at + INTERVAL '2 hours'
WHERE e.job = 'nightly-export' AND e.expected_at < now() - INTERVAL '2 hours'
  AND f.ts IS NULL

3. Count duplicates. Group start events by scheduled_for and flag any group with more than one run ID. Because the Kubernetes docs say jobs should be idempotent, a duplicate is a signal to check your idempotency key, not only to alert.

4. Watch duration and overlap. Alert when a run exceeds roughly twice its trailing 14-run median, and when a second started arrives while the first has no finished. The default Kubernetes concurrencyPolicy is Allow, so overlapping runs are possible unless you changed it.

Thresholds and pitfalls

Set the grace period as a fraction of the interval, not a fixed number. Ten minutes suits an hourly job and two hours suits a daily one. Too tight and every slow day pages someone. Too loose and a missed run is found after the next one is due.

Watch for these false positives:

• Daylight saving time. A local-time schedule skips or repeats an hour twice a year. Schedule in UTC or set the time zone explicitly. Kubernetes CronJobs have a time zone setting for this reason.
• Log delay. If your pipeline is behind, finish events arrive late and look missing. Alert on pipeline freshness first and silence run alerts when it fails.
• Legitimate empty runs. A job with no work to do should still log finished with rows: 0. Otherwise you cannot tell skipped from idle.
• Skipped starts. A start that misses its deadline is silent in your application logs. Watch the platform events too.

Keep one rule per job family rather than one per job. A shared rule over a job_last_success_timestamp_seconds label set covers new jobs as soon as they report.

What to do when it fires

Start at the scheduler and work toward the job. Check whether the schedule is suspended, whether the controller or scheduler process is healthy, and whether a deadline or concurrency setting skipped the run. Then check the runner: image pulls, quota, node pressure and secrets. Only then read the job's own logs.

Backfill by running the missed window by hand, using the same scheduled_for value so an idempotency check can reject a double run. Then add the missing evidence that made this hard to see, usually a scheduled_for field or a success gauge.

Related guides: stale data covers the downstream effect of a job that stopped, and crash loops covers jobs that start and die. Webhook gaps applies the same absence logic to inbound events. If you run durable workflows, the Temporal, Inngest and Trigger.dev integrations show how to get run events into your logs, and the Prometheus page covers the metrics side.