Clock Skew: How to Detect Timestamp Drift Between Hosts

Last verified on Nov 8, 2026.

When two machines disagree about the time, every system that orders events by timestamp starts to lie. A trace shows a child span starting before its parent. A log search for "what happened before the error" misses the cause because the cause was stamped four minutes later by a host with a slow clock. Token validation, certificate checks, rate limit windows and cache expiry all depend on agreeing about now.

Skew is quiet. Nothing errors until a signature check fails or an incident review cannot reconstruct the sequence. You need to measure it directly.

What the evidence looks like

In logs, skew appears as events that violate causality or arrive impossibly late or early.

web-1  2026-11-02T12:00:01.204Z  POST /checkout received request_id=r_77
db-2   2026-11-02T11:56:03.990Z  query ok request_id=r_77 duration_ms=12
web-1  2026-11-02T12:00:01.310Z  POST /checkout 200 request_id=r_77

This is an example. The database logged the query almost four minutes before the request that caused it. In traces, the same problem is a span that starts earlier than its parent or ends later than the parent that waits for it. In metrics, a host's samples land in the wrong scrape interval or appear to come from the future.

The OpenTelemetry logs data model gives you two timestamps to compare. Timestamp is the time the event occurred, measured by the origin clock. ObservedTimestamp is the time the event was observed by the collection system. The difference between them is lag plus skew, and you can chart it per source.

How to detect it step by step

1. Read the clock offset directly. NTP is specified in RFC 5905. With chrony, chronyc tracking prints System time, the number of seconds the clock is fast or slow of NTP time, alongside Last offset and RMS offset. In Prometheus, the node exporter's timex collector exposes metrics including node_timex_offset_seconds (the time offset between the local system and the reference clock) and node_timex_sync_status:

abs(node_timex_offset_seconds) > 0.05
or
node_timex_sync_status == 0

2. Compare event time to arrival time. Per source, take the median of observed - event time over recent minutes. Normal lag is positive and small. Negative values mean the source is ahead of the collector. A median that drifts upward by whole seconds across a day is a slowly drifting clock.

SELECT host,
       percentile_cont(0.5) WITHIN GROUP (ORDER BY observed_ts - event_ts) AS median_lag
FROM logs
WHERE observed_ts > now() - INTERVAL '10 minutes'
GROUP BY host
HAVING abs(extract(epoch FROM percentile_cont(0.5) WITHIN GROUP (ORDER BY observed_ts - event_ts))) > 5

3. Check causality across services. Join events that share a request or trace ID and look for effects stamped before their causes. A count of negative gaps per host pair tells you which host is off.

4. Check traces. Count spans whose start precedes the parent's start. Group by service and host to find the producer of the odd clocks.

5. Check reference time and sync status. Monitor the time service itself: is the daemon running, is the host synchronized and which servers is it using? Amazon documents the Amazon Time Sync Service for EC2 instances and warns that inaccurate time can cause AWS to reject requests whose signature date does not match.

Thresholds and pitfalls

For logs and traces, an offset above a few hundred milliseconds starts to scramble sequences between fast services. Offset of several seconds breaks signature and token checks. Choose a limit from your tightest dependency, then alert at half of it.

Pitfalls:

• Lag is not skew. Batching, buffering and slow exporters add positive delay. Look at the minimum or the lower percentiles to separate a clock that is wrong from a pipeline that is slow.
• Client clocks. Browser and mobile timestamps are set by the user's device. Stamp events on arrival at your edge and keep the client time as a separate field.
• Virtual machine pauses. A suspended or migrated VM wakes with a stale clock until it resynchronizes. Short spikes after a resume are not a problem if they correct themselves.
• Containers. Containers share the host kernel's clock. Fix it on the host, not in each container.
• Time zones. An offset of exactly one hour or a whole number of hours is a time zone or daylight saving mistake, not drift. Logs should be in UTC.

What to do when it fires

Find the host with the largest offset and check whether its time daemon is running and can reach its servers. Restart or reconfigure it and watch the offset converge. Where clocks cannot be trusted, such as with client timestamps, order by arrival time or by a logical counter, and use the source timestamp for display only.

After a correction, expect old events to remain out of order in storage. Mark the affected window in your incident notes so no one chases a false sequence. Related guides: field drift covers other changes in event fields, missed schedules covers jobs that depend on accurate time and stale data covers freshness gaps. Collection paths include OpenTelemetry, Prometheus and Fluent Bit.