Latency Spikes: How to Detect and Attribute Sudden Slowdowns

Last verified on Nov 8, 2026.

A latency spike is a step change: response times jump within minutes and stay there, or jump and fall back. A deploy that adds an unindexed query, a dependency that starts timing out, a cache flush and a noisy neighbor on a shared host all produce the same graph. Unlike slow drift, a spike has a start time, and the start time is the best clue you will get.

The goal is to detect the jump quickly, with few false alarms, and to leave the responder knowing what changed at that moment.

What the evidence looks like

In an access log, a spike is a shift in the duration column for the same route at a specific minute.

12:03:10 GET /api/search 200  118ms  release=2026.11.07-1
12:03:41 GET /api/search 200  131ms  release=2026.11.07-1
12:05:02 GET /api/search 200 1480ms  release=2026.11.07-2
12:05:03 GET /api/search 200 1622ms  release=2026.11.07-2

This is an example. The release changed at 12

How to detect it step by step

1. Measure with histograms and aggregate correctly. The Prometheus documentation notes that histogram quantiles are computed on the server with histogram_quantile() and that averaging precomputed quantiles across instances is statistically meaningless. Compute the quantile after summing the buckets.

2. Compare now with a recent baseline. For a step change, a short baseline works better than a weekly one. Compare the last 5 minutes with the same 5 minutes 1 day ago and with the previous hour:

histogram_quantile(0.95, sum by (le, route) (rate(http_request_duration_seconds_bucket[5m])))
  > 2 *
histogram_quantile(0.95, sum by (le, route) (rate(http_request_duration_seconds_bucket[5m] offset 1d)))

3. Require persistence. Prometheus alerting rules support a for duration so a condition must hold before the alert fires. Two or three minutes removes single slow requests without delaying real incidents much.

4. Use two windows. The Google SRE workbook adds a shorter window to its multi-burn-rate alerts so they fire only while the problem is still happening, which reduces false positives. The same shape works for latency: require both the 5-minute and the 1-hour ratios to be elevated, and clear the alert when the short one recovers.

5. Attribute the jump by slicing. Group the same query by release, route, status class, region and dependency:

SELECT date_trunc('minute', ts) AS minute, release, route,
       percentile_cont(0.95) WITHIN GROUP (ORDER BY duration_ms) AS p95_ms,
       count(*) AS requests
FROM access_log
WHERE ts > now() - INTERVAL '30 minutes' AND status < 500
GROUP BY 1, 2, 3 ORDER BY 1

6. Overlay change events. Log deploys, flag flips, config pushes and scaling events as events with timestamps, and show them on the same chart. A spike that starts within a minute of a change has a short list of suspects.

Thresholds and pitfalls

A ratio of 2 against the same time yesterday, held for 3 minutes, is a reasonable starting point for user-facing routes. Add an absolute floor, such as p95 above 500 ms, so a move from 5 ms to 12 ms does not alert. Use the SLO or the timeout as the hard ceiling.

Watch for these:

• Mix shift. If traffic moves toward a slower route, the blended percentile rises with no regression. Alert per route.
• Fast failures. The SRE book advises separating the latency of successful and failed requests, because a quick error can hide a slow success path. Filter to successful requests for the latency alert and alert on errors separately.
• Low traffic. With a handful of requests per minute, one slow call moves p95. Require a minimum request count.
• Top bucket saturation. If your highest finite histogram bucket is 10 seconds, a rise from 12 to 60 seconds is invisible. Keep a bucket above the timeout.
• Warm-up after deploy. Cold caches and connection setup cause short, expected bumps. A short for period or a deploy-aware silence helps, but do not silence long.
• Retries. Latency measured per attempt hides what the user waited. Measure at the edge too.

What to do when it fires

Look at the slices before the code. If one release, one region or one dependency carries the spike, the cause is usually obvious. Roll back a release if the start matches a deploy and the impact is large, then investigate with traces or query logs. If a dependency is slow, see vendor degradation and use timeouts and circuit breakers.

A deploy that slows a route often introduces new errors too. Sazabi's novel event detection notices a new kind of error appearing in your logs, for example after a deploy, and it does not measure latency, so keep the latency alert described here alongside it. See also latency drift for slow creep, tail latency for the slowest requests and performance degradation for the group. To get histograms into your stack, use the Prometheus or OpenTelemetry integrations.