Latency Monitoring: How to Catch Slow Drift Over Weeks

Last verified on Nov 8, 2026.

Some latency problems never page anyone. Search gets 5% slower every week, nobody deploys anything dramatic and every individual day looks fine. After a quarter, the endpoint is 60% slower than it was, the timeout margin has gone and the next small traffic increase turns into an incident. Gradual drift is the cost of data growth, bloated queries, creeping payload sizes and dependencies that slowly degrade.

Alerts built on a fixed threshold catch this late, because the threshold is set above the old normal. You need to compare the present with the past on a time scale of weeks.

What the evidence looks like

Drift shows up as a slow slope in a percentile, not as a spike. The same route measured daily looks like this:

route=/search   week  p50_ms  p95_ms  p99_ms  requests
                W40    118     410     880    2.1M
                W41    121     431     930    2.2M
                W42    125     455     990    2.2M
                W43    131     482    1060    2.3M

This is an example. The p95 climbs about 6% a week while traffic grows 3%. No single week trips a 500 ms alert, but the line only goes one way.

Look for the same slope in correlated signals: rows scanned per query, response size, time spent in one downstream span, garbage collection pauses and queue depth. These explain the drift and tell you where to look.

How to detect it step by step

1. Record latency as a histogram, per route. The Prometheus documentation explains that histograms expose bucketed counts and that quantiles are calculated on the server with histogram_quantile(). It also warns that averaging precomputed quantiles across instances gives statistically meaningless results, so aggregate buckets first. The Google SRE book makes the same case for distributions: an average of 100 ms at 1,000 requests per second can hide 1% of requests taking five seconds.

2. Compare each day with the same day last week. Use the offset modifier:

histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{route="/search"}[1d])))
/
histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{route="/search"}[1d] offset 7d)))

A ratio of 1.0 means no change. Alert when it stays above about 1.15 for several days.

3. Fit a trend to a recorded series. Save the daily p95 as a recording rule, then use deriv(), which calculates the per-second derivative with simple linear regression, or predict_linear(), which predicts a value some seconds from now. For example, ask when the 28-day trend crosses your timeout:

predict_linear(route:request_duration_seconds:p95_1d{route="/search"}[28d], 14 * 86400) > 2.0

4. Do the same from logs if you lack metrics. From access logs with a duration field:

SELECT date_trunc('week', ts) AS week,
       percentile_cont(0.95) WITHIN GROUP (ORDER BY duration_ms) AS p95_ms
FROM access_log
WHERE route = '/search' AND status < 500
GROUP BY 1 ORDER BY 1

5. Check where the time goes. If you have traces, group span durations by operation and compare the same two weeks. One downstream call growing while the rest hold steady is the answer to most drift questions.

Thresholds and pitfalls

Measure successes only. The SRE book notes that a fast HTTP 500 can mask a slow successful path, so latency of successful and failed requests should be separated.

Histogram buckets limit what you can see. If your nearest buckets are 250 ms and 500 ms, a move from 300 ms to 450 ms is invisible between them. Add buckets around the values your users and timeouts care about.

Other traps:

• Traffic mix. If a slow route gets more of the overall traffic, the blended percentile rises with no real regression. Slice by route and request type.
• Seasonality. Compare to the same weekday, not yesterday. Month-end jobs and weekend traffic change the baseline.
• Data volume. A table that doubled in size may be the real cause, which is a capacity question, not a bug. Chart row counts next to latency.
• Low traffic. A route with 200 requests a day has a noisy p95. Use weekly windows.

A rule of thumb is to alert on a sustained relative change, such as 15% over a week or more, and to review the slope on a dashboard monthly. Pair this with a hard threshold so a slow slope does not hide a sudden jump.

What to do when it fires

Pin down the start of the slope from the daily series and look at what changed near it: a release, a schema migration, a feature flag or a traffic source. Then use the trace or query breakdown to find the operation that grew. Typical fixes are a missing index, a query that fetches more columns than it needs, a cache with a falling hit rate and an unbounded list in a response.

Write the fix and add a budget: an alert at a ratio of 1.15 against the previous month keeps it from returning. For sudden jumps, see latency spikes. For the slowest requests specifically, see tail latency and performance degradation. To ship the metrics, use the Prometheus, OpenTelemetry or Grafana integrations.