Performance Degradation: How to Detect It Early

Last verified on Nov 8, 2026.

Performance degradation is latency, throughput or timeout behavior getting worse enough that users or callers feel it, without the system going down. It arrives in three shapes: a step change after a deploy, a slow creep over weeks and a tail that gets worse while the median looks fine.

Each shape needs a different check. A single "p95 above X" alert catches only the second and third at their worst.

What to measure

Start from the request path. Google's SRE book recommends tracking the latency of failed requests separately from successful ones, since a slow error is worse than a fast one. If you filter errors out of your latency chart, a failure mode that times out will look like an improvement.

OpenTelemetry defines http.server.request.duration as a stable histogram metric in seconds, and http.client.request.duration for outgoing calls, so most instrumented services already emit what you need. Also measure queue wait time, query time and startup time, which are where creep usually starts.

An example request log line with the fields that make analysis possible:

2026-11-08T13:41:07Z msg="request" route=/api/search method=GET status=200 duration_ms=1840 db_ms=1522 db_queries=412 version=4.18.0

The 412 queries explain the 1.8 seconds. Logging query counts per request makes N+1 behavior visible.

Detect it step by step

1. Use histograms, not averages. The Prometheus docs point out that quantiles precomputed by instances cannot be aggregated across replicas, while histogram buckets can be. An example query for a per-route 95th percentile:

# Example
histogram_quantile(0.95,
  sum by (le, route) (rate(http_request_duration_seconds_bucket[5m])))

2. Detect step changes by comparing with the same window a week earlier, which controls for time of day and weekday:

# Example: ratio of current p95 to the p95 one week ago, per route
histogram_quantile(0.95, sum by (le, route) (rate(http_request_duration_seconds_bucket[15m])))
/
histogram_quantile(0.95, sum by (le, route) (rate(http_request_duration_seconds_bucket[15m] offset 1w)))

3. Detect creep with a longer view. Compare the weekly median or p95 of each route over the last eight weeks and flag a sustained positive slope. A route that gets 3 to 5 percent slower each week never trips a threshold but doubles in four months.
4. Detect tail problems by comparing p99 to p50 per route. A widening gap with a flat median means a minority of requests got slower: a cold cache, a hot shard or one slow dependency.
5. Tie changes to causes. Group by release version, by host and by downstream dependency to find the dimension where the shift lives.

Thresholds and pitfalls

Use a ratio and a floor, and require the condition to hold for several minutes. For example, p95 at least 30 percent above last week and above an absolute value users can notice, for 10 minutes. Treat those figures as starting points and tune them per route.

• Low traffic makes percentiles unstable. Require a minimum request count in the window.
• Histogram quantiles are interpolated estimates, and the Prometheus docs say accuracy improves with higher bucket resolution. Put a bucket boundary at your latency target so the comparison you care about is exact.
• Traffic mix changes the aggregate. A route that gets slower may be offset by growth in a fast route. Always alert per route.
• Batch jobs, backups and cache warmups create regular slowdowns. The week-over-week comparison absorbs them, and a day-over-day comparison does not.
• Client-side slowness is a separate measurement. Server timing will not show it, so see web vitals regressions.

What to do when it fires

Find the dimension first. If it lines up with a release, compare the diff, or roll back. If it lines up with one dependency, check that vendor's status and your timeout and retry settings. If it lines up with load, look at resource headroom in the capacity risks guide.

For specific shapes, read the guides on latency spikes, tail latency and latency drift. To get histograms flowing, see Prometheus and OpenTelemetry. A related guide on frontend monitoring covers what happens in the browser.