p99 Latency: How to Detect Tail Latency Before Users Complain

Last verified on Nov 8, 2026.

Tail latency is what the slowest few percent of requests experience. A dashboard that shows the median or the average can stay green while one request in a hundred takes ten times longer. Those users reload, retry and open tickets, and the retries add load that makes the tail worse.

The Google SRE book warns against designing monitoring around mean latency and recommends counting requests in latency buckets so you can tell a slow average from a very slow tail. It also notes that the 99th percentile over a short window, such as one minute, is an early signal of saturation. The paper The Tail at Scale adds that the tail is harder to keep low as systems and utilization grow, and that brief slow episodes can dominate overall performance at scale.

What the evidence looks like

The classic shape is a flat median with a climbing p99. This example shows two series for one route over an hour:

route=/api/search   p50=84ms    p99=310ms     (09:00)
route=/api/search   p50=88ms    p99=1,240ms   (10:00)

Request logs show the same thing from raw rows. A few requests carry a long duration_ms, and they often share a cause: one customer, one shard, one upstream call or one cold start.

2026-11-08T10:14:03Z GET /api/search status=200 duration_ms=2210 db_ms=2034 tenant=t_912

How to detect it

1. Record request duration as a histogram, not an average. Use buckets with a boundary at each service level objective you care about, such as 300 ms, so the percentile near that value is accurate.
2. Label by route and status class. Keep labels bounded. A per-user label will cause its own problem, covered in cardinality jumps.
3. Compute p50 and p99 over the same window and watch their ratio.

In PromQL, the histogram_quantile function reads the buckets:

# p99 per route over 5 minutes
histogram_quantile(0.99,
  sum by (route, le) (rate(http_request_duration_seconds_bucket[5m])))

# p99 divided by p50, per route
histogram_quantile(0.99, sum by (route, le) (rate(http_request_duration_seconds_bucket[5m])))
/
histogram_quantile(0.50, sum by (route, le) (rate(http_request_duration_seconds_bucket[5m])))

Include le in the sum by clause. The function needs it to rebuild the distribution. If you only have logs, compute it in SQL:

SELECT route,
       percentile_cont(0.5)  WITHIN GROUP (ORDER BY duration_ms) AS p50,
       percentile_cont(0.99) WITHIN GROUP (ORDER BY duration_ms) AS p99,
       count(*) AS requests
FROM request_log
WHERE ts > now() - interval '15 minutes'
GROUP BY route
HAVING count(*) >= 500
ORDER BY p99 DESC;

Once a route is flagged, split its slow requests by tenant, host, region, upstream and status. A tail problem concentrated in one value points to the cause quickly. Spread evenly across all values, it points to saturation, garbage collection or a shared dependency.

Thresholds and pitfalls

Use two conditions. Alert when p99 crosses an absolute limit that matters to users, and separately when p99 divided by p50 rises well above its usual value for that route. The ratio catches a tail that worsens while the absolute number still looks acceptable. In Prometheus, put a for duration on the rule so Prometheus holds the alert in a pending state before it fires. The alerting rules docs describe how it works.

Common mistakes:

• Averaging p99 across servers. Percentiles do not combine by averaging, and Prometheus notes that summary quantiles cannot be aggregated across instances. Sum histogram buckets instead.
• Low traffic. With few requests, a couple of slow ones set the p99. Require a minimum request count, as the SQL above does.
• Bucket layout. A percentile is only as precise as the bucket it falls in. If your top bucket is 10 seconds, every slower request looks like 10 seconds.
• Time-of-day effects. Nightly jobs, cache expiry and batch traffic shift the tail on a schedule. Compare with the same hour yesterday or last week.
• Client-side timing. Server duration leaves out queueing and network time. Compare with a load balancer or client measurement if the numbers disagree with user reports.

What to do when it fires

Check recent deploys and configuration changes first. Then look for the shared cause: a slow query, a saturated connection pool, a retry storm or a noisy neighbor. Pull a trace for one of the slowest requests and read its longest span. If the tail follows load, add capacity or shed work. If it follows one customer or key, isolate that traffic.

Related guides: latency drift covers slow, steady change, and capacity risks covers running out of headroom. If you already run Prometheus, the Prometheus page explains how to forward its metrics, and OpenTelemetry covers sending traces.