The Four Golden Signals of Monitoring

Last verified on Nov 8, 2026.

Dashboards grow until nobody can read them. The golden signals are Google's answer to the question of where to start: four measurements that, taken together, tell you whether a user-facing service is healthy.

The Four Signals

The SRE book lists them as latency, traffic, errors, and saturation, and says to focus on these four if you can only measure four metrics. It also says that if you measure all four and page a human when one is problematic, or nearly problematic for saturation, the service will be at least decently covered.

Latency

The time it takes to service a request. The SRE book makes a point teams skip: separate the latency of successful requests from the latency of failures. A 500 caused by a lost database connection may return in a millisecond. Mixing those in drags the average down while users suffer. On the other hand, a slow error is worse than a fast one, so track error latency instead of simply dropping it.

Use percentiles, not averages. The same chapter shows how a service with a 100 ms average at 1,000 requests per second can have 1% of requests taking 5 seconds. The latency percentiles page covers how to compute them.

Traffic

How much demand is placed on the system, measured in a unit that fits the service. For a web service that is usually HTTP requests per second, perhaps split between static and dynamic content. The book's other examples: network I/O or concurrent sessions for audio streaming, and transactions and retrievals per second for a key-value store. Traffic is the denominator for the other signals. A change in error count means little until you know whether traffic also changed.

Errors

The rate of requests that fail, in three ways:

• Explicitly, such as HTTP 500s.
• Implicitly, such as a 200 response that carries the wrong content.
• By policy, such as "if you committed to one-second responses, any request over one second is an error".

The book points out that catching 500s at the load balancer finds completely failed requests, while only end-to-end tests catch wrong content. Your error signal is only as good as the failure modes it can see. The policy case is the bridge to an SLI.

Saturation

How full the service is, emphasizing the most constrained resource: memory for a memory-bound system, I/O for an I/O-bound one. The book warns that many systems degrade before they reach 100% utilization, so a utilization target is essential. It adds that latency increases are often a leading indicator, and that the 99th percentile over a small window such as one minute can be an early warning. Saturation also covers prediction, such as a disk that will fill in four hours.

Querying Them

Prometheus-style, with assumed metric names http_requests_total and http_request_duration_seconds:

# Traffic: requests per second
sum(rate(http_requests_total[5m]))

# Errors: share of requests returning 5xx
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

# Latency: 95th percentile across all instances
histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

Prometheus's histogram guide shows this aggregation. Summing bucket rates before taking the quantile is what makes it safe to combine instances.

From access logs in SQL, with request_logs(ts, status, duration_ms):

SELECT date_trunc('minute', ts) AS minute,
       count(*) / 60.0 AS traffic_rps,
       count(*) FILTER (WHERE status >= 500)::float / count(*) AS error_ratio,
       percentile_cont(0.95) WITHIN GROUP (ORDER BY duration_ms) FILTER (WHERE status < 500) AS p95_ok_ms
FROM request_logs
WHERE ts >= now() - interval '1 hour'
GROUP BY 1
ORDER BY 1;

Saturation has no log field. It comes from resource metrics: CPU, memory, connection pool use, queue depth. See capacity risks for which signs show up in logs first.

Not Every Service Is a Web Service

The signals were written for request-driven systems. Prometheus's instrumentation guidance splits services into three types and says what matters for each:

• Online-serving: queries, errors, and latency, which line up with the first three signals.
• Offline processing: items in, in progress, and out, plus the last time something was processed. It also suggests a heartbeat item that travels through every stage.
• Batch jobs: the last time the job succeeded, the stage durations, and the overall runtime.

For a pipeline, "traffic" is items arriving and "latency" is time from entry to exit. Adapt the names to the system, keeping the questions.

Mistakes

• Averaging. Look at the tail. See tail latency.
• Paging on causes. The book separates symptoms from causes: page on what users feel, and keep cause-level metrics for debugging.
• Skipping saturation. It is the signal aimed at prediction, as the book's "disk will fill in four hours" example shows. The other three tell you about trouble that has already started.

A related set of ideas, with slightly different scope, is on the USE and RED methods page. For the signals in practice, see latency spikes and performance degradation.

Sazabi is an AI-native observability and security platform. Sazabi ingests OpenTelemetry metrics (Prometheus metrics through an OpenTelemetry Collector) and OTLP traces through the same intake as your logs. It stores each metric data point and each span as a log record that its agent queries alongside your logs, so there is no separate metrics store to run.