Capacity Monitoring: Catch Resource Limits Early

Last verified on Nov 8, 2026.

A capacity risk is a resource heading toward a limit: a disk filling, memory climbing, a connection pool running dry, a cloud quota almost used up. The failure arrives later and looks unrelated. A full disk shows up as write errors. A drained pool shows up as latency on every endpoint that touches the database.

Capacity monitoring is about catching the trend while you still have hours or days to act.

What to watch

Google's SRE book names four golden signals: latency, traffic, errors and saturation. It defines saturation as how full your service is, emphasizing the most constrained resources, and notes that many systems degrade in performance before they reach 100 percent. That second point is why alerting at the limit is too late.

Resources worth tracking on any stack:

• Disk space and inodes on every volume that takes writes.
• Memory against its limit, per process or container.
• Connection pools and thread pools: in use, waiting, timeouts.
• Queue depth and age of the oldest item.
• Cloud quotas and API rate limits.
• Certificate, token and domain expiry dates.

Evidence in logs and metrics

Capacity trouble leaves early warnings before the hard failure. Example log lines from an application whose pool is filling:

2026-11-08T11:30:04Z level=warn msg="db pool wait" waited_ms=410 in_use=48 max=50
2026-11-08T11:30:09Z level=warn msg="db pool wait" waited_ms=1190 in_use=50 max=50
2026-11-08T11:30:31Z level=error msg="db pool timeout" waited_ms=5000 max=50

Waiting time appears minutes before timeouts. If you log pool waits, you can alert on them. If you only log timeouts, you find out at the end.

In Kubernetes, memory limits are enforced by the kernel through out of memory kills. The Kubernetes docs say a container that exceeds its limit may be terminated, and that this happens only when the kernel detects memory pressure, so an over-allocating container is not always killed right away. Restarts with an OOM reason are the evidence; the crash loops guide covers them.

Detect it step by step

1. Record current usage and the limit for each resource, so you can compute headroom.
2. Alert on headroom and on trend together. Prometheus provides predict_linear(), which uses simple linear regression to predict a series value some seconds ahead and needs at least two samples. An example query for a disk metric (the metric name here is a placeholder):

# Example: fire if the volume is predicted to be full within 24 hours
predict_linear(disk_free_bytes{mount="/data"}[6h], 24 * 3600) < 0

3. Alert on waiting, not only on failing. Pool wait time, queue age and retry counts rise before errors do.
4. Track quotas against their usage. AWS lets you create CloudWatch alarms that notify you when you are close to a Service Quotas threshold, which gives time to request an increase.
5. Track expiry dates as numbers of days remaining and alert at several lead times. See expiring certificates.

Thresholds and pitfalls

Pick thresholds from consequences. A volume that grows 1 percent a day gives you weeks at 80 percent full. A volume that grows 5 percent an hour does not. Page on time to exhaustion and ticket on percentage.

• Linear prediction misleads on sawtooth data. A disk that fills and is rotated every night looks like it is about to overflow if the window ends at the peak. Use a window that covers whole cycles, or predict from daily minimums.
• A language runtime may hold memory it could free. Look at usage after garbage collection or at the container limit, not at the raw number alone.
• Spiky load can briefly saturate a pool without any risk. Require the condition for several minutes.
• Limits that scale automatically need a different check: watch the ceiling of the autoscaler, not the current size.
• Very low-traffic systems produce noisy ratios. Add an absolute floor.

What to do when it fires

Work out whether it is growth or a leak. A steady slope that matches traffic means you need more capacity or a retention change. A slope that persists when traffic falls is a leak or a runaway writer, and more capacity only delays it.

Short term: free space, raise a limit, shed load or scale out. Long term: add the missing headroom alert and a runbook entry. Slow performance often arrives first, so pair this with the performance degradation guide. To collect resource metrics from your stack, see the Prometheus and OpenTelemetry integrations, and AWS for cloud quotas.