Cost Monitoring: Find Waste and Telemetry Blind Spots

Last verified on Nov 8, 2026.

Efficiency problems waste money or leave you unable to see your own systems. They rarely cause outages, so they do not page anyone, and they compound. A retry loop adds cost every hour it runs. A service that stopped logging is a hole in every investigation you do later.

This guide treats cost monitoring as two halves: spend that grew without a reason, and telemetry that stopped telling the truth. The second half is often the cheaper one to fix.

Spend that grew without a reason

Five causes account for most surprises, and each has its own detailed guide:

• Overall cloud bills drifting up: cloud spend anomalies.
• Model usage tied to tokens, prompts and agents: LLM spend spikes.
• Metric labels that multiply stored series: cardinality jumps.
• Log volume that jumps after a deploy, often a verbose line in a loop.
• Retry storms, where failed calls trigger more calls and more cost.

The general method for the cost side is in cost anomalies. Here is the quickest check you can build today: log bytes per service per day, compared with the previous week.

-- Example: services whose log volume grew at least 2x week over week
WITH daily AS (
  SELECT service, date_trunc('day', ts) AS day, sum(length(message)) AS bytes
  FROM logs
  WHERE ts >= current_date - 8
  GROUP BY 1, 2
)
SELECT t.service, t.bytes AS yesterday_bytes, w.bytes AS last_week_bytes,
       round(t.bytes::numeric / nullif(w.bytes, 0), 2) AS growth
FROM daily t
JOIN daily w ON w.service = t.service AND w.day = t.day - interval '7 days'
WHERE t.day = current_date - 1
  AND t.bytes > 2 * w.bytes
ORDER BY t.bytes - w.bytes DESC;

If one service jumped, group its lines by message text to find the culprit. Usually it is a handful of lines.

Telemetry that stopped telling the truth

Observability gaps are the second half. They are failures of your monitoring, not of your product, and they hide until you need the data.

A service went quiet. Silence is not health. In Prometheus, the up series is 1 if the instance is reachable and 0 if the scrape failed, and absent() returns a one-element vector when no series match, which the docs note is useful for alerting when no series exist. absent_over_time() does the same for a range. Example:

# Example: no series for the checkout job in the last 10 minutes
absent_over_time(up{job="checkout"}[10m])

For logs, compare the lines per service in the last 15 minutes with the same window one week ago, and alert when a service that normally logs falls to zero.

Unreadable format. A deploy changes a log format from JSON to plain text, or a new field changes type, and your parsing silently drops the structure. Watch the share of lines that fail to parse per source. A jump from 0.1 percent to 30 percent is a format change.

Blind spots. A service, region or queue was never instrumented, so nothing alerts in it. Keep an inventory of what you run and compare it with the services that appear in telemetry.

Clock skew. The OpenTelemetry log data model separates Timestamp, the time at the source clock, from ObservedTimestamp, the time the collection system saw the event. A gap between them that grows, or source timestamps in the future, mean a host clock is wrong. That breaks ordering, joins and every alert with a time window. See clock skew.

Sensitive data in logs. Secrets and personal data stored in logs are an exposure and a compliance cost. See the exposure guide.

Thresholds and pitfalls

• Intentional changes cause most cost jumps. Tie checks to a deploy or an event annotation so a launch does not read as waste.
• For quiet-service checks, set the window to several times the normal gap between logs. A batch service that runs hourly will look silent for most of every hour.
• Cheap data is not free data. Dropping logs to save money creates the blind spot described above, so reduce volume by removing noise, not by sampling errors.
• Check for the alert itself failing. A rule that cannot evaluate is silent in the same way.

What to do when it fires

For a cost jump, find the producer, fix it at the source and add a budget or volume alert on that service. For a gap, restore the data first, then add the check that would have caught it. Revisit costs each month at the unit level, cost per request or per customer, rather than only the total. For collecting metrics and logs consistently, see OpenTelemetry and Prometheus.