High Cardinality: How to Detect a Cardinality Explosion Early

Last verified on Nov 8, 2026.

Cardinality is the number of distinct values a field takes. A status label has a handful. A user_id label has millions. In a metrics system, every unique combination of label values becomes its own time series, so one new unbounded label can multiply storage, memory and query time overnight. The Prometheus naming guide puts it plainly: do not use labels to store dimensions with many different values, such as user IDs or email addresses.

The problem is the jump, not the level. A system sized for 400,000 series that suddenly holds 4 million is in trouble even if both numbers are individually plausible. In logs and traces, the same jump affects indexes and cost, and it often signals a bug: a route that now logs a raw URL with an ID in it, or a cache that stores a key per request.

What the evidence looks like

On the metrics side, you will see series counts and new-series rates climb together. On the data side, you will see a field whose distinct values per minute leap. This example shows a log line that introduces the problem, with a request ID leaking into a metric label:

2026-11-08T08:31:07Z level=info msg="request done" route=/api/items/8841207 status=200 cache_key=items:8841207:v3

If the framework used the raw path as the route label instead of the route definition, every item ID becomes a new series. The cache key field shows the same shape: a distinct value on almost every line.

How to detect it

1. Track total series and the rate of new series. In Prometheus, the TSDB exposes prometheus_tsdb_head_series for the current head series and prometheus_tsdb_head_series_created_total for series created. Each scrape also reports scrape_series_added, the approximate number of new series in that scrape.
2. Alert on growth rather than on a fixed ceiling.

# Head series grew more than 30% in the last hour
prometheus_tsdb_head_series
  > 1.3 * (prometheus_tsdb_head_series offset 1h)

# Targets adding many new series per scrape
sum by (job) (scrape_series_added) > 5000

3. When it fires, find the culprit. The /api/v1/status/tsdb endpoint returns seriesCountByMetricName, labelValueCountByLabelName and seriesCountByLabelPair. The first names the metric, the second names the label with too many values.

curl -s 'http://localhost:9090/api/v1/status/tsdb?limit=20'

4. For logs and events, count distinct values per field and compare with the previous window.

SELECT field,
       count(DISTINCT value) AS distinct_now,
       lag(count(DISTINCT value)) OVER (PARTITION BY field ORDER BY bucket) AS distinct_before
FROM (
  SELECT date_trunc('hour', ts) AS bucket, k AS field, v AS value
  FROM log_fields   -- one row per field and value on each log line
  WHERE ts > now() - interval '2 hours'
) t
GROUP BY bucket, field
ORDER BY distinct_now DESC;

Add guardrails at the source. Prometheus scrape configs accept sample_limit and label_limit, which fail a scrape that exceeds them (zero means no limit). OpenTelemetry SDKs support a cardinality limit with a default of 2000 and report measurements beyond it under the attribute otel.metric.overflow. If you see that attribute in your data, the SDK is already hiding a jump from you.

Thresholds and pitfalls

Growth percentage over an hour is the most useful rule, because it adapts to system size. Pair it with a ceiling tied to your real capacity so a slow, steady climb still gets noticed.

Expect these false alarms and misses:

• Deploys. Pod or instance labels change on every rollout, which raises series count for a short time as old series go stale. Wait out the churn period before alerting, or exclude instance-level labels from the rule.
• New customers. A real increase in tenants raises legitimate cardinality. Check the tenant count before debugging.
• Slow leaks. A field that adds a few hundred values a day never trips a percentage rule. Review the top labels by value count weekly.
• Histograms. Each bucket is a series, so adding a route label to a histogram multiplies the cost by the number of buckets.

What to do when it fires

Identify the metric and label, then find the commit or config change that introduced it. Fix it at the source: replace raw paths with the route definition, drop request-scoped IDs from labels and send them as log or trace fields instead. Then remove the damage. Drop the label with relabeling or delete the bad series, then confirm the new-series rate returns to normal.

Related guides: tail latency shows why labels must stay bounded when you chart percentiles, and cloud spend anomalies covers the bill that a cardinality jump can inflate. The Prometheus and OpenTelemetry pages cover forwarding the underlying data.