How High Cardinality Hurts Prometheus

Prometheus is cheap right up to the point where it is not, and the line between the two is almost never the number of samples. A Prometheus server can ingest a million samples a second on modest hardware. What runs it out of memory, makes restarts take twenty minutes, and turns a Grafana dashboard into a wall of duplicate tiles is the number of distinct time series it has to keep track of.

"High cardinality" is the name for that, and it is usually explained as a rule: don't put user IDs in labels. That rule is correct and it is not enough. Once you understand how the storage engine actually holds a series, you can predict which labels will hurt, how much, and when, and you can design metrics for a fleet that churns thousands of Pods a day without guessing.

So this starts at the bottom.

Part 1: What Prometheus stores

Prometheus pulls. On a schedule, usually every 15 to 60 seconds, it fetches a text page from each target. The page is a list of lines like:

http_requests_total{method="GET", status="200", engine="chrome"} 18342

Three parts. A metric name. A set of label name and value pairs. A number. Prometheus stamps the number with the scrape time and stores it.

Cardinality is the count of distinct series. The arithmetic is multiplication: a metric with a method label of 4 values, a status label of 5 values and an engine label of 7 values can have up to 140 series. Add a label with a thousand values and it is 140,000. Add one whose values never repeat, like a request ID, and it is unbounded.

That is the model most explanations stop at. The reason it matters is in the next part.

Part 2: How the storage engine holds a series

Prometheus's storage is its own time series database, usually just called the TSDB. It has two halves: a recent, in-memory half called the head, and a set of immutable blocks on disk for everything older. A sample's life goes head first, then block, then deletion by retention.

The head block

When a sample arrives for a series, Prometheus finds that series' in-memory record and appends the sample to its current chunk. A chunk is a small compressed buffer of consecutive samples for one series. Prometheus uses a delta-of-delta and XOR encoding, the design usually referred to as Gorilla compression, which is why the storage documentation says it stores an average of only one to two bytes per sample. Samples are cheap. That is the first half of the story.

Each series also has an in-memory record that holds its label set, references to its chunks, and the current open chunk. When a chunk fills, roughly 120 samples by default in the current implementation, it is written to disk and memory-mapped, and only a reference stays in RAM. So the per-series memory is not mainly samples. It is the labels, the index entries, the chunk references and the bookkeeping. That cost is paid once per series regardless of how many samples it has. Series are expensive. That is the second half.

The write-ahead log

The head lives in memory, so a crash would lose it. To prevent that, every new series and every batch of samples is also appended to a write-ahead log on disk, in segments of 128 MiB by default. The log has record types for series, which carry the full label set, and for samples, which carry a series reference and values.

On restart, Prometheus replays the log to rebuild the head. Every series record must be read and every series re-created in memory before the server can serve queries. This is the mechanism behind slow restarts on high-cardinality instances: replay time is roughly proportional to the number of series that existed in the head window, not to the number of samples.

Compaction to blocks

The head keeps a window of recent data, two hours by default. When the head spans one and a half times that, three hours, the oldest two hours are written out as a persistent block and the head is truncated. At that truncation, series that have no chunks left in the head are garbage collected from its index, and the write-ahead log is truncated and checkpointed to match.

A block on disk is a directory: the chunk files, a meta.json describing its time range, a tombstones file for deletions, and an index. The index is what makes queries possible and what makes cardinality expensive at rest. It holds a symbol table of every distinct string in every label pair, a series section listing every series with its labels and chunk references, and postings lists, which for every label pair record which series carry it.

Blocks are periodically compacted together into larger blocks covering longer ranges, and retention deletes whole blocks once they age out.

Part 3: Where cardinality actually costs you

Now put the two halves together. Samples cost one to two bytes and compress beautifully. Series cost memory, index entries, log records and replay time, and none of that compresses. Five specific places hurt.

Head memory. Every series alive in the head window has an in-memory record, plus its entry in the head's own index and postings. Double the series, roughly double that memory. This is the number behind most Prometheus out-of-memory restarts.

The two-to-three-hour tail. A series is only removed from the head at the truncation after its last sample, which is between two and three hours later. So the head does not hold the series that exist now. It holds every series that existed at any point in the last two to three hours. On a static system those are the same set. On a system where series come and go, they are not, and that difference is the whole problem in the worked example below.

Restart time. Write-ahead log replay walks every series record. An instance with five million series in its window can take many minutes to come back, during which alerts are blind.

Query cost. A query like rate(http_requests_total[5m]) has to find every matching series through the postings lists, then decode chunks for each. A thousand series is fast. A million is not, and a dashboard with ten such panels multiplies it. Queries that look innocent are expensive because of the series behind them, not the math.

Money, on managed platforms. Google Managed Prometheus, Amazon Managed Prometheus, Grafana Cloud and similar services bill or limit on active series, because the same cost structure applies to them. Cardinality is a line item.

Part 4: Which labels explode

With the mechanism in hand, the dangerous labels are predictable. Any label whose values are unbounded or short-lived multiplies series by churn:

Label	Why it explodes	Where it belongs instead
Pod name on an autoscaling fleet	Unique forever, replaced constantly	Logs, with the Pod name in the line
Request or trace ID	Unique per request	Traces or logs
User or customer ID	Unbounded	Logs, or an aggregate metric
Raw URL path	Includes IDs and query strings	Route template label
Client IP	Effectively unbounded	Logs
Histogram `le` buckets	Multiplies every other label by bucket count	Fewer buckets, or native histograms

The histogram row deserves a sentence. A histogram exposes one series per bucket through the le label. Twelve buckets on a metric that already carries engine, status and pod is twelve times whatever that product was. Histograms are the most common way a reasonable label set becomes an unreasonable series count.

The same metric at four scales

One request-duration histogram, twelve buckets, on a service with seven engines and five status values. Watch what each added label does:

Labels on the metric	Series	Change
`engine`, `status`	35	baseline
+ `le` (12 buckets)	420	x12
+ `pod`, 40 stable replicas	16,800	x40
+ `pod`, autoscaling fleet, ~3,600 Pods per generation	1,512,000	x3,600
+ three generations inside the head window	~4,500,000	x3 again

Nothing in that table is a mistake anyone makes deliberately. Each step is one reasonable-sounding decision: measure latency properly, keep per-Pod detail, scale the fleet, and the last row is not a decision at all, it is the storage engine holding what you already created. Four and a half million series from one metric is where a Prometheus falls over, and the fix is visible in the table: remove pod and you are back to 420.

Part 5: Measuring it before it measures you

Prometheus tells you exactly where cardinality lives, if you ask.

The head series count is a metric, prometheus_tsdb_head_series, and its trend over a day is the single most useful graph for capacity. A sawtooth that climbs for three hours and drops is normal head truncation. A line that climbs and does not drop is a leak.

The TSDB stats endpoint at /api/v1/status/tsdb breaks it down: series count by metric name, value count by label name, memory by label name, and series count by label pair, with a limit parameter that defaults to ten entries each and goes up to ten thousand. In practice, the label-value-count-by-label-name list identifies the culprit in one look. A label with forty thousand values is the problem, whatever it is called.

Four queries answer almost every cardinality question. Run them in order.

# 1. How many series exist right now, and is the trend flat?
prometheus_tsdb_head_series

# 2. Which metrics own those series? Usually one or two dominate.
topk(10, count by (__name__)({__name__!=""}))

# 3. Churn: distinct values a label took over the window, versus now.
count(count by (pod) (count_over_time(up[6h])))
count(count by (pod) (up))

# 4. Which targets are expensive to scrape?
topk(10, scrape_samples_scraped)

Query 3 is the one that surprises people. On a static deployment the two numbers match. On an autoscaling fleet the first can be an order of magnitude larger than the second, and that ratio is your churn multiplier, the same one that appeared in the last row of the table above.

Query 2 is where to start if the head series count is already too high, because cardinality is almost never spread evenly. One metric with an unfortunate label usually accounts for most of it, and the TSDB stats endpoint will name the label.

Part 6: Controlling it

There are four levers, and they act at different points in the pipeline.

Design labels around the question people ask. For every label on a high-churn workload, write down who queries by it. On the fleet in the worked example, the on-call question is always "what failed on which engine", never "what failed on which Pod", so engine stays and pod goes. A label nobody queries by is pure cost. This is the lever that fixes the problem; the other three enforce the fix.

Drop at scrape time with relabeling. metric_relabel_configs runs after the scrape and before storage. An action of labeldrop on a regex of label names removes the label before the series is ever created; an action of drop on a metric name discards whole metrics. Because it happens before ingestion, the series never touch the head, the log, or the index. This is the cheapest possible fix and the least reversible, since the data was never stored.

metric_relabel_configs:
  # Remove the churning labels from every metric on this job.
  - regex: 'pod|instance|container_id'
    action: labeldrop

  # Keep pod on the handful of metrics where a single replica matters.
  - source_labels: [__name__]
    regex: 'kube_pod_container_status_restarts_total|container_oom_events_total'
    action: keep

Order matters and the two rules above are illustrative rather than a drop-in: labeldrop applies to every metric in the job, so a real config splits the "keep pod" metrics into their own scrape job rather than trying to express both rules in one list. That split is the practical shape of the mixed-fleet advice at the end of this article.

Whatever you write, verify it on a live target with /api/v1/targets/metadata or by curling the target and comparing, because a regex that matches nothing fails silently and looks exactly like a regex that works.

Guard with limits. Prometheus offers per-scrape limits: sample_limit on samples per scrape, label_limit on labels per sample, and length limits on label names and values. They default to zero, meaning off. When a scrape exceeds a limit, the entire scrape is treated as failed, which is deliberately harsh. They are a guardrail against a deploy that suddenly adds a request ID label, not a tuning tool.

Aggregate with recording rules, then drop the raw series. A recording rule that sums a metric by engine into a new metric gives dashboards and alerts a low-cardinality series to read. On its own it does not reduce cost, because the raw series still exist. Paired with relabeling that drops the raw high-cardinality version, it does. The order matters: build the aggregate, repoint the consumers, then drop the source.

Part 7: A worked example from an autoscaling fleet

The fleet is a headless browser platform on Kubernetes: workers that scale on queue depth from a few dozen nodes to around 600, on Spot capacity, so Pods are replaced constantly by scale events, preemptions and deploys.

The visible symptom was a dashboard showing the same metric three times. A self-hosted Prometheus had scraped one Service address, which collapsed all Pods behind a single series per metric. Moving to a managed Prometheus that scrapes each Pod individually meant every restart started a new series carrying the new Pod name. Panels that plotted raw series drew every generation still inside the head window. Thirty-one series in a six-hour window where the correct number was seven, one per subscription.

The arithmetic from Part 3 says why. At peak, roughly 3,600 worker Pods. Say each exposes fifty series that carry the pod label; that is 180,000 series per generation. Three scale cycles inside the two-to-three-hour head window is three generations, so around 540,000 series in memory for a fleet whose steady-state count is a fraction of that. Those numbers are illustrative, but the shape is exactly what the head-series graph showed.

The fix followed the levers in order:

Ask the question. Who queries by Pod? Nobody. Who queries by engine and subscription? Everyone.
Patch the dashboards. max by (subscription) on gauges and rate() on counters collapsed 31 series to 7 immediately.
Drop the label. labeldrop on pod at scrape time, so the churn never reaches the store.
Keep the detail where it is cheap. Per-Pod investigation goes to logs, where a Pod name is text inside a line rather than a dimension that creates a series.

Label	Who asks about it	Distinct values	Kept on metrics?
`engine`	On-call, dashboards, alerts	7	Yes
`subscription`	Autoscaling, backlog panels	7	Yes
`status`	Error-rate alerts	5	Yes
`pod`	Nobody, in practice	Thousands per day	No
`node`	Occasionally	Hundreds per day	No, use events

Part 8: What you give up, and when that matters

Dropping a label loses the ability to see one outlier along that dimension in the metrics. With pod gone, a single degraded worker shows up as a slightly worse engine-level percentile and nothing more.

On a Spot fleet where a worker lives minutes to hours, that does not matter: the bad Pod is replaced before anyone could act on it, and the bad-node case surfaces through node events and restart counters, which are kept at low cardinality. On a stateful service with a dozen long-lived Pods, it matters a great deal, because one degraded replica can sit there for days. There, pod has a name in the "who asks" column and belongs on the metrics.

The mixed-fleet answer is to keep pod on a small set of high-signal metrics, restarts, out-of-memory kills, readiness, and drop it from everything else. The cost of that mitigation is two relabel rules to maintain and two places to look during an investigation. Cheap, compared to the alternative.

What to take from this

Prometheus stores samples almost for free and pays per series: in head memory, in index and postings entries, in write-ahead log records, and in replay time on restart. Cardinality is the series count, and on anything that churns, the series count in the head is the live count multiplied by the generations inside a two-to-three-hour window. Labels whose values are unbounded or short-lived are the multipliers. Measure with the head-series metric and the TSDB stats endpoint, decide per label whether a human queries by it, drop the rest before ingestion, guard with limits, and put unique identifiers in logs where they cost nothing.

Continue learning

If the head block now makes sense, the next thing to understand is the full compaction and retention lifecycle, because it explains disk usage patterns and why deleting a metric does not free space immediately. After that, recording rules and relabeling as a pair, since one shapes what is stored and the other what is queried and most teams reach for the wrong one first. Then remote write, which is where managed platforms and long-term storage sit, and where the same series arithmetic turns into a bill.

Sources and further reading

Everything in Parts 2 and 3 comes from these, and they are the right order to read them in.

Prometheus: Storage. The official description of the on-disk layout, the two-hour blocks, the WAL, and the capacity-planning formula.
Ganesh Vernekar's TSDB series, written as a Prometheus maintainer: the Head block, WAL and checkpoint, and persistent blocks and the index. The clearest explanation of where a series lives and what a restart replays.
Prometheus TSDB on-disk format docs and the TSDB README in the Prometheus repository. The authoritative byte-level description of the index, chunks, WAL and tombstones, maintained alongside the code.
Gorilla: A Fast, Scalable, In-Memory Time Series Database, VLDB 2015. The compression scheme behind one to two bytes per sample.
Prometheus: Configuration. metric_relabel_configs, sample_limit, label_limit and the length limits, with their exact semantics.
Prometheus: HTTP API, TSDB stats. The endpoint that finds your worst label in one call.
Prometheus: Recording rules and Instrumentation best practices. The official guidance on aggregation and on not overusing labels.
Prometheus: Metric types. Histograms, the le label, and native histograms.

FAQ

Is the pod label always wrong? No. It is wrong on high-churn, short-lived workloads where nobody queries by it, and right on long-lived stateful Pods where one replica can be the problem for days.

Why does memory not drop when I scale the fleet down? Because dead series stay in the head until the next truncation after their last sample, two to three hours. Watch the head-series metric over a few hours before concluding there is a leak.

Do sample limits reduce cardinality? They cap it by failing scrapes that exceed the limit. That protects the server but drops all data from that target, so treat them as an alarm, and fix the labels.

Does a bigger Prometheus fix this? It postpones it. The cost is per series and churn keeps generating series. Fix the multipliers first, then size the server for the count that remains.