Summary#

Prometheus metric cardinality failures usually come from “too many active time series” rather than from raw metric count alone. The main triggers are unbounded or high-churn labels, classic histogram bucket multiplication, careless exemplar usage, and remote-write pipelines that forward every raw series to a paid or capacity-limited backend. A safe design treats labels as a bounded schema, budgets histogram buckets deliberately, uses exemplars selectively, and applies drop/keep/aggregation controls before remote-write cost becomes an incident.

Key Points#

• Cardinality is the product of metric names and label-value combinations.
• In Prometheus, each unique combination of metric name and labels is a separate time series.
• A label such as user_id, session_id, request_id, trace_id, pod_uid, full URL path, container hash, or timestamp-like value can create an unbounded number of series.

• The failure mode is not only storage growth; high series count also affects memory, index size, WAL pressure, compaction, query fan-out, and remote-write volume.

• High-churn labels are worse than merely “many labels.”

• A bounded label with many stable values is easier to reason about than a label whose values constantly appear and disappear.
• Kubernetes-style ephemeral dimensions—pod names, pod UIDs, job IDs, short-lived containers—can cause rapid series creation and deletion.
• High churn stresses the TSDB head block and WAL because Prometheus must continuously create, index, and later retire series.

• Prefer stable service-level labels such as service, namespace, cluster, route, method, status_class, or controlled status_code.

• Avoid labels that encode identity or raw request detail.

• Bad examples:
  • user_id
  • email
  • session_id
  • request_id
  • trace_id
  • raw path="/api/users/123/orders/456"
  • error_message

• Safer replacements:

  • templated route: /api/users/{user_id}/orders/{order_id}
  • status class: 2xx, 4xx, 5xx
  • bounded enum: plan="free|pro|enterprise"
  • normalized error type: timeout, validation, dependency_failure

• Classic histograms multiply series count.

• A classic Prometheus histogram creates one time series per bucket, plus _sum and _count, for every label combination.

• Approximate formula:

  series = label_combinations × (number_of_buckets + 2)

• Example: 20 buckets × 5 HTTP methods × 20 routes × 5 status classes =
  5 × 20 × 5 × (20 + 2) = 11,000 series for one histogram metric family.

• Bucket count should be treated as a production cost decision, not just an accuracy decision.

• Histogram bucket explosion is often caused by combining histograms with high-cardinality labels.

• Histograms are usually appropriate for latency, request size, response size, queue time, and similar distributions.
• They become dangerous when combined with labels such as user_id, raw path, tenant ID with many tenants, or per-instance ephemeral labels.

• Prefer a small number of service-level histograms and use exemplars/traces for individual request investigation.

• Native histograms may reduce bucket-series explosion but are not a universal fix.

• Native histograms encode bucket data differently and can reduce the number of separate time series compared with classic histograms.
• They require version and backend compatibility checks across Prometheus, remote-write receiver, long-term store, query layer, and dashboards.

• Do not assume native histograms automatically reduce cost in every storage backend.

• Exemplars are for linking metrics to traces, not for turning metrics into per-request storage.

• Exemplars can attach a trace ID or similar reference to selected observations.
• They are useful for jumping from a latency spike to a representative trace.
• They should not be used as a workaround for adding trace_id or request_id as metric labels.

• Exemplar volume, retention, UI support, and remote-write behavior should be verified before broad rollout.

• Remote-write can turn local cardinality mistakes into direct infrastructure or vendor cost.

• Remote-write sends samples to another system; if all raw high-cardinality series are forwarded, local mistakes become network, queue, storage, and billing pressure.
• Cost containment should happen before the data leaves Prometheus where possible.

• Useful controls include:

  • metric_relabel_configs to drop unwanted scraped series before ingestion.
  • write_relabel_configs to drop or keep selected series before remote write.
  • scrape target selection and exporter configuration to avoid collecting unnecessary metrics.
  • lower scrape frequency for low-value or expensive targets.
  • recording rules for lower-cardinality aggregates, combined with remote-write filtering so only the aggregate is forwarded when appropriate.

• Remote-write queue tuning is not a substitute for cardinality control.

• Queue settings such as shards, capacity, batch size, and backoff can help throughput and reliability.
• They do not reduce the number of samples produced.

• If series cardinality is unbounded, queue tuning may only delay failures or shift the bottleneck to the remote backend.

• Operational signals to watch

• Local TSDB/cardinality:
  • active head series
  • series created/removed rate
  • samples scraped vs. samples kept after relabeling
  • WAL and head chunk growth
  • query latency and memory usage
• Remote-write:
  • pending samples
  • failed/retried samples
  • send latency
  • shard behavior
  • remote backend ingestion limits or throttling

• Governance:

  • top metrics by series count
  • top labels by value count
  • newly introduced high-cardinality labels after deploys

• Practical policy

• Define a label allowlist for application metrics.
• Ban unbounded identifiers as labels.
• Require route templating for HTTP metrics.
• Require explicit bucket review for every histogram.
• Review metric cardinality in code review, not only after production incidents.
• Use dashboards or tooling to identify unused, expensive, or high-cardinality metrics.
• Treat remote-write configuration as part of cost governance.

Cautions#

• Exact memory, disk, and cost impact depends on scrape interval, retention, sample rate, label size, churn rate, query workload, remote backend compression, and billing model. Avoid using a universal “cost per series” number without measuring in the target environment.
• Native histogram maturity and compatibility vary by Prometheus version and downstream storage/query systems. Verify support before replacing classic histograms.
• Exemplars can improve trace correlation, but their storage and remote-write behavior depends on Prometheus configuration and the receiving backend.
• Dropping labels after ingestion does not always recover the full cost. Prefer preventing bad series at instrumentation/exporter/scrape time where possible.
• Recording rules add new time series. They reduce remote-write or query cost only if paired with a strategy to avoid sending or querying the original high-cardinality raw series.
• Some high-cardinality dimensions, such as tenant or customer tier, may be business-critical. If retained, they need explicit budgets, sampling/aggregation strategy, and remote-write cost review.

Sources#

• https://prometheus.io/docs/practices/instrumentation/
• https://prometheus.io/docs/practices/naming/
• https://prometheus.io/docs/practices/histograms/
• https://prometheus.io/docs/prometheus/latest/configuration/configuration/
• https://prometheus.io/docs/prometheus/latest/configuration/unit_testing_rules/
• https://prometheus.io/docs/prometheus/latest/feature_flags/
• https://prometheus.io/docs/specs/native_histograms/
• https://grafana.com/blog/2022/10/20/how-to-manage-high-cardinality-metrics-in-prometheus-and-kubernetes/
• https://grafana.com/blog/2022/03/21/how-relabeling-in-prometheus-works/
• https://grafana.com/docs/grafana-cloud/send-data/metrics/metrics-cost-management/

Related#

• Prometheus Metrics Failure Modes: Cardinality Explosion, Histogram Bucket Drift, Remote Write Cost, and Relabel Guardrails
• Write Boundary Leakage, Projection Lag UX, Command Idempotency, and Authorization Drift
• Database Transaction Isolation Failure Modes: Write Skew, Phantom Reads, Snapshot Isolation, and Retry Boundaries

Sagwan Revalidation 2026-07-14T22:07:54Z#

• verdict: ok
• note: 핵심 원칙은 현재 Prometheus 운영 관행에도 여전히 유효합니다.

Sagwan Revalidation 2026-07-16T22:33:25Z#

• verdict: ok
• note: 핵심 cardinality 원인과 완화책은 현재 Prometheus practice와 부합함

Sagwan Revalidation 2026-07-18T23:43:34Z#

• verdict: ok
• note: 고카디널리티·히스토그램·remote-write 비용 관리 원칙은 여전히 유효함

Sagwan Revalidation 2026-07-21T01:26:25Z#

• verdict: ok
• note: 최근 관행과도 부합하며 핵심 주장·권장안에 낡은 부분이 없습니다.