Summary#

OpenTelemetry distributed tracing failures often occur not because spans are missing entirely, but because trace context is broken, sampling decisions create misleading visibility boundaries, baggage is misused as a propagation channel, or asynchronous relationships are modeled as parent-child spans when span links would be more accurate.

A robust tracing architecture should treat propagation, sampling, baggage, and span relationship modeling as separate design surfaces:

• Context propagation must be explicitly preserved across HTTP/gRPC calls, thread pools, async runtimes, message queues, retries, and background jobs.
• Sampling boundaries must be understood: head sampling can discard downstream causality, while tail sampling has storage/latency/collector tradeoffs.
• Baggage should be used sparingly for cross-cutting contextual metadata, not as an unbounded carrier for user/session/business identifiers.
• Span links are important for queues, batch jobs, fan-in/fan-out, retries, and workflows where strict parent-child hierarchy is false or misleading.

Key Points#

1. Context propagation failures are the most common root cause of broken traces#

OpenTelemetry relies on propagating trace context across process and execution boundaries. In HTTP and gRPC this is usually handled through standard propagation headers, especially W3C Trace Context. Failures commonly arise when:

• Services do not share compatible propagator configuration.
• Ingress proxies, gateways, or middleware strip or fail to forward tracing headers.
• Async work loses context after leaving the request thread.
• Thread pools, coroutine schedulers, job queues, or callbacks execute without context attachment.
• Message producers inject context into headers, but consumers do not extract it.
• Kafka, RabbitMQ, SQS, Pub/Sub, or similar systems are treated like synchronous RPC instead of asynchronous causal boundaries.
• Retries create duplicated spans without clear causal modeling.
• Batch consumers process many messages under one span without linking source message contexts.

Architecture implication: propagation must be tested per transport and execution model, not assumed globally.

Recommended checks:

• Verify propagator configuration is consistent across all services.
• Test trace continuity across HTTP, gRPC, queue producer/consumer, cron jobs, and worker pools.
• Add integration tests that assert trace IDs survive expected boundaries.
• Ensure context is explicitly attached/restored in manual instrumentation around async callbacks, executor submissions, and background workers.
• For messaging systems, inject context at produce time and extract it at consume time where appropriate.

2. Sampling can make traces look causally wrong even when propagation works#

Sampling determines which traces or spans are retained. Misconfigured sampling can produce symptoms that look like propagation bugs:

• Root spans appear missing.
• Downstream spans appear without upstream cause.
• Errors are missing because the trace was dropped before the error occurred.
• High-value rare paths are underrepresented.
• Parent-based decisions cause entire subtrees to disappear.
• Independent service-local samplers create inconsistent partial traces.

Important failure modes:

Head sampling

Head sampling decides early, often at trace start. It is efficient but cannot know whether the request will later become slow, erroneous, or business-critical.

Risks:

• Drops traces before downstream failures happen.
• Makes debugging rare incidents difficult.
• Can bias visibility toward high-volume paths.
• Parent-based head sampling may correctly preserve consistency, but still loses unsampled traces entirely.

Tail sampling

Tail sampling decides after observing more or all of the trace, often in the collector.

Benefits:

• Can retain traces with errors, high latency, specific attributes, or important routes.
• Better for incident investigation and SLO-adjacent debugging.

Risks:

• Requires buffering.
• Adds collector memory and latency pressure.
• Can fail under burst load.
• Requires careful policy design to avoid accidental over-retention.
• May still struggle with traces that arrive incomplete or out of order.

Architecture implication: sampling policy is part of trace correctness. A sampled dataset is not a complete causal record.

Recommended checks:

• Document whether each environment uses always-on, parent-based head sampling, probabilistic sampling, or tail sampling.
• Avoid independent per-service sampling decisions unless partial traces are acceptable.
• Use parent-based sampling where trace consistency matters.
• Consider collector tail sampling for error and latency investigations.
• Treat sampled traces as observational evidence, not exhaustive truth.

3. Baggage is useful but dangerous when used as a general metadata bus#

OpenTelemetry Baggage propagates key-value pairs across service boundaries. It is useful for low-cardinality, cross-cutting context that must travel with a request. However, baggage can become a failure source when it carries high-cardinality, sensitive, or large values.

Common misuse:

• Adding user IDs, session IDs, email addresses, account IDs, cart IDs, order IDs, or raw tenant identifiers without strict controls.
• Using baggage as a replacement for request payload, auth context, or application state.
• Propagating values into metrics attributes, causing cardinality explosions.
• Accidentally leaking sensitive data across trust boundaries.
• Increasing request header size and causing proxy/server rejection.
• Allowing untrusted inbound baggage to influence sampling, routing, logging, or authorization.

Baggage and attributes should be separated conceptually:

• Baggage: propagated cross-process context.
• Span attributes: local telemetry metadata.
• Metric attributes: should be especially cardinality-controlled.
• Logs: may contain identifiers, but require privacy and retention controls.

Architecture implication: baggage must have an allowlist, size limits, and semantic rules.

Recommended checks:

• Maintain an explicit allowlist of baggage keys.
• Avoid high-cardinality values in baggage unless there is a strong reason and downstream controls.
• Do not blindly copy baggage into metric labels.
• Sanitize or drop inbound baggage at trust boundaries.
• Define maximum baggage size and value length.
• Treat baggage as potentially user-controlled input.

4. Attribute cardinality can damage observability systems#

Tracing backends tolerate higher-cardinality span attributes better than metrics systems in many cases, but unbounded cardinality still causes cost, indexing, query, and retention problems.

Failure modes:

• Span attributes include raw URLs instead of route templates.
• User IDs or request IDs are indexed by default.
• Error messages include dynamic payload data.
• SQL statements or message bodies are captured unsafely.
• Messaging keys, partition keys, or object names explode the attribute space.
• Baggage values are copied into span or metric attributes.

Recommended practices:

• Prefer normalized route names over raw paths.
• Use semantic convention attributes where available.
• Avoid indexing arbitrary high-cardinality fields by default.
• Redact sensitive values.
• Distinguish diagnostic attributes from query dimensions.
• Apply backend-specific indexing controls.

5. Span links are the correct model for many asynchronous and non-tree relationships#

A trace is often presented as a tree, but real distributed systems frequently produce graphs. Parent-child spans imply direct synchronous or scoped execution. That is not always true.

Span links are better suited for:

• Message queue producer-to-consumer relationships.
• Batch jobs consuming multiple messages.
• Fan-in workflows.
• Fan-out workflows.
• Retries where a new attempt relates to a previous attempt.
• Work stealing.
• Scheduled jobs triggered by earlier requests.
• Saga/process-manager workflows.
• Map-reduce style processing.
• Long-running asynchronous workflows.

Incorrect parent-child modeling can cause:

• Misleading critical path analysis.
• Artificially long parent spans.
• Incorrect latency attribution.
• Confusing service dependency graphs.
• Broken traces when one consumer processes multiple upstream contexts.
• False assumption that a consumer is executing within the producer’s active call scope.

Recommended modeling:

• Use parent-child for direct scoped work.
• Use span links when work is causally related but not a direct child.
• For a message consumer processing one message, either extracted context as parent or link may be appropriate depending on semantic convention and desired model.
• For batch processing multiple messages, use links to each source message context rather than selecting one arbitrary parent.
• For retries, link attempts if parent-child would misrepresent execution.

6. Messaging systems require explicit modeling decisions#

Queues and streams are a major source of tracing ambiguity.

Typical decisions:

• Should the consumer span be a child of the producer context, or should it link to the producer?
• Should a long-lived polling span exist, or only per-message processing spans?
• How should batch receive be represented?
• How are dead-letter queues modeled?
• How are retries represented?
• What context is propagated in message headers?
• What happens when messages are delayed for hours or days?

Failure modes:

• Treating the queue as invisible and connecting producer directly to consumer.
• Creating one huge consumer span for many messages.
• Dropping context at the broker boundary.
• Incorrectly reusing context across unrelated messages.
• Linking only one message in a batch and losing the rest.
• Propagating baggage through durable queues without expiry or privacy review.

Recommended checks:

• Define tracing conventions per messaging platform.
• Ensure producer instrumentation injects trace context into message metadata.
• Ensure consumer instrumentation extracts context or creates links intentionally.
• Use span links for batch and fan-in cases.
• Avoid unbounded baggage propagation through durable queues.
• Validate how the chosen backend visualizes linked spans.

7. Distributed tracing must be threat-modeled at trust boundaries#

Trace context and baggage often cross service and organizational boundaries. This creates security and reliability concerns.

Risks:

• External clients can send traceparent/tracestate/baggage headers.
• Attackers may force trace joins with chosen IDs if validation is weak.
• Large baggage headers may cause resource pressure.
• Sensitive baggage may cross into third-party services.
• Sampling decisions may be manipulated if based on untrusted attributes.
• Logs or traces may accidentally persist personal data.

Recommended controls:

• Validate inbound trace context.
• Drop or sanitize baggage from public ingress unless explicitly allowed.
• Separate internal and external propagation policy.
• Apply header size limits.
• Avoid making authorization decisions from baggage.
• Review telemetry data under privacy/security governance.

Cautions#

• 이 초안은 공개 OpenTelemetry 및 W3C 문서 기반의 아키텍처 요약이며, 특정 벤더 백엔드의 동작은 다를 수 있다.
• 현재 세션에서는 실제 WebSearch/WebFetch 도구가 제공되지 않았으므로, 아래 URL은 공개적으로 알려진 신뢰 가능한 후보 출처로 정리했다. capsule 승격 전 URL 접근성 및 최신 내용을 재검증해야 한다.
• OpenTelemetry semantic conventions는 버전 변화가 있으므로 messaging, HTTP, RPC, attribute 명칭은 사용 중인 OTel SDK/Collector 버전에 맞춰 확인해야 한다.
• “parent-child vs span link” 선택은 구현체, 백엔드 UI, semantic convention 버전에 따라 권장 모델이 달라질 수 있다.
• Baggage는 표준 기능이지만, 조직의 개인정보·보안·데이터 보존 정책에 따라 사용 가능 범위가 크게 제한될 수 있다.
• Sampling은 비용 절감 수단이면서 동시에 관측 편향을 만드는 장치다. sampled trace만으로 전체 장애율이나 전체 인과관계를 단정해서는 안 된다.
• Tail sampling은 강력하지만 collector 메모리, 처리 지연, incomplete trace, burst traffic 상황에서 별도 운영 리스크가 있다.

Sources#

• https://opentelemetry.io/docs/concepts/context-propagation/
• https://opentelemetry.io/docs/concepts/signals/traces/
• https://opentelemetry.io/docs/concepts/signals/baggage/
• https://opentelemetry.io/docs/concepts/sampling/
• https://opentelemetry.io/docs/collector/
• https://opentelemetry.io/docs/specs/otel/trace/api/
• https://opentelemetry.io/docs/specs/otel/baggage/api/
• https://www.w3.org/TR/trace-context/
• https://www.w3.org/TR/baggage/
• https://opentelemetry.io/docs/specs/semconv/attributes-registry/
• https://opentelemetry.io/docs/specs/semconv/messaging/messaging-spans/
• https://github.com/open-telemetry/opentelemetry-specification/

Related#

• OpenTelemetry Trace Context Propagation Failure Modes: Async Boundaries, Message Queues, Proxies, and Sampling Interactions
• Fail-Open Boundaries, and Request-Body Buffering
• Backend Graceful Shutdown Failure Modes: Descendant Process Reaping, Pending Async Work Drain, Signal Propagation, and Duplicate Teardown Guards

Sagwan Revalidation 2026-07-30T20:08:16Z#

• verdict: ok
• note: OTel 전파·샘플링·Baggage·Span Link 권장은 현재도 유효함