Distributed Tracing and Sampling Strategy TODO
Concept
Distributed tracing is an observability technique that groups the path a single request takes across multiple services into a unit called a trace, recording each unit of work as a span with a parent-child relationship. For spans from different processes to be grouped into the same trace, the trace ID and parent span ID must be propagated through request headers, and this propagation convention has to be standardized for heterogeneous systems to link up. Storing every request is too costly, so sampling is necessary: head-based sampling makes a probabilistic decision at the start of a trace and propagates that decision downstream so the trace doesn't get fragmented. Tail-based sampling waits until the trace finishes and looks at the whole thing before deciding, so it can selectively keep error or slow requests — but it requires buffering until completion, which costs memory and structural complexity. Either way, the sampling rate has to be recorded alongside the data so aggregate metrics can be reconstructed without bias.
Failures tend to happen in the tail rather than the average, and pure random sampling alone means the slow and failed request traces you actually need for post-mortem analysis end up missing.
Code & Formula
# 분산 트레이싱과 샘플링 전략 — trace_id 전파로 스팬을 하나의 트레이스로 묶고,
# 헤드 기반 샘플링(시작 시 결정)과 테일 기반 샘플링(완료 후 에러/느린 트레이스만 선별)을 비교한다.
import random
import uuid
random.seed(3)
class Span:
def __init__(self, trace_id, name, duration_ms, error=False, parent=None):
self.trace_id = trace_id
self.span_id = uuid.uuid4().hex[:8]
self.parent = parent.span_id if parent else None
self.name = name
self.duration_ms = duration_ms
self.error = error
def make_trace(slow=False, error=False):
trace_id = uuid.uuid4().hex[:8]
root = Span(trace_id, "api.handle", random.uniform(5, 15))
child = Span(trace_id, "match.execute", random.uniform(3, 8), parent=root)
tail = Span(trace_id, "db.settle", 200 if slow else random.uniform(2, 6),
error=error, parent=child)
return [root, child, tail]
def head_sample_decision(trace_id, rate=0.1):
"""트레이스 시작 시점에 확률적으로 결정하고, 이 결정을 모든 자식 스팬에 전파한다."""
return (int(trace_id, 16) % 1000) < rate * 1000
def tail_sample_decision(spans, latency_threshold_ms=100):
"""트레이스가 끝난 뒤 전체를 보고, 에러거나 느리면 남긴다."""
total = sum(s.duration_ms for s in spans)
has_error = any(s.error for s in spans)
return has_error or total > latency_threshold_ms
traces = (
[make_trace() for _ in range(20)]
+ [make_trace(slow=True) for _ in range(2)]
+ [make_trace(error=True) for _ in range(2)]
)
head_kept = sum(1 for t in traces if head_sample_decision(t[0].trace_id))
tail_kept_important = sum(
1 for t in traces
if tail_sample_decision(t) and (any(s.error for s in t) or sum(s.duration_ms for s in t) > 100)
)
important_total = sum(1 for t in traces if any(s.error for s in t) or sum(s.duration_ms for s in t) > 100)
head_kept_important = sum(
1 for t in traces
if head_sample_decision(t[0].trace_id) and (any(s.error for s in t) or sum(s.duration_ms for s in t) > 100)
)
print(f"전체 트레이스: {len(traces)}, 그중 중요한(에러/느림) 트레이스: {important_total}")
print(f"헤드 샘플링(10%)으로 보존된 트레이스: {head_kept}, 그중 중요한 것: {head_kept_important}")
print(f"테일 샘플링으로 보존된 중요한 트레이스: {tail_kept_important} / {important_total}")
print("-> 테일 샘플링은 저장량을 줄이면서도 중요한 트레이스를 놓치지 않는다.")
docs/code/algorithms/algorithms-48.py
Exercise
Set up two or more services, propagate trace context with OpenTelemetry, and check directly — with the head sampling rate turned down — whether a given trace cuts off partway through.
Practical Connection
The flow of a user order from the API through the matching engine to transaction submission and receipt confirmation has long, asynchronous spans because of chain confirmation latency, so recording the transaction hash as a span attribute and preserving failed traces via tail sampling is critical for incident analysis.
If you study this on a given day, add a note link and a ✅ to this line in the source curriculum (docs/knowledge/dev-100-curriculum.md) and this spot will lead straight to the note body. You can also write directly on this page — but regenerating overwrites it, so it's safer to keep anything you want to save as markdown under docs/algorithms/.