Tail Latency — Hedged Requests and Load Shedding TODO
Concept
Tail latency refers to response times in the tail of the distribution — p99, p999 — not the average. When a single request fans out to multiple backends, the overall response is bound by the slowest one, so a rare delay on one individual server gets amplified into a common delay at the aggregate level. A hedged request is a technique where, if the first request hasn't gotten a response by, say, its p95 latency, a second identical request goes out to another replica and whichever response comes back first is used — extra load rises only a few percent, but the tail shrinks substantially. Load shedding runs the opposite direction: instead of queueing requests beyond capacity, it rejects them quickly at the door, in order to protect the latency target for requests already accepted. Because wait time worsens exponentially as the queue grows, this is usually paired with mechanisms like discarding requests past their deadline or adaptive concurrency limits.
A dashboard that only tracks average response time misses user-facing failures entirely, and a service that queues without bound under overload collapses further once retries pile on top. Managing the tail and shedding load are basic building blocks of availability design.
Code & Formula
# 테일 레이턴시 — hedged request(느리면 복제본에 한 번 더 요청)로 p99 꼬리를 줄이는 걸 시뮬레이션한다.
# 백엔드 3대 중 하나가 가끔 크게 느려질 때, 단일 요청 대비 hedge 를 걸면 총 요청은 조금 늘지만 꼬리는 크게 줄어든다.
import random
random.seed(11)
def backend_latency_ms(server_id):
# 대부분 10ms 내외, 가끔(5%) 200ms 근처로 튀는 "롱테일" 서버를 흉내낸다.
if random.random() < 0.05:
return random.uniform(150, 250)
return random.uniform(5, 15)
def single_request(num_backends=3):
server = random.randrange(num_backends)
return backend_latency_ms(server), 1 # (지연, 사용한 요청 수)
def hedged_request(hedge_delay_ms=20, num_backends=3):
"""첫 요청이 hedge_delay_ms 안에 안 끝나면 다른 서버에 한 번 더 보내고, 먼저 끝난 쪽을 쓴다."""
server1 = random.randrange(num_backends)
latency1 = backend_latency_ms(server1)
if latency1 <= hedge_delay_ms:
return latency1, 1 # 첫 응답이 충분히 빨랐다 -> hedge 발동 안 함
server2 = (server1 + 1) % num_backends
latency2 = backend_latency_ms(server2)
finish = min(latency1, hedge_delay_ms + latency2) # 두 번째 요청은 hedge_delay 이후에 출발
return finish, 2
def percentile(values, p):
s = sorted(values)
idx = int(len(s) * p) - 1
return s[max(0, idx)]
N = 5000
single_latencies, single_calls = [], 0
hedged_latencies, hedged_calls = [], 0
for _ in range(N):
lat, calls = single_request()
single_latencies.append(lat)
single_calls += calls
for _ in range(N):
lat, calls = hedged_request()
hedged_latencies.append(lat)
hedged_calls += calls
print(f"{'metric':>14} {'single':>10} {'hedged':>10}")
print(f"{'p50 (ms)':>14} {percentile(single_latencies, 0.50):>10.1f} {percentile(hedged_latencies, 0.50):>10.1f}")
print(f"{'p99 (ms)':>14} {percentile(single_latencies, 0.99):>10.1f} {percentile(hedged_latencies, 0.99):>10.1f}")
print(f"{'total calls':>14} {single_calls:>10} {hedged_calls:>10}")
print(f"\nextra request overhead: {(hedged_calls / single_calls - 1) * 100:.1f}% (요청 수는 조금만 늘었다)")
docs/code/algorithms/algorithms-44.py
Exercise
Set up three identical backends, artificially inject delay into one, then measure and tabulate p50, p99, and total request count for a single request versus a hedged request issued after p95 latency.
Practical Connection
A service like Verex that talks to multiple RPC providers can hedge eth_call and receipt lookups to keep one node's momentary delay from turning into delayed order fills or settlement transactions.
If you study this on a given day, add a note link and a ✅ to this line in the source curriculum (docs/knowledge/dev-100-curriculum.md) and this spot will lead straight to the note body. You can also write directly on this page — but regenerating overwrites it, so it's safer to keep anything you want to save as markdown under docs/algorithms/.