Observing Production with eBPF TODO
Concept
eBPF is a sandboxed VM that runs safely inside the kernel. When a user-written program is loaded into the kernel, a verifier statically checks termination and memory-access safety; only programs that pass are JIT-compiled and attached to kprobes, uprobes, tracepoints, perf events, network hooks, and the like. The program and userspace exchange data through shared structures called maps (hash maps, arrays, ring buffers, etc.). The key benefit is being able to observe a running system's internal events without writing a new kernel module or restarting/recompiling the application. Higher-level tools like bpftrace and BCC wrap this whole process down to a one-line script.
It's nearly the only way to trace a latency spike or a specific syscall bottleneck in production that won't reproduce elsewhere, without a code change or redeploy, and it reaches layers that application logs never touch.
Code & Formula
# eBPF로 프로덕션 관측 — 커널 프로브가 이벤트를 map에 쌓고, 유저스페이스가 읽어 히스토그램을 낸다.
# 실제 커널 훅 대신 "probe가 이벤트를 map에 기록한다"는 구조만 순수 파이썬으로 흉내낸다.
import random
from collections import defaultdict
random.seed(1)
class EbpfMap:
"""kprobe/tracepoint가 기록하는 공유 map(dict)을 흉내"""
def __init__(self):
self.hist = defaultdict(int) # latency bucket(us, 로그 스케일) -> count
def record(self, latency_us):
bucket = 1
while bucket * 2 <= latency_us:
bucket *= 2
self.hist[bucket] += 1
def bpftrace_like_probe(events, ebpf_map):
"""실제로는 커널이 syscall 진입/종료 시각차를 계산해 넣어주는 부분을 시뮬레이션"""
for latency_us in events:
ebpf_map.record(latency_us)
# 유휴 상태: 대부분 짧은 지연
idle_events = [max(1, int(random.gauss(50, 15))) for _ in range(200)]
# 부하 상태: 디스크 I/O 경합으로 꼬리가 길어짐
loaded_events = [max(1, int(random.gauss(50, 15))) for _ in range(200)]
loaded_events += [int(random.gauss(4000, 800)) for _ in range(20)] # I/O 경합 스파이크
idle_map = EbpfMap()
loaded_map = EbpfMap()
bpftrace_like_probe(idle_events, idle_map)
bpftrace_like_probe(loaded_events, loaded_map)
def print_hist(name, m):
print(f"--- {name} (syscall latency, us, log2 bucket) ---")
for bucket in sorted(m.hist):
print(f" <= {bucket:5d}us : {'#' * m.hist[bucket]} ({m.hist[bucket]})")
print_hist("idle", idle_map)
print_hist("loaded", loaded_map)
print("\n부하 상태에서 4000us대 버킷이 새로 등장 -> 재배포 없이 I/O 경합 구간을 특정.")
docs/code/algorithms/algorithms-47.py
Exercise
Use a one-line bpftrace script to pull a histogram of disk I/O latency or syscall call counts for a specific process, and compare it under load versus idle.
Practical Connection
When sync stalls on a blockchain node, or tail latency in Verex's backend API, come from the filesystem or network stack rather than application code, this lets you narrow down the cause with evidence instead of guessing.
If you study this on a given day, add a note link and a ✅ to this line in the source curriculum (docs/knowledge/dev-100-curriculum.md) and this spot will lead straight to the note body. You can also write directly on this page — but regenerating overwrites it, so it's safer to keep anything you want to save as markdown under docs/algorithms/.