Transformer compute structure, KV cache, and inference serving (continuous batching, PagedAttention), plus quantization/distillation/LoRA trade-offs TODO
Concept
Transformer inference splits into a prefill stage, which processes the whole prompt at once, and a decode stage, which generates tokens one at a time — the two stages bottleneck on different things. During decode, the key/value vectors of past tokens are kept in a KV cache so they don't need to be recomputed every step; this cache grows with sequence length and the number of concurrent requests, so GPU memory itself becomes the concurrency limit. PagedAttention manages the KV cache in fixed-size pages instead of one large contiguous block, eliminating fragmentation and over-reservation so more requests can be served concurrently. Continuous batching doesn't fix the batch composition per request — at every token-generation step it drops finished requests and admits new ones, cutting GPU idle time. On the model side, quantization lowers weights and activations to lower precision, distillation trains a small model to mimic a large model's outputs, and LoRA freezes the original weights and trains/swaps only small low-rank matrices — each has a different trade-off across quality, memory, and serving flexibility.
LLM serving cost and latency are shaped far more by KV cache management and batching strategy than by model choice, so without understanding this structure, the only lever you have left is buying more GPUs.
Code & Formula
# 트랜스포머 계산 구조·KV 캐시 — 단일 헤드 self-attention을 numpy로 구현하고,
# 자기회귀 decode 시 K/V를 매 스텝 재계산하지 않고 캐시에 append 만 하는 것을 시연.
import numpy as np
np.random.seed(0)
d_model, d_k = 8, 4
Wq = np.random.randn(d_model, d_k) * 0.1
Wk = np.random.randn(d_model, d_k) * 0.1
Wv = np.random.randn(d_model, d_k) * 0.1
def softmax(x):
e = np.exp(x - x.max(axis=-1, keepdims=True))
return e / e.sum(axis=-1, keepdims=True)
def attention(q, K, V):
scores = (q @ K.T) / np.sqrt(d_k) # (1, seq_len)
weights = softmax(scores)
return weights @ V, weights # (1, d_k), (1, seq_len)
# prefill: 초기 프롬프트 3토큰을 한 번에 처리하며 K/V 캐시를 채운다
tokens = np.random.randn(3, d_model) * 0.1
K_cache = tokens @ Wk
V_cache = tokens @ Wv
print("prefill 후 KV 캐시 길이:", len(K_cache))
# decode: 새 토큰마다 Q만 새로 계산하고, K/V는 "재계산 없이" 캐시에 append 만 한다
for step in range(3):
new_token = np.random.randn(1, d_model) * 0.1
q = new_token @ Wq
k_new, v_new = new_token @ Wk, new_token @ Wv
K_cache = np.vstack([K_cache, k_new]) # O(1) append, 과거 K 재계산 없음
V_cache = np.vstack([V_cache, v_new])
out, weights = attention(q, K_cache, V_cache)
print(f"decode step {step}: KV 캐시 길이={len(K_cache)}, "
f"attention weights={np.round(weights[0], 3).tolist()}")
print("\n캐시가 없다면 매 decode 스텝마다 전체 시퀀스의 K/V를 O(n) 재계산해야 한다.")
docs/code/algorithms/algorithms-97.py
Exercise
Serving the same model, ramp up concurrent request count and prompt length, measure throughput and p95 latency, and find the point where the estimated KV cache memory hits its limit and requests start queuing.
Practical Connection
The direct blockchain connection is weak, but as a low-latency service that caches and batches per-request state, this calls for the same mindset as batching and memory-budget design in an order-book engine or an indexer.
If you study this on a given day, add a note link and a ✅ to this line in the source curriculum (docs/knowledge/dev-100-curriculum.md) and this spot will lead straight to the note body. You can also write directly on this page — but regenerating overwrites it, so it's safer to keep anything you want to save as markdown under docs/algorithms/.