RAG design — retriever quality metrics and evaluation harnesses (golden sets, property tests, regression) TODO
Concept
RAG retrieves relevant documents from an external knowledge store for a given query, then generates an answer grounded in that evidence; quality has to be evaluated separately for the retrieval stage and the generation stage. Retrieval quality is measured with metrics like recall@k (whether the correct document lands in the top k), MRR (the rank of the first correct hit), and nDCG (which weights results by rank). Generation quality is split into groundedness — whether the answer is actually supported by the retrieved evidence — and relevance to the query; an unsupported claim is a failure of the generation stage, not the retrieval stage. An evaluation harness is built from a golden dataset of fixed queries with expected evidence, property tests that check invariants like "don't answer if there's no evidence," and regression tests that rerun previously passing cases every time chunking, embeddings, or the prompt changes. Without this harness, you can only judge the effect of a parameter change by feel, which makes it impossible to tell improvement from regression.
RAG systems tend to degrade quietly — without a regression harness, you only find out that changing one chunking size broke a whole class of queries after user complaints have piled up.
Code & Formula
# RAG 설계·리트리버 품질 지표 — 토이 문서 집합에서 코사인 유사도 리트리버로
# top-k 검색을 수행하고 recall@k 로 검색 품질을 측정하는 최소 예시.
import math, re
from collections import Counter
docs = [
"raft leader election uses randomized timeouts",
"b+tree index supports range queries efficiently",
"lsm tree favors write throughput over read amplification",
"vector database uses hnsw for approximate nearest neighbor search",
"merkle tree enables logarithmic membership proofs",
]
def to_vec(text):
words = re.findall(r"[a-z]+", text.lower())
return Counter(words)
def cosine(a: Counter, b: Counter) -> float:
keys = set(a) | set(b)
dot = sum(a[k] * b[k] for k in keys)
na = math.sqrt(sum(v * v for v in a.values()))
nb = math.sqrt(sum(v * v for v in b.values()))
return dot / (na * nb) if na and nb else 0.0
doc_vecs = [to_vec(d) for d in docs]
def retrieve(query, k=3):
qv = to_vec(query)
scored = [(cosine(qv, dv), i) for i, dv in enumerate(doc_vecs)]
scored.sort(reverse=True)
return scored[:k]
query = "how does approximate nearest neighbor search work"
top = retrieve(query, k=3)
print(f"질의: {query!r}")
for score, i in top:
print(f" top: score={score:.3f} doc[{i}]={docs[i]!r}")
# recall@k 평가: 이 질의의 정답 문서는 doc[3] (hnsw ANN) 이라고 가정
gold_idx = 3
eval_set = [("how does approximate nearest neighbor search work", 3),
("what helps range queries on sorted keys", 1),
("how are membership proofs made small", 4)]
def recall_at_k(eval_set, k):
hits = 0
for q, gold in eval_set:
retrieved_ids = [i for _, i in retrieve(q, k)]
hits += gold in retrieved_ids
return hits / len(eval_set)
print(f"\nrecall@1 = {recall_at_k(eval_set, 1):.2f}")
print(f"recall@3 = {recall_at_k(eval_set, 3):.2f}")
docs/code/algorithms/algorithms-98.py
Exercise
Build a golden set of thirty queries with expected evidence documents from your own corpus, measure recall@5 across two or three different chunk sizes, and record the results in a table.
Practical Connection
This applies directly to building an internal query tool over codebase docs or market-rules docs, and the habit of setting up metrics and regression tests first is the same engineering discipline used to manage performance regressions in a matching engine or settlement logic.
If you study this on a given day, add a note link and a ✅ to this line in the source curriculum (docs/knowledge/dev-100-curriculum.md) and this spot will lead straight to the note body. You can also write directly on this page — but regenerating overwrites it, so it's safer to keep anything you want to save as markdown under docs/algorithms/.