Copy English

Why

Verex's settlement pipeline (oracle lookup → settlement → payout) is literally a Saga, and its RPC/indexer calls want a circuit breaker as a baseline.

How it works

Circuit Breaker (inter-service calls): trip the circuit and fail fast once failures cross a threshold, then probe recovery half-open after a cooldown. Saga (data consistency): resolve a distributed transaction as a chain of local transactions plus compensating transactions — eventual consistency without 2PC.

Related code

"""Circuit Breaker + Saga: two microservice patterns in one toy demo.

Circuit Breaker: an inter-service call site that trips OPEN after N failures,
fails fast while open, then probes recovery via a HALF_OPEN trial after a
cooldown. Saga: a settlement pipeline of local steps, each with a compensating
rollback, undone in reverse order the moment one step fails.
"""
import time


class CircuitBreaker:
    def __init__(self, fail_threshold=3, cooldown=0.05):
        self.fail_threshold = fail_threshold
        self.cooldown = cooldown
        self.failures = 0
        self.state = "CLOSED"
        self.opened_at = None

    def call(self, fn):
        if self.state == "OPEN":
            if time.monotonic() - self.opened_at >= self.cooldown:
                self.state = "HALF_OPEN"
            else:
                raise RuntimeError("circuit open: failing fast")
        try:
            result = fn()
        except Exception:
            self.failures += 1
            if self.state == "HALF_OPEN" or self.failures >= self.fail_threshold:
                self.state, self.opened_at = "OPEN", time.monotonic()
            raise
        else:
            self.failures, self.state = 0, "CLOSED"
            return result


def flaky_rpc(calls=[0]):
    calls[0] += 1
    if calls[0] <= 3:
        raise ConnectionError("oracle RPC timeout")
    return "oracle_price=42"


breaker = CircuitBreaker(fail_threshold=2, cooldown=0.02)
for i in range(5):
    try:
        print(f"attempt {i}: {breaker.call(flaky_rpc)} (state={breaker.state})")
    except Exception as e:
        print(f"attempt {i}: failed ({e}) (state={breaker.state})")
    time.sleep(0.03)

# --- Saga: oracle lookup -> settlement -> payout, with compensations ---
ledger = []

def oracle_lookup():
    ledger.append("oracle_locked")
def undo_oracle_lookup():
    ledger.remove("oracle_locked")

def settle():
    ledger.append("settled")
def undo_settle():
    ledger.remove("settled")

def payout():
    raise RuntimeError("payout provider unavailable")

steps = [(oracle_lookup, undo_oracle_lookup), (settle, undo_settle), (payout, None)]
completed = []
try:
    for step, compensate in steps:
        step()
        completed.append(compensate)
    print("saga completed:", ledger)
except Exception as e:
    print(f"saga failed at step: {e} -- rolling back")
    for compensate in reversed(completed):
        if compensate:
            compensate()
    print("ledger after rollback:", ledger)

docs/code/pocs/circuit-breaker-saga.py

← All Dev Notes · Workspace Index · Top ↑

Copy 한국어

마이크로서비스 디자인 패턴 2제 정리

Circuit Breaker(서비스 간 통신 — 실패 임계치 넘으면 회로 개방, 유예 후 half-open 복구 탐색)와 Saga(분산 트랜잭션을 로컬 트랜잭션 연쇄+보상 트랜잭션으로) 정리.

아직 범위 미정 — 정독 노트. (Notion 지시, 2026-08-07 추가.)

왜

Verex 연결: 정산 파이프라인(오라클 조회→정산→페이아웃)이 정확히 Saga 구조이고, RPC·인덱서 호출부에는 Circuit Breaker가 기본기입니다.

동작 방식

① Circuit Breaker(서비스 간 통신): 연쇄 장애 방지 — 실패가 임계치를 넘으면 회로를 열어 호출을 즉시 실패시키고, 유예 후 half-open으로 회복을 탐색. ② Saga(데이터 일관성): 분산 트랜잭션을 로컬 트랜잭션의 연쇄 + 보상 트랜잭션(compensation)으로 풀기 — 2PC 없이 최종 일관성.

관련 코드

"""Circuit Breaker + Saga: two microservice patterns in one toy demo.

Circuit Breaker: an inter-service call site that trips OPEN after N failures,
fails fast while open, then probes recovery via a HALF_OPEN trial after a
cooldown. Saga: a settlement pipeline of local steps, each with a compensating
rollback, undone in reverse order the moment one step fails.
"""
import time


class CircuitBreaker:
    def __init__(self, fail_threshold=3, cooldown=0.05):
        self.fail_threshold = fail_threshold
        self.cooldown = cooldown
        self.failures = 0
        self.state = "CLOSED"
        self.opened_at = None

    def call(self, fn):
        if self.state == "OPEN":
            if time.monotonic() - self.opened_at >= self.cooldown:
                self.state = "HALF_OPEN"
            else:
                raise RuntimeError("circuit open: failing fast")
        try:
            result = fn()
        except Exception:
            self.failures += 1
            if self.state == "HALF_OPEN" or self.failures >= self.fail_threshold:
                self.state, self.opened_at = "OPEN", time.monotonic()
            raise
        else:
            self.failures, self.state = 0, "CLOSED"
            return result


def flaky_rpc(calls=[0]):
    calls[0] += 1
    if calls[0] <= 3:
        raise ConnectionError("oracle RPC timeout")
    return "oracle_price=42"


breaker = CircuitBreaker(fail_threshold=2, cooldown=0.02)
for i in range(5):
    try:
        print(f"attempt {i}: {breaker.call(flaky_rpc)} (state={breaker.state})")
    except Exception as e:
        print(f"attempt {i}: failed ({e}) (state={breaker.state})")
    time.sleep(0.03)

# --- Saga: oracle lookup -> settlement -> payout, with compensations ---
ledger = []

def oracle_lookup():
    ledger.append("oracle_locked")
def undo_oracle_lookup():
    ledger.remove("oracle_locked")

def settle():
    ledger.append("settled")
def undo_settle():
    ledger.remove("settled")

def payout():
    raise RuntimeError("payout provider unavailable")

steps = [(oracle_lookup, undo_oracle_lookup), (settle, undo_settle), (payout, None)]
completed = []
try:
    for step, compensate in steps:
        step()
        completed.append(compensate)
    print("saga completed:", ledger)
except Exception as e:
    print(f"saga failed at step: {e} -- rolling back")
    for compensate in reversed(completed):
        if compensate:
            compensate()
    print("ledger after rollback:", ledger)

docs/code/pocs/circuit-breaker-saga.py

← 전체 개발 노트 · 워크스페이스 인덱스 · 맨 위 ↑