verex

Observability — OpenTelemetry-first instrumentation, swappable backend

Goal: decide how Verex answers “which step is slow?” — and instrument in a way that does not marry us to whichever vendor we pick this year.

Source: conversation 2026-08-17 KST (Datadog / OpenTelemetry walkthrough); no external doc. Prompted by the Datadog agent question — what an agent actually collects, and where traces come from.

1. One-liner

Metrics say “it’s slow”, traces say “this step is slow”, logs say “here’s why” — and only the first comes free from an agent. Traces are generated by a library running inside the process, so the instrumentation choice is a code decision, not an infra one. Instrument with OpenTelemetry (a protocol, not a product) and the export target becomes one config line.

2. Why OTel rather than a vendor SDK

• No penalty. Datadog ingests OTLP natively, and GCP Cloud Trace accepts it too — choosing OTel does not mean giving up either product, it just adds an exit door.
• The alternative is a rewrite. A vendor SDK imported in N call sites means switching costs N edits; OTel means changing where the Collector exports to.
• Known trade-offs, recorded honestly: vendor SDKs still have somewhat deeper auto-instrumentation, the OTel logs signal is the least mature of the three (metrics and traces are stable), and running a Collector adds one more moving part.

3. Verex touchpoints

Verex track	What it needs	Note
ChainJob worker (api)	Distributed trace: enqueue → build tx → submit → confirm	The one place metrics structurally cannot answer the question. Highest value, do first
api-indexer.md (S4–S5)	Request traces + span from API → indexer	Do together with the worker — same trace context or the picture breaks at the boundary
mm-agent.md (S2.5 → S6)	Quote/decision latency	Second candidate; only once the worker path proves the setup
web-ui.md (S3)	RUM (browser)	Lowest priority — defer until there are real users
GCP infra	Export target	Cloud Run has no host for a normal agent: sidecar, serverless-init, or direct OTLP export

4. Standing rule — metric cardinality

Never tag a metric with an unbounded value. In Verex the tempting ones are exactly the dangerous ones: market_id, user_id, tx_hash, order_id. Each distinct tag combination bills as a separate custom metric, so one line of code becomes a five-figure invoice. Those belong on spans and logs (where high cardinality is the point), never on metrics. Writing this down before anyone adds the first one is cheaper than finding it on a bill.

Features

• Observability instrumentation (exploratory)
  • Instrument the ChainJob worker with the OTel SDK first — enqueue → submit → confirm as one trace; verify the span boundaries match the steps we actually suspect
  • Propagate trace context across the API → worker → indexer boundary before adding any second service, or the traces stop at the seam
  • (you) Decide the export target for now — Cloud Trace (already present, effectively free at this size) vs Datadog (better product, real cost). The point of OTel is that this is reversible, so pick the cheap one and move on
  • (you) Decide SDK-direct export vs running a Collector — direct is fewer parts; a Collector buys sampling, redaction and multi-export. Probably direct until it hurts
  • Pick the Cloud Run delivery path (sidecar / serverless-init / direct OTLP) and note the cold-start cost of whichever wins
  • Record the cardinality rule (§4) wherever metrics get added, not just here