A user clicks “checkout” in your e-commerce app. The request fans out to 50+ microservices - payment validation, inventory reservation, fraud scoring, shipping cost calculation, tax computation, loyalty point deduction, notification dispatch. The response comes back in 340ms. Two weeks later, a customer reports their order never arrived and your logs show nothing obviously wrong. You need to reconstruct exactly what happened to that specific request across 50 services, in the exact order it happened, with timing.

That reconstruction problem is what distributed tracing solves. Think of it like a shipping manifest that follows a package through every warehouse, truck, and sorting facility - except your package is an HTTP request, your facilities are microservices, and the manifest must be assembled after the fact from notes left by each facility. At small scale, you can manually correlate log lines by timestamp and request ID. At 50,000 requests per second across 50 services, that produces 2.5 million log entries per second, and finding any single request’s journey requires scanning all of them.

The challenge compounds because each service only sees its own slice. Service A knows it called Service B and got a response in 12ms. Service B knows it was called by something and that it called Service C. Neither service has the full picture. Trace context propagation - threading a common identifier through every hop - is the mechanism that makes assembly possible. But then you face a storage problem: at 50,000 req/s with 20 spans per request, that is 1 billion spans per day. Storing all of them costs a fortune. Dropping spans randomly destroys the coherence that makes traces useful.

We need to solve for three things simultaneously: (1) context propagation that works across languages, protocols, and async boundaries without developer friction, (2) sampling that retains interesting traces while discarding boring ones, and (3) storage and indexing that enables sub-second lookup of any trace by ID across billions of stored spans.

Requirements and Constraints

Functional Requirements

• Instrument services with minimal code changes - ideally automatic instrumentation via library injection
• Propagate trace_id and span_id through HTTP headers, gRPC metadata, and message queue message attributes
• Collect spans from 50+ microservices and correlate them into a single trace tree
• Support both head-based sampling (decide at trace start) and tail-based sampling (decide after trace completes)
• Allow engineers to look up any trace by trace_id and get the full span tree in under 1 second
• Support querying traces by service name, operation name, error flag, and duration range
• Display a waterfall timeline of spans with parent-child relationships
• Retain trace data for 30 days (hot) and 1 year (cold archive)

Non-Functional Requirements

• Ingest: 50,000 requests/second, ~20 spans per request = 1 billion spans per day
• Write latency: span ingestion P99 under 10ms (tracing must not slow down the application)
• Query latency: trace lookup by ID under 1 second P95
• Availability: 99.9% for ingest path (span loss is acceptable), 99.95% for query path
• Durability: sampled spans must not be lost once the sampling decision is made
• Sampling rate: configurable per-service, per-operation, globally (target 1-10% for head-based)
• SDK overhead: under 1ms CPU per span, under 2KB memory overhead per active span

Constraints and Assumptions

• We are not building a full APM - metrics and logs are out of scope
• Service mesh integration (Envoy sidecar auto-instrumentation) is assumed for greenfield services
• Clock skew across nodes up to 50ms is expected - we handle it in the query layer, not at ingest
• We assume OpenTelemetry wire format (OTLP/gRPC) as the standard SDK protocol

High-Level Architecture

The tracing system has six major components that handle the journey from span generation to query response.

The Instrumentation SDK runs inside each service process. It creates spans, injects trace context into outgoing calls, extracts context from incoming calls, and batches completed spans for export. The SDK is the only component that touches the hot path of user requests - everything else is off the critical path.

The Collector Fleet is a pool of stateless agents that receive spans over OTLP/gRPC, perform light validation and enrichment (adding host metadata, normalizing service names), and forward spans to the sampling layer. Collectors are deployed as a sidecar or as a shared service per availability zone.

The Tail Sampler is the most interesting component. It buffers spans for a configurable window (typically 30 seconds) while waiting for the trace to complete, then evaluates sampling rules against the full trace. Traces with errors, high latency, or explicit user tagging are kept; the rest are dropped based on a configured rate.

The Span Store is a time-series-aware columnar store (Cassandra or a purpose-built system like Jaeger’s Cassandra backend or Tempo). Spans are written partitioned by (trace_id, service) with a secondary index on trace_id alone for fast full-trace assembly.

The Trace Index is a separate inverted index (Elasticsearch or ClickHouse) that maps searchable attributes - service name, operation, error code, duration bucket, custom tags - to trace_id values. Queries against the index return a list of trace_ids which are then fetched from the Span Store.

The Query API receives frontend requests, executes the two-phase lookup (index then store), assembles the span tree, and returns the waterfall JSON.