Imagine a dress rehearsal where the actors on stage are performing the real show, but there is a second, identical company performing in a dark theater next door with the exact same script. The audience sees only the main stage. The dark company makes every mistake, improvises freely, and sometimes forgets their lines - but no one notices because the audience never sees them. What you get is a complete picture of how the new production would go without any risk to the paying audience.

Shadow traffic testing is that dark theater. When you have a new service you want to validate against real production traffic, you do not A/B test it with 1% of users. You copy every inbound request, send the copy to the new service asynchronously, and compare what the new service would have returned against what the old service actually returned. Users see only the old service. The new service sees 100% of real traffic. You get a confidence signal that no staging environment, load test, or synthetic traffic generator can match.

The challenge is not the copying itself - that is straightforward. The hard problems are the side effects. The new service must not write to the production database, send emails, charge payment cards, or make third-party API calls when processing shadow requests. And the comparison must be smart: two responses that are semantically identical but structurally different (timestamps, ordering, UUIDs) must be recognized as equivalent, while two responses that differ on a single business-critical field must be flagged as divergent. Finally, the shadow path must add zero latency to the production request path - if the mirroring subsystem is slow, it cannot slow down real users.

We need to solve for three constraints simultaneously: zero production latency impact, complete side-effect suppression in the shadow target, and intelligent divergence comparison that surfaces signal without noise.

Requirements and Constraints

Functional Requirements

• Mirror a configurable percentage (1-100%) of inbound HTTP requests to a shadow service
• Shadow requests must be fully isolated: no writes to shared databases, no external API calls, no emails
• Compare shadow responses against production responses field-by-field
• Generate a divergence report with a confidence score updated in near real-time
• Support sampling rate control - adjustable without a service restart
• Support request transformation - the ability to modify shadow request headers or body before forwarding

Non-Functional Requirements

• Production request latency overhead: under 1ms added latency (shadow dispatch must be fully async)
• Shadow request throughput: up to 50,000 requests/second mirrored at peak
• Divergence report refresh: updated within 60 seconds of change in divergence rate
• Confidence score: computed over rolling 5-minute windows, updated every 30 seconds
• Data retention: raw comparison results retained for 72 hours; aggregated metrics retained for 30 days
• Shadow failures must never surface as errors to production callers

Constraints and Assumptions

• The shadow target service is owned by the same team; we trust it not to crash production DBs
• HTTP/gRPC traffic; not applicable to raw TCP streams or binary protocols without framing
• We accept eventual consistency in the divergence report - real-time comparison is not required
• The comparison system ignores non-deterministic fields (timestamps, trace IDs) by configuration

High-Level Architecture

The system has three major planes: the mirror plane (request duplication at the edge), the shadow execution plane (isolated service + side-effect suppression), and the comparison plane (response diffing and confidence reporting).

The mirror plane sits at the API gateway or service mesh layer. When a request arrives, the gateway dispatches it to the production service synchronously (blocking the caller) and simultaneously enqueues a copy of the request to a Shadow Queue. The gateway returns the production response immediately - the shadow dispatch adds microseconds, not milliseconds.

The shadow execution plane reads from the Shadow Queue, applies request transformations if configured, and forwards the request to the Shadow Target with a special X-Shadow-Mode: true header. The Shadow Target uses an isolation layer to suppress all side effects - writes are intercepted and dropped, external calls are mocked, email sends are no-oped.

The comparison plane receives both responses asynchronously, runs the divergence scoring algorithm, and writes results to the Comparison Store. A reporting service aggregates results into a live confidence dashboard.