Every microservices architecture eventually confronts the same question: when a service instance degrades but does not crash - responding slowly, returning errors for a subset of requests, exhausting its connection pool - how quickly can the system detect it and remove it from the load-balanced pool? Manual detection means someone’s pager goes off, they log in, diagnose, and drain the instance. In a 50-service mesh with 5 instances per service, that is not a process that scales. You need the mesh itself to notice, react, and recover - before the degraded instance cascades failures upstream.

Think of a service mesh like the nervous system in a body. Individual neurons (sidecar proxies) sense local conditions - response latency, error rates, connection drops - and report to the brain (control plane). The brain synthesizes signals across the entire body, makes decisions (reroute, remove, restart), and sends commands back without the person (the developer) having to do anything. The autonomic nervous system handles 99% of the body’s maintenance without conscious thought.

The challenge is distinguishing a sick instance from a slow external dependency. If payment-service-3 is slow, it might be degraded - or the database it talks to might be degraded. Blindly evicting every slow instance can cascade-evict the entire fleet if a shared dependency is slow. The self-healing logic needs to correlate symptoms across multiple instances to attribute fault correctly.

At scale, we need to solve for: sub-minute detection of degraded instances, automatic traffic draining without dropping in-flight requests, remediation actions that do not cause more disruption than the original fault, and enough human override capability that operators can stop automated healing when the automation itself is wrong.

Requirements and Constraints

Functional Requirements

• Continuously probe health of every service instance (liveness, readiness, deep health)
• Detect degradation via multiple signals: error rate, latency percentile, connection saturation
• Automatically reroute traffic away from degraded instances within 30 seconds of detection
• Trigger remediation actions (restart, scale-out, alert) based on configurable policies
• Support canary health gating - block a canary from receiving more traffic if its error rate exceeds threshold
• Provide a circuit breaker per upstream service at the sidecar level

Non-Functional Requirements

• Health check latency: probes complete in <5 seconds
• Reroute propagation: traffic weights updated across all sidecars within 10 seconds
• False positive rate: <1% of healthy instances incorrectly evicted per day
• Availability: control plane failure must not prevent sidecars from enforcing known-good state
• Scale: support 10,000 service instances across 200 services

Constraints

• Application code must not be modified - all mesh logic lives in the sidecar
• Out of scope: service discovery (we assume a service registry exists), mutual TLS (separate concern)
• Automated restarts are rate-limited to 3 per hour per instance to prevent restart storms

High-Level Architecture

The system has two planes: a data plane of sidecar proxies co-located with each service instance, and a control plane that aggregates health signals and pushes routing decisions.

Each service pod runs a sidecar proxy (think Envoy or a custom implementation) that intercepts all inbound and outbound traffic. The sidecar maintains local state: a circuit breaker per upstream service, an error rate counter over a rolling window, and a health probe agent that periodically checks the local application.

The control plane has three components: a health aggregator that collects signals from all sidecars, a remediation engine that applies policies to decide what action to take, and a configuration distributor that pushes updated route weights back to sidecars.

The Sidecar Proxy