It’s Monday morning when the support ticket arrives: “Customer says they’re logged in but can’t access premium features.” The user service shows the account as active with a premium subscription. The billing service shows the same account as suspended for non-payment. Both systems are running perfectly. Both databases are consistent. Both answers are correct.

The customer paid their bill on Friday at 4:58 PM. The payment service processed it successfully and marked the account as paid. But the billing service was down for maintenance. When it came back online at 6 PM, it didn’t know about the payment. It suspended the account for non-payment at midnight, exactly as designed. The user service never received the suspension notice because the billing service was busy catching up on weekend processing.

This is distributed systems reality - multiple sources of truth that diverge over time, each making locally correct decisions based on incomplete information. It’s like two security guards at different entrances to a building, each with their own visitor log, making access decisions independently. The building has two different realities of who’s allowed inside, and both guards are following their procedures perfectly.

Why This Happens

The instinct when building microservices is to give each service its own database to ensure independence and avoid the coordination overhead of shared storage. Each service becomes the authoritative source for its domain - user service owns user data, billing service owns payment data, subscription service owns subscription states.

But business operations span domains. A payment affects billing status, which affects subscription access, which affects user permissions. When services make decisions based on their local state without coordination, they create divergent views of reality.

Payment processed successfully (payment service)
  -> Billing service offline during update
    -> User service never receives payment notification
      -> Billing service suspends for non-payment  
        -> User service still shows active
          -> Conflicting states across services

The problem is amplified by event processing delays. Modern systems use message queues and event streams to communicate between services. When one service is slow to process events or goes offline temporarily, the event lag creates temporary inconsistency that can become permanent if not handled properly.

The Naive Solution (and where it breaks)

Most teams reach for distributed transactions or synchronous API calls between services to maintain consistency. The thinking is that if all services agree on changes before committing them, there can be no divergent state.

Distributed transactions use two-phase commit (2PC) to ensure all services either commit together or rollback together. But 2PC requires all participating services to be available and responsive. If the billing service is down when a payment is processed, the entire transaction blocks until billing comes back online.

Synchronous API calls between services create similar problems. When the payment service tries to notify the billing service about a successful payment, it must wait for billing to respond. If billing is slow or down, the payment processing slows or fails entirely.