The Day the Numbers Lied

It’s a Tuesday afternoon. Alex, a senior engineer at a fast-growing fintech startup, is staring at a support ticket. A user named Priya insists she earned 650 loyalty points over the weekend. The system shows 500.

No exception in the logs. No failed transaction. No error email. Every service returned 200 OK. The application behaved perfectly, from its own perspective. It wrote data. The database accepted it. The numbers are just… wrong.

Three weeks earlier, two microservices had a collision neither of them knew about.

The Order Service processed Priya’s purchase and prepared to add 150 loyalty points. At the exact same millisecond, the Referral Service activated a referral bonus and prepared to add 400 points. Both services read her current balance of 100 points at the same moment. Both did their math. Both wrote their result. One won. The other’s update was silently overwritten, taking 150 points from Priya without a trace.

This is the lost update problem. It is not a bug in the traditional sense. It is a design gap. And it surfaces in production long after the code ships.

---

The Failure Chain

Here is how the collision played out at the database level:

Timeline (milliseconds)

t=0ms    Priya's balance: 100 pts

t=1ms    Order Service:    SELECT balance FROM users WHERE id=42
         Referral Service: SELECT balance FROM users WHERE id=42
         Both read:        balance = 100

t=5ms    Order Service:    calculates new_balance = 100 + 150 = 250
t=6ms    Referral Service: calculates new_balance = 100 + 400 = 500

t=10ms   Order Service:    UPDATE users SET balance = 250 WHERE id=42
         Result:           balance is now 250 ✓

t=12ms   Referral Service: UPDATE users SET balance = 500 WHERE id=42
         Result:           balance is now 500 ← OVERWRITES 250

Expected: 100 + 150 + 400 = 650
Actual:   500
Lost:     150 points, silently

Both services succeeded. Both returned success codes. The database accepted both writes without complaint. This is Last-Write-Wins behavior - and it silently deleted 150 points from Priya’s account.

The cruel part: you will not catch this in unit tests. You might not catch it in integration tests. You will catch it when an angry customer calls.

The core issue: A read-then-write operation across two separate services is not atomic. There is a window between the read and the write where another process can change the value underneath you.

---

Why “Just Use Transactions” Does Not Scale