Imagine a post office that must deliver every letter exactly once, to the right mailbox, across three different postal services that each use incompatible address formats - and the recipient may have moved since you last checked. That is the core engineering tension of a push notification routing system. At 1,000 notifications per second - which scales to roughly 100 million per day for a mid-sized consumer app - the naive approach of a single service calling APNs and FCM directly collapses under the weight of connection management, error handling, and fan-out complexity.

The problem is deceptively multi-dimensional. First, you have heterogeneous delivery channels: Apple Push Notification service (APNs) speaks HTTP/2 with p8 JWT credentials; Firebase Cloud Messaging (FCM) offers a gRPC batch API plus a legacy REST endpoint; Web Push uses RFC 8030 with VAPID authentication. A single user might have three active devices across all three channels simultaneously. Second, you have delivery guarantees in tension with cost: APNs and FCM are themselves unreliable - they queue messages for offline devices, silently drop them after TTL expiry, and return opaque errors for invalid tokens. Getting at-least-once delivery requires your system to track state and retry, but naive retries cause duplicates. Third, you have scale forcing architectural decisions that seem premature at small scale: at 10 req/s a single Python service calling both APNs and FCM works fine. At 10,000 req/s you need connection pools, batching, fan-out parallelism, and a separate retry pipeline.

The third dimension is token hygiene. Device tokens are not permanent. iOS generates a new APNs token when the user reinstalls the app or restores from backup. Android tokens refresh periodically with FCM. A stale token registry wastes network calls and, worse, you might deliver notifications to the wrong device if a token has been reassigned. This is the kind of silent bug that causes GDPR violations - sending a notification to the prior owner of a recycled phone number or device token.

We need to solve for channel routing correctness, delivery durability with deduplication, and graceful handling of partial failures simultaneously.

Requirements and Constraints

Functional Requirements

• Accept notification payloads via REST or gRPC and route them to the correct channel (APNs, FCM, or Web Push) based on stored device tokens.
• Support fan-out: one notification to all devices owned by a user (multi-device, multi-channel).
• Guarantee at-least-once delivery: if the upstream gateway (APNs, FCM) returns a transient error, retry with exponential backoff.
• Deduplicate: if the same notification is submitted twice within a configurable window (default 24 hours), drop the second submission without dispatching.
• Track delivery receipts: record SENT, DELIVERED, FAILED, and EXPIRED states per notification per device.
• Support both visible notifications (shown to user, appear in notification tray) and silent notifications (background data payloads that wake the app without UI).
• Provide a token registration API: apps call it on launch to upsert their current device token, platform, and app version.

Non-Functional Requirements

• Throughput: 100M notifications/day average, peak 10x (1.16M/hour burst).
• Latency: 95th-percentile end-to-end dispatch latency under 500ms from API ingestion to gateway call.
• Availability: 99.9% for the ingestion API; 99.5% for the dispatch layer (tolerate gateway outages via retry queue).
• Durability: zero notification loss for retryable failures; dead-letter queue for terminal failures.
• Deduplication window: 24-hour sliding window; configurable to 5 minutes for high-frequency alerts.
• Storage: receipt logs retained 90 days; token registry indefinitely (cleaned by stale token eviction).

Constraints

• We do not build our own device-level delivery guarantee beyond what APNs/FCM provide - we rely on them to handle offline queuing and re-delivery to devices.
• Out of scope: in-app notification inbox (a separate read model problem), email and SMS channels, notification scheduling (run a cron on top of this system for that).
• Assumption: one user_id maps to up to 10 concurrent active devices.

High-Level Architecture

The system has five major tiers. An API Gateway handles authentication, rate limiting, and routes inbound calls to the Deduplication Service, which checks a Redis sliding window before allowing a message to proceed. Approved messages flow to the Token Registry (Cassandra) to resolve user_id into device tokens, then to the Router Engine which selects the right channel per device and hands off to channel-specific senders. The Receipt Tracker writes delivery outcomes to PostgreSQL and feeds failures back into a Kafka retry queue.

Data flows like this: a service calls POST /v1/notify with a user_id, a title, a body, and an optional data payload. The API Gateway validates the JWT and applies per-caller rate limits. The Deduplication Service computes an idempotency key (sha256(caller_id + user_id + content_hash)) and checks Redis with SET key 1 NX EX 86400. If the key already exists, we return 200 OK immediately with a deduplicated: true flag - success from the caller’s perspective, no dispatch. If new, the message enters the routing pipeline. The Router looks up all active device tokens for the user, groups them by channel, and dispatches to each channel’s sender concurrently. Each sender maintains a connection pool to the upstream gateway, reports ACK or FAIL back to the Receipt Tracker, and enqueues failures into the Kafka retry topic.

The separation between the Router Engine and the per-channel Senders is the key structural decision. The Router handles business logic - which devices to target, priority rules, silent vs. visible type selection, and notification batching for bulk sends. The Senders handle the mechanical details of each protocol - HTTP/2 multiplexing for APNs, batch sizing for FCM’s v1 API, VAPID key signing for Web Push. This boundary means you can swap or upgrade an individual sender without touching routing logic.

The Device Token Registry

The Token Registry’s job is to map user_id to a list of (device_id, platform, token, token_updated_at) rows, where a single user may have up to 10 active rows.

The naive approach - a single Postgres table with a (user_id, device_id) primary key - works until you hit high fan-out read volume. Every notification triggers at least one token lookup, which at 10,000 notifications/second means 10,000 reads/second on the tokens table. Postgres handles this fine with good indexing, but this table becomes a hot spot when you add token refreshes, stale evictions, and multi-region replication. We choose Cassandra because its partition model turns user_id-keyed reads into single-node operations, RF=3 gives us fault tolerance without sacrificing read performance, and write throughput for token upserts is excellent.

The lookup path adds Redis as a read-through cache with a 5-minute TTL. On cache miss, we read from Cassandra and populate the cache. The cache hit rate approaches 95% in practice because notifications for the same user often cluster in time (a broadcast event hitting many users simultaneously means the first few lookups populate the cache, and subsequent lookups for the same user hit Redis). The 5-minute TTL means a newly registered token is visible within 5 minutes of registration, which is acceptable for all use cases except very low-latency post-install campaigns.

Stale token eviction is the subtle piece. When APNs returns a 410 Unregistered response, it is telling you definitively that the token is dead. FCM returns UNREGISTERED in its error payload. The Sender does not delete the token synchronously - that would add latency to the critical path. Instead, it publishes a token_eviction_event to a dedicated Kafka topic. A TokenJanitor consumer processes these asynchronously, deletes the row from Cassandra, and invalidates the Redis cache entry.