Imagine every service in your infrastructure as a water main. At 3 AM on a Tuesday, one of those pipes silently starts leaking. By the time an alert fires, the damage is done - unless you already have every pressure reading from every sensor stored somewhere you can query in seconds. That is what a log aggregation pipeline is: the nervous system of an infrastructure, collecting every signal every service emits and making it queryable before the on-call engineer has finished reading the alert notification.

At 1 million events per second across a fleet of thousands of services, the challenge is not collecting the logs - it is doing everything else simultaneously. You must accept and acknowledge each event within milliseconds so the producer does not block. You must parse, tokenize, and index each event so a full-text search can return in under a second. You must replicate the data across failure domains so no hardware failure loses a log that a compliance audit will need six months from now. And you must tier the storage so 90-day retention costs tens of thousands of dollars per month rather than hundreds of thousands.

The engineering tension is threefold. Write throughput and query latency pull in opposite directions: the data structures that make bulk ingestion fast (append-only log segments) are exactly the wrong structures for random-access search (inverted indexes). Retention cost and query coverage also conflict: compressing old logs to cold storage is cheap, but querying across 90 days of compressed columnar data is orders of magnitude slower than querying a hot index. Finally, durability and latency fight each other at the ingest layer - acknowledging a write before it hits durable storage is fast but loses data on crash, while fsyncing every event before acknowledging is safe but limits throughput to a tiny fraction of what the hardware can do.

Four architectural decisions determine whether you solve all three tensions or merely trade one for another. The write-ahead buffer absorbs burst traffic and decouples producers from the slow indexing process. Log sharding distributes the index across machines so no single node becomes a bottleneck. Columnar storage on the cold path compresses repeated field patterns - like log level or service name - by 10x to 50x compared to raw JSON. And query fanout with result merging lets a single user query span thousands of shards without the user experiencing the latency of sequential shard scanning.

Requirements and Constraints

Functional Requirements

• Accept structured and semi-structured log events (JSON, syslog, CEF) from application services, infrastructure agents, and batch shippers
• Ingest at least 1 million events per second at sustained load, with burst headroom to 3 million events per second
• Index each event within 5 seconds of receipt so it is discoverable via full-text search
• Support full-text search queries across all indexed fields with sub-second response time for queries spanning the last 24 hours
• Support time-range queries and field-value filters (service=payments AND level=ERROR AND message:“timeout”) with result pagination
• Retain all logs for 90 days with tiered storage: hot for 7 days, warm for 30 days, cold for 90 days
• Expose a query API compatible with common log query languages (LogQL subset, Lucene syntax)
• Stream live tail of incoming logs filtered by service or pod label in real-time

Non-Functional Requirements

• Throughput: 1 million events per second sustained, 3 million burst; events average 512 bytes, peak at 8 KB
• Ingest latency: end-to-end acknowledgment within 50ms; indexing within 5 seconds of receipt
• Query latency: P99 under 800ms for 24-hour window full-text searches; P99 under 4 seconds for 7-day window queries
• Durability: zero data loss after acknowledgment; at-least-once delivery from producer to index
• Availability: 99.9% uptime for ingest; 99.5% for query API (brief degradation during compaction is acceptable)
• Retention cost: under $40,000 per month for 90-day retention at 1 million events/second with 512-byte average event size

Constraints

• Log producers use fire-and-forget agents (Fluent Bit, Vector, Logstash) - they will retry on failure but cannot buffer more than 60 seconds locally
• Some log fields (user IDs, payment amounts) are PII-sensitive and must be masked before indexing
• Compliance requires that no log older than 90 days is retained on any durable storage tier
• The query API must return results within 1 second for 80% of production queries from the on-call dashboard
• Budget cap is $50,000 per month total infrastructure including ingest, storage, and query serving

High-Level Architecture

The system decomposes into six tiers, each isolated so that a failure or slowdown in one tier cannot cascade into others.

The Ingest Gateway is a stateless HTTP and gRPC endpoint fleet that accepts log batches, validates schema, applies PII masking, and writes to the write-ahead buffer. It never touches durable storage directly.

The Write-Ahead Buffer is a partitioned Kafka cluster. Each log event is assigned to a partition by its shard key (the hash of service name and date). Kafka provides durable buffering, backpressure absorption, and the ability for index workers to replay from any offset if they fall behind or crash.

The Index Workers are stateful pods that consume from Kafka partitions, parse log fields, tokenize text, build in-memory MemTables, and flush those tables to immutable SSTables on hot storage. Each worker owns a fixed set of shard IDs - this is log sharding by partition assignment.

The Hot Storage tier stores the last 7 days of SSTables with full inverted indexes. It is optimized for random-access reads and serves the majority of interactive queries.

The Cold Storage tier holds day-level columnar Parquet files compressed with Zstandard. It is optimized for sequential scan over large time ranges. Warm storage (days 7-30) sits between: partially indexed columnar files on faster NVMe rather than object storage.

The Query API receives search requests, resolves the affected shard set, fans out sub-queries to the shards in parallel, merges and ranks results, and returns a paginated response to the client.

The Ingest Layer and Write-Ahead Buffer