Every time a developer pushes a commit, a chain of events kicks off: clone the repository, install dependencies, run tests across dozens of shards, build container images, push artifacts, and finally deploy. That chain is a CI/CD pipeline - and for a company with hundreds of engineers, thousands of those chains are starting simultaneously. At 10,000 concurrent jobs, the coordination problem dwarfs the execution problem. The hard part is not running a shell script on a machine. The hard part is deciding which machine runs it, what happens when that machine dies mid-run, and how the engineer watching their terminal sees log output appear line by line in near-real-time.

Think of a CI/CD orchestrator like a construction dispatch center. Thousands of job orders arrive every minute. Each job has a priority, a set of dependencies on prior jobs, and a requirement for a specific type of equipment. Dispatchers assign each order to a crew. If a crew’s van breaks down mid-job, the dispatcher must detect the failure, reassign the partially-done work, and ensure nothing is billed twice. The dispatcher does not swing a hammer - it keeps everything organized so the crews can work without stepping on each other.

The naive approach - a single process that polls a database table, fires off shell commands, and tails log files - collapses quickly. At 100 concurrent jobs, a single coordinator becomes a bottleneck. At 1,000, the log-tailing I/O alone saturates the network interface. At 10,000, a single coordinator crash drops all in-flight state. Distributed systems theory gives us a vocabulary for these failure modes: the coordinator needs to be stateless, job ownership needs to be enforced with leases, and log delivery needs a separate fanout path from the job execution path.

Three architectural decisions define every serious CI/CD orchestrator. First, the job queue must provide at-least-once delivery with visibility timeouts, so a job is not considered done until a worker explicitly marks it complete. Second, the worker pool must report heartbeats and have its lease revoked on silence, ensuring jobs return to the queue when a worker dies. Third, log streaming must be decoupled from job execution - workers write to a side channel, and clients subscribe to that channel independently, so neither slow log consumers nor network hiccups can block job progress.

Requirements and Constraints

Functional Requirements

• Accept pipeline definitions as YAML or JSON, describing a DAG (directed acyclic graph) of stages and jobs
• Queue and dispatch up to 10,000 concurrent build and test jobs across a heterogeneous worker pool
• Stream job logs to clients in real-time with per-line timestamps and ordering guarantees
• Store build artifacts (binaries, container images, test reports) with content-addressable references
• Support job retries with configurable backoff, max attempts, and per-step idempotency keys
• Expose REST and WebSocket APIs for job submission, status queries, log tailing, and artifact download
• Cancel in-flight jobs and propagate cancellation signals to running worker processes

Non-Functional Requirements

• Throughput: 10,000 concurrent jobs; peak intake of 5,000 new jobs per minute
• Latency: job dispatch within 500ms of submission for standard-priority jobs; log lines visible within 200ms of emission
• Durability: zero job loss on coordinator or worker crash; at-least-once execution per job
• Availability: 99.9% uptime for the dispatch and queuing layer; graceful degradation if artifact storage is temporarily unavailable
• Scalability: worker pool auto-scales from 100 to 5,000 nodes within 90 seconds based on queue depth
• Observability: per-job execution traces, queue depth metrics, worker utilization dashboards

Constraints

• Workers are untrusted - they run arbitrary user code in isolated containers or VMs
• Build artifacts can be large (up to 10 GB per job); they must not flow through the coordinator
• Log volume can reach 50 MB per job and 500 GB per hour at peak; logs must be queryable after job completion
• The system must support both ephemeral cloud workers and persistent on-premise runners

High-Level Architecture

The system decomposes into six layers, each with a clear ownership boundary.

The API Gateway accepts webhook events from Git providers and REST calls from developers. It validates pipeline YAML, resolves the DAG structure, and enqueues individual jobs. It is stateless and horizontally scalable behind a load balancer.

The Job Queue is the durable backbone of the system. Each job becomes a message with a visibility timeout. Workers acquire jobs by dequeuing messages. If the visibility timeout expires before the worker sends a heartbeat extension, the message becomes visible again and another worker picks it up.

The Worker Pool consists of ephemeral nodes that poll the job queue, execute the assigned job in an isolated environment, and publish results. Workers never share state with each other - all coordination flows through the queue and the state store.

The State Store (a PostgreSQL cluster with a Redis cache) tracks every job’s lifecycle: PENDING, QUEUED, ASSIGNED, RUNNING, SUCCESS, FAILED, RETRYING, and CANCELLED. The state store is the single source of truth for pipeline progress.