Unix cron has worked since 1975. For most of its life that was fine - cronjobs sent email digests, rotated logs, and ran batch imports on a single server. Then we moved to container orchestration, auto-scaling groups, and multi-region deployments. Now the same cron expression runs on twelve nodes simultaneously, firing the same job twelve times per minute. The “solved” problem is suddenly very broken.

Think of a distributed cron scheduler like a postal sorting office with multiple shifts. Each shift is responsible for packages arriving during its window. If shifts overlap, two sorters grab the same package and deliver duplicates. If no one shows up for a shift because of a maintenance window, packages pile up and downstream customers wait. The engineering challenge isn’t delivering one package - it’s coordinating hand-offs across unlimited shifts without gaps or double-deliveries at any scale.

The naive multi-node approach fails because cron is stateless by design. Standard cron reads a schedule file and fires jobs independently on each host. Adding a second node doesn’t provide redundancy - it provides duplication. You need external state to coordinate which node fires a job at any given tick. That coordination introduces the classic distributed systems triad: leader election to designate one node as the active scheduler, distributed locking to prevent races during failover, and exactly-once semantics to ensure a job fires exactly once per schedule tick regardless of who’s running.

Scale adds another dimension. A single scheduler node can poll a database of 100,000 jobs every second without breaking a sweat. At 10 million jobs, a full table scan every second destroys your database. You need job partitioning - dividing the job space across multiple scheduler nodes so each node only watches its slice. And when any node goes down, the backlog of missed jobs needs catchup execution that replays them in order without triggering a thundering herd. We need to solve for all of this simultaneously.

Requirements and Constraints

Functional Requirements

• Register jobs with a cron expression, target URL or queue topic, and metadata payload
• Fire each job within 500ms of its scheduled time under normal load
• Guarantee exactly-once execution per schedule tick - no duplicates, no skips under normal operation
• Detect and replay jobs missed during scheduler downtime (configurable misfire threshold)
• Support job cancellation, pause, and resume without dropping in-flight executions
• Expose per-job execution history with status, start time, duration, and result

Non-Functional Requirements

• Scale to 10 million registered jobs across the scheduler cluster
• Support 10,000 concurrent job executions per minute peak
• 99.9% scheduler uptime with automatic leader failover in under 30 seconds
• Sub-second scheduling jitter under 10,000 jobs/minute load
• Execution history retained for 90 days
• Tolerate clock drift up to 200ms between nodes

Constraints and Assumptions

• Workers that run job logic are separate services - the scheduler handles dispatch only
• System clocks are NTP-synchronized within 100ms; DB clock is the authoritative time source
• Jobs missed beyond the misfire threshold (default: 1 hour) are advanced to the next future tick, not replayed
• Maximum registered jobs per tenant: 100,000

High-Level Architecture

Five components make up the system: an API layer for job registration and management, a Job Store (PostgreSQL) as the single source of truth for all job state, a Scheduler Cluster of nodes that compete for leadership and dispatch due jobs, a Lock Store (etcd or Redis) for distributed coordination, and a Worker Pool that executes the actual job logic.

Data flows like this: a client registers a job through the API, which writes a row to the Job Store with status=SCHEDULED and a computed next_run_at timestamp. The active scheduler leader polls the Job Store continuously for jobs where next_run_at <= NOW(), acquires a distributed lock per job to prevent concurrent dispatch, transitions the job to RUNNING, and pushes the job payload onto a task queue. Workers consume the queue, execute the job handler, then report back to the Job Store with COMPLETED or FAILED. The leader also computes the next next_run_at using the cron expression and resets status=SCHEDULED for recurring jobs immediately after dispatch - not after execution completes.

Follower nodes in the scheduler cluster do almost nothing during normal operation. They hold a standby lease and watch for the leader’s lease to expire. When the leader crashes, whichever follower acquires the etcd or DB-backed lease first becomes the new leader and resumes dispatching from exactly where the old leader stopped, using the Job Store as the authoritative handoff point.

The Job Store

The Job Store is the heart of the scheduler. Every scheduling decision flows through it: when a job is due, whether it’s already running, when it last executed, and what comes next. It must serve high-throughput poll queries with sub-10ms latency while handling concurrent writers without phantom reads.

A single PostgreSQL table with the right indexes handles this at significant scale. The key design choice is using the database’s clock - NOW() in SQL - rather than the application server’s clock for all scheduling decisions. This eliminates clock skew as a correctness concern: even if two scheduler nodes have drifted clocks, both use the DB’s authoritative time when querying for due jobs.

-- Core job registry and execution state
CREATE TABLE scheduled_jobs (
  id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  tenant_id       UUID NOT NULL,
  name            VARCHAR(255) NOT NULL,
  cron_expression VARCHAR(128) NOT NULL,
  handler_url     TEXT NOT NULL,
  payload         JSONB DEFAULT '{}',
  status          VARCHAR(20) NOT NULL DEFAULT 'SCHEDULED',
  partition_key   SMALLINT NOT NULL,
  next_run_at     TIMESTAMPTZ NOT NULL,
  last_run_at     TIMESTAMPTZ,
  last_run_status VARCHAR(20),
  misfire_grace   INTERVAL NOT NULL DEFAULT '1 hour',
  max_retries     SMALLINT NOT NULL DEFAULT 3,
  retry_count     SMALLINT NOT NULL DEFAULT 0,
  lock_token      UUID,
  lock_expires_at TIMESTAMPTZ,
  concurrent_policy VARCHAR(20) NOT NULL DEFAULT 'SKIP',
  created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  updated_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  CONSTRAINT status_check CHECK (
    status IN ('SCHEDULED','RUNNING','FAILED','PAUSED','CANCELLED')
  ),
  CONSTRAINT concurrent_policy_check CHECK (
    concurrent_policy IN ('SKIP','ALLOW')
  )
);

-- Primary dispatch index: only due SCHEDULED jobs, per partition
CREATE INDEX idx_jobs_dispatch ON scheduled_jobs (partition_key, next_run_at, status)
  WHERE status = 'SCHEDULED';

-- Stuck job recovery: find RUNNING jobs whose lock TTL has expired
CREATE INDEX idx_jobs_stuck ON scheduled_jobs (lock_expires_at)
  WHERE status = 'RUNNING';

-- Per-tenant listing
CREATE INDEX idx_jobs_tenant ON scheduled_jobs (tenant_id, created_at DESC);

-- Scheduler lease table for leader election
CREATE TABLE scheduler_leases (
  lease_name  VARCHAR(64) PRIMARY KEY,
  holder_id   VARCHAR(128) NOT NULL,
  expires_at  TIMESTAMPTZ NOT NULL,
  acquired_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);