A user asks an AI agent: “Research our competitor’s pricing, summarize their feature differences, and draft a competitive analysis document.” This is not a 200ms API call. It involves 10-20 tool invocations, multiple LLM completions, file reads, web searches, and intermediate reasoning - spanning 30 seconds to 5 minutes. Midway through step 14 of 20, your Kubernetes pod gets evicted during a node rotation. Without a recovery mechanism, the user waits 5 minutes, gets no result, and the task never completes.

Traditional request-response HTTP is the wrong model for agent workloads. Think of it like the difference between asking a barista for a coffee (30 seconds, synchronous is fine) and commissioning a contractor to renovate your kitchen (weeks, they need to be able to pick up where they left off after a weekend). Agent tasks are contracting work: long-lived, multi-step, interruptible, and expensive enough that you cannot afford to re-run from scratch.

The engineering challenge is threefold: agent state persistence (what has the agent done so far, what messages have been exchanged, what tool results are cached), resumable execution (if a worker dies at step 14, restart from step 14 not step 1), and cost cap enforcement (prevent a runaway agent from spending $500 in one task before anyone notices). Add parallel sub-agent execution for decomposable tasks, and you have a non-trivial distributed systems problem wrapped in an AI infrastructure problem.

We need to solve for durability, resumability, cost control, and parallel execution simultaneously, without making the developer API complex.

Requirements and Constraints

Functional Requirements

• Accept task submissions with a goal, constraints (max tokens, max steps, timeout), and optional tool allowlist
• Execute agent tasks asynchronously, returning a task_id immediately
• Persist full agent state (conversation history, tool call results, step index) between execution steps
• Resume task execution from the last successful checkpoint on worker failure or restart
• Support spawning parallel sub-agents for decomposable sub-tasks
• Enforce per-task token budget and step count caps with graceful shutdown
• Provide real-time task status and result retrieval
• Produce a complete execution trace for debugging and billing

Non-Functional Requirements

• Task pickup latency: p99 under 5 seconds from submission
• Resumption latency: p99 under 10 seconds after worker failure
• Concurrent active tasks: 10,000 across all workers
• Maximum task duration: 24 hours
• Checkpoint durability: no more than 1 step of work lost on crash
• Token accounting accuracy: within 1% of actual LLM usage
• Trace completeness: 100% of tool calls and LLM calls captured

Constraints and Assumptions

• Workers are stateless containers (Kubernetes pods) that can be killed anytime
• The LLM API is external (OpenAI, Anthropic) - calls are not idempotent by default
• Sub-agents share the parent task’s token budget
• No human-in-the-loop required for tool approval (configurable but off by default)

High-Level Architecture

The system has four logical layers. The orchestration layer contains the task queue (Redis or SQS), the orchestrator process (which decomposes goals and spawns sub-agents), the state store (durable checkpoint storage), and the cost tracker (token ledger). The agent worker pool is a fleet of stateless containers, each capable of executing any agent task, scaling horizontally based on queue depth. The tool execution layer provides sandboxed access to external capabilities: web search, code execution, file I/O, and external APIs. The trace layer captures every LLM call, tool call, and state transition with full context.

When a task arrives, it enters the queue with a priority level. A worker dequeues it, creates the initial checkpoint, and begins the agent execution loop: call the LLM, receive a tool call or final answer, execute the tool (if any), checkpoint the state, repeat. If the worker dies, the task returns to the queue with its checkpoint pointer intact. The next worker to pick it up reads the checkpoint and continues from the last saved step.

Agent State Persistence