A general contractor building a custom home never personally pours the foundation, wires the electrical panel, and hangs the drywall. They start with a blueprint, break it into a dependency graph (foundation before framing, framing before electrical, electrical before drywall), and dispatch specialized trades to work in parallel wherever the blueprint allows it. Every trade reads from and writes to the same physical job site: if the framer moves a wall, the electrician working from yesterday’s plan needs to know before they cut into it, so the site keeps one shared, current set of facts everyone works from. And every trade’s work gets logged, so an inspector, or the contractor themselves, can reconstruct exactly which crew did what, and when, if a wall ends up in the wrong place.

Orchestrating a team of AI agents to solve a complex task is the same coordination problem, minus the physical world. A planner agent decomposes a task like “produce a competitive analysis of three SaaS pricing pages, complete with screenshots and a summary table” into a dependency graph of sub-tasks: scrape each page, extract pricing tiers, generate screenshots, then synthesize a summary once all three scrapes finish. It spawns a specialized sub-agent for each node the moment that node’s dependencies clear, coordinates their outputs through a shared memory store so a synthesis agent can see what the scraping agents already found without re-deriving it, and gives an operator full visibility into what every one of those agents did, every tool call, every memory read, every token spent, because an LLM-driven agent that silently goes off-script is a far harder failure to catch than a process that simply crashes.

The naive approach, a single agent looping through tool calls sequentially inside one long context window, breaks down fast. Sequential execution means a task with 15 independent sub-tasks takes roughly 15 times as long as it needs to, because nothing about a single agent’s control loop lets two unrelated sub-tasks run at once. Widening the context window to hold everything does not fix the coordination problem either: once several agents are genuinely running in parallel, each one needs a consistent view of what the others have already discovered, and a private, per-agent context cannot deliver that. Two agents scraping the same page independently is wasted work, and a synthesis agent starting before its dependencies finish is an agent working from stale or missing facts.

The forces genuinely in tension are parallelism, consistency, and observability. Running sub-agents in parallel is what makes the system fast, but parallel agents writing to shared state without any concurrency control corrupt each other’s work exactly like two threads writing an unguarded counter. Full observability into every agent’s reasoning and every memory access is what makes a wrong final answer debuggable at all, but naively logging synchronously on every agent action reintroduces the very latency parallelism was supposed to remove. We need to solve for an explicit task decomposition DAG that captures real dependencies instead of an implicit ordering baked into one agent’s script, a shared memory layer that gives every agent a consistent view of accumulated facts without forcing them into lockstep, and an observability layer that captures everything without becoming the bottleneck it exists to help you debug.

Requirements and Constraints

Functional Requirements

• Accept a high-level goal from a user or an upstream service and produce a task decomposition DAG via a planner agent, where nodes are sub-tasks and edges are dependencies
• Spawn a specialized sub-agent (research, code, retrieval, critic, and similar agent types) for each DAG node the moment that node’s dependencies are satisfied, respecting a configurable max-parallelism budget per session
• Provide a shared memory store every agent in a session can read from and write to, with namespaced keys per session, per node, and global scope, supporting both exact key-value facts and semantic recall over free text
• Support inter-agent communication via the shared memory store for broadcast facts, plus a lightweight direct message channel between a parent agent and the children it spawned
• Aggregate sub-agent outputs into a single coherent result per DAG node, and ultimately per top-level task, resolving conflicting or overlapping claims rather than silently picking one
• Detect a failing, crashed, or stuck sub-agent, propagate that failure to dependent nodes instead of letting them start on missing or corrupted inputs, and support configurable retry and re-planning
• Emit a full execution trace, every spawn, every memory read and write, every tool call, every token spent, queryable per session for debugging and post-hoc review
• Let an operator inspect an in-flight or completed run, replay a specific agent’s reasoning trace, and cancel a running session on demand

Non-Functional Requirements

• Concurrency: sustain 500 concurrent orchestration sessions, averaging 12 sub-agents per session, bursting to 800 concurrent sessions with up to 40 sub-agents for complex tasks, roughly 13,000 concurrently active sub-agents at peak
• Planner latency: produce a validated DAG of up to 20 nodes within p99 4 seconds
• Spawn latency: sub-agent availability under p99 800ms on a cold sandbox provision, under p99 120ms when served from the warm pool
• Shared memory latency: p99 under 30ms for key-value reads and writes, p99 under 150ms for semantic/vector recall queries
• Shared memory throughput: sustain 50,000 reads/sec and 8,000 writes/sec in aggregate across all sessions at peak
• Trace durability: capture 100% of agent actions with zero silent loss, retrievable for 30 days
• Failure isolation: one sub-agent crashing or timing out must never corrupt shared memory state or let a dependent node silently proceed on partial output
• Availability: 99.95% for the orchestration control plane; a single shared memory shard outage may degrade the sessions pinned to it but must never cascade to unrelated sessions

Constraints and Assumptions

• The underlying LLM inference stack is treated as an external dependency with its own latency and cost budget; we do not design model serving here
• Sub-agents run inside sandboxed, resource-limited execution environments; the sandboxing mechanism itself (a microVM or a syscall-filtered container) is out of scope
• Tool use (web search, code execution, file access) is delegated to per-agent tool adapters; individual tool implementations are not designed here
• The core design assumes a single-region deployment; cross-region scaling is addressed separately in the Scaling section

High-Level Architecture

The system has six major components: the Planner Agent, the Orchestrator / Scheduler, the Agent Spawner Pool, the Shared Memory Store, the Result Aggregator, and the Trace and Observability Collector.

A goal arrives at the Planner Agent, which decomposes it into a task decomposition DAG: a set of nodes, each carrying instructions and an agent_type, connected by dependency edges. The Orchestrator / Scheduler takes that DAG and tracks which nodes are ready (all dependencies satisfied), handing ready nodes to the Agent Spawner Pool, which provisions a sandboxed sub-agent for each one, drawing from a warm pool where possible to avoid a cold-start penalty. Each running sub-agent reads accumulated facts from the Shared Memory Store before acting, does its work (calling tools, calling an LLM, or both), and writes its findings back to that same store under its own node’s namespace so any later node that depends on it can see the result without the two agents ever talking to each other directly.

As nodes complete, the Orchestrator marks them succeeded and re-evaluates readiness for their dependents, letting the DAG execute with maximum available parallelism rather than in fixed synchronized waves. Once every node feeding into the final result has succeeded, the Result Aggregator merges their outputs, resolving any conflicting claims, into the single answer returned to the caller. Running the entire time, and never blocking any of the above, the Trace and Observability Collector receives an asynchronous stream of every spawn, memory access, tool call, and token spent, and writes it to a durable, queryable trace store.

The Planner Agent and Task Decomposition DAG

This component’s job is to turn one ambiguous, natural-language goal into an explicit, validated dependency graph of sub-tasks that the rest of the system can execute mechanically, with no further judgment calls required from anything downstream.

The instinct to let the planner’s raw LLM output drive execution directly is dangerous: an LLM asked to produce a task breakdown will occasionally emit a node that depends on a task it never actually defined, or worse, a cycle, task A depends on B which depends on A, that would deadlock a naive scheduler forever. The Planner Agent instead treats its own LLM output as untrusted input to a validation and indexing step, and only a DAG that survives validation is allowed to reach the Orchestrator.

# Planner Agent: turn an LLM-produced task breakdown into a validated, executable DAG
# Demonstrates: schema validation, cycle detection, topological levels for parallel scheduling
from dataclasses import dataclass, field
from collections import deque

@dataclass
class TaskNode:
    node_key: str
    agent_type: str
    instructions: str
    depends_on: list[str] = field(default_factory=list)

class InvalidDAG(Exception):
    pass

def validate_and_index(nodes: list[TaskNode]) -> dict[str, TaskNode]:
    by_key = {n.node_key: n for n in nodes}
    if len(by_key) != len(nodes):
        raise InvalidDAG("duplicate node_key in planner output")
    for node in nodes:
        for dep in node.depends_on:
            if dep not in by_key:
                raise InvalidDAG(f"node '{node.node_key}' depends on unknown node '{dep}'")
    return by_key

def topological_levels(nodes: list[TaskNode]) -> list[list[str]]:
    # Returns nodes grouped into levels: every node in a level can run in parallel,
    # and every node in level N+1 depends on at least one node in level N or earlier.
    by_key = validate_and_index(nodes)
    in_degree = {key: len(node.depends_on) for key, node in by_key.items()}
    dependents: dict[str, list[str]] = {key: [] for key in by_key}
    for node in nodes:
        for dep in node.depends_on:
            dependents[dep].append(node.node_key)

    ready = deque(key for key, deg in in_degree.items() if deg == 0)
    levels: list[list[str]] = []
    visited = 0
    while ready:
        level = list(ready)
        levels.append(level)
        ready.clear()
        for key in level:
            visited += 1
            for dependent in dependents[key]:
                in_degree[dependent] -= 1
                if in_degree[dependent] == 0:
                    ready.append(dependent)

    if visited != len(by_key):
        stuck = [key for key, deg in in_degree.items() if deg > 0]
        raise InvalidDAG(f"cycle detected, nodes never became ready: {stuck}")
    return levels

Think of the planner as the general contractor drawing up the dependency schedule before a single trade shows up on site: it decides “electrical after framing” once, in one place, so that no individual trade has to negotiate ordering with every other trade it happens to interact with. Skip validation and hand raw planner output straight to the scheduler, and a single malformed node, one typo in a depends_on reference, silently strands part of the DAG forever, since nothing ever notices that a phantom dependency will never be satisfied.