A general contractor building a custom home never personally pours the foundation, wires the electrical panel, and hangs the drywall. They start with a blueprint, break it into a dependency graph (foundation before framing, framing before electrical, electrical before drywall), and dispatch specialized trades to work in parallel wherever the blueprint allows it. Every trade reads from and writes to the same physical job site: if the framer moves a wall, the electrician working from yesterday’s plan needs to know before they cut into it, so the site keeps one shared, current set of facts everyone works from. And every trade’s work gets logged, so an inspector, or the contractor themselves, can reconstruct exactly which crew did what, and when, if a wall ends up in the wrong place.

Orchestrating a team of AI agents to solve a complex task is the same coordination problem, minus the physical world. A planner agent decomposes a task like “produce a competitive analysis of three SaaS pricing pages, complete with screenshots and a summary table” into a dependency graph of sub-tasks: scrape each page, extract pricing tiers, generate screenshots, then synthesize a summary once all three scrapes finish. It spawns a specialized sub-agent for each node the moment that node’s dependencies clear, coordinates their outputs through a shared memory store so a synthesis agent can see what the scraping agents already found without re-deriving it, and gives an operator full visibility into what every one of those agents did, every tool call, every memory read, every token spent, because an LLM-driven agent that silently goes off-script is a far harder failure to catch than a process that simply crashes.

The naive approach, a single agent looping through tool calls sequentially inside one long context window, breaks down fast. Sequential execution means a task with 15 independent sub-tasks takes roughly 15 times as long as it needs to, because nothing about a single agent’s control loop lets two unrelated sub-tasks run at once. Widening the context window to hold everything does not fix the coordination problem either: once several agents are genuinely running in parallel, each one needs a consistent view of what the others have already discovered, and a private, per-agent context cannot deliver that. Two agents scraping the same page independently is wasted work, and a synthesis agent starting before its dependencies finish is an agent working from stale or missing facts.

The forces genuinely in tension are parallelism, consistency, and observability. Running sub-agents in parallel is what makes the system fast, but parallel agents writing to shared state without any concurrency control corrupt each other’s work exactly like two threads writing an unguarded counter. Full observability into every agent’s reasoning and every memory access is what makes a wrong final answer debuggable at all, but naively logging synchronously on every agent action reintroduces the very latency parallelism was supposed to remove. We need to solve for an explicit task decomposition DAG that captures real dependencies instead of an implicit ordering baked into one agent’s script, a shared memory layer that gives every agent a consistent view of accumulated facts without forcing them into lockstep, and an observability layer that captures everything without becoming the bottleneck it exists to help you debug.

Requirements and Constraints

Functional Requirements

• Accept a high-level goal from a user or an upstream service and produce a task decomposition DAG via a planner agent, where nodes are sub-tasks and edges are dependencies
• Spawn a specialized sub-agent (research, code, retrieval, critic, and similar agent types) for each DAG node the moment that node’s dependencies are satisfied, respecting a configurable max-parallelism budget per session
• Provide a shared memory store every agent in a session can read from and write to, with namespaced keys per session, per node, and global scope, supporting both exact key-value facts and semantic recall over free text
• Support inter-agent communication via the shared memory store for broadcast facts, plus a lightweight direct message channel between a parent agent and the children it spawned
• Aggregate sub-agent outputs into a single coherent result per DAG node, and ultimately per top-level task, resolving conflicting or overlapping claims rather than silently picking one
• Detect a failing, crashed, or stuck sub-agent, propagate that failure to dependent nodes instead of letting them start on missing or corrupted inputs, and support configurable retry and re-planning
• Emit a full execution trace, every spawn, every memory read and write, every tool call, every token spent, queryable per session for debugging and post-hoc review
• Let an operator inspect an in-flight or completed run, replay a specific agent’s reasoning trace, and cancel a running session on demand

Non-Functional Requirements

• Concurrency: sustain 500 concurrent orchestration sessions, averaging 12 sub-agents per session, bursting to 800 concurrent sessions with up to 40 sub-agents for complex tasks, roughly 13,000 concurrently active sub-agents at peak
• Planner latency: produce a validated DAG of up to 20 nodes within p99 4 seconds
• Spawn latency: sub-agent availability under p99 800ms on a cold sandbox provision, under p99 120ms when served from the warm pool
• Shared memory latency: p99 under 30ms for key-value reads and writes, p99 under 150ms for semantic/vector recall queries
• Shared memory throughput: sustain 50,000 reads/sec and 8,000 writes/sec in aggregate across all sessions at peak
• Trace durability: capture 100% of agent actions with zero silent loss, retrievable for 30 days
• Failure isolation: one sub-agent crashing or timing out must never corrupt shared memory state or let a dependent node silently proceed on partial output
• Availability: 99.95% for the orchestration control plane; a single shared memory shard outage may degrade the sessions pinned to it but must never cascade to unrelated sessions

Constraints and Assumptions

• The underlying LLM inference stack is treated as an external dependency with its own latency and cost budget; we do not design model serving here
• Sub-agents run inside sandboxed, resource-limited execution environments; the sandboxing mechanism itself (a microVM or a syscall-filtered container) is out of scope
• Tool use (web search, code execution, file access) is delegated to per-agent tool adapters; individual tool implementations are not designed here
• The core design assumes a single-region deployment; cross-region scaling is addressed separately in the Scaling section

High-Level Architecture

The system has six major components: the Planner Agent, the Orchestrator / Scheduler, the Agent Spawner Pool, the Shared Memory Store, the Result Aggregator, and the Trace and Observability Collector.

A goal arrives at the Planner Agent, which decomposes it into a task decomposition DAG: a set of nodes, each carrying instructions and an agent_type, connected by dependency edges. The Orchestrator / Scheduler takes that DAG and tracks which nodes are ready (all dependencies satisfied), handing ready nodes to the Agent Spawner Pool, which provisions a sandboxed sub-agent for each one, drawing from a warm pool where possible to avoid a cold-start penalty. Each running sub-agent reads accumulated facts from the Shared Memory Store before acting, does its work (calling tools, calling an LLM, or both), and writes its findings back to that same store under its own node’s namespace so any later node that depends on it can see the result without the two agents ever talking to each other directly.

As nodes complete, the Orchestrator marks them succeeded and re-evaluates readiness for their dependents, letting the DAG execute with maximum available parallelism rather than in fixed synchronized waves. Once every node feeding into the final result has succeeded, the Result Aggregator merges their outputs, resolving any conflicting claims, into the single answer returned to the caller. Running the entire time, and never blocking any of the above, the Trace and Observability Collector receives an asynchronous stream of every spawn, memory access, tool call, and token spent, and writes it to a durable, queryable trace store.

The Planner Agent and Task Decomposition DAG

This component’s job is to turn one ambiguous, natural-language goal into an explicit, validated dependency graph of sub-tasks that the rest of the system can execute mechanically, with no further judgment calls required from anything downstream.

The instinct to let the planner’s raw LLM output drive execution directly is dangerous: an LLM asked to produce a task breakdown will occasionally emit a node that depends on a task it never actually defined, or worse, a cycle, task A depends on B which depends on A, that would deadlock a naive scheduler forever. The Planner Agent instead treats its own LLM output as untrusted input to a validation and indexing step, and only a DAG that survives validation is allowed to reach the Orchestrator.

# Planner Agent: turn an LLM-produced task breakdown into a validated, executable DAG
# Demonstrates: schema validation, cycle detection, topological levels for parallel scheduling
from dataclasses import dataclass, field
from collections import deque

@dataclass
class TaskNode:
    node_key: str
    agent_type: str
    instructions: str
    depends_on: list[str] = field(default_factory=list)

class InvalidDAG(Exception):
    pass

def validate_and_index(nodes: list[TaskNode]) -> dict[str, TaskNode]:
    by_key = {n.node_key: n for n in nodes}
    if len(by_key) != len(nodes):
        raise InvalidDAG("duplicate node_key in planner output")
    for node in nodes:
        for dep in node.depends_on:
            if dep not in by_key:
                raise InvalidDAG(f"node '{node.node_key}' depends on unknown node '{dep}'")
    return by_key

def topological_levels(nodes: list[TaskNode]) -> list[list[str]]:
    # Returns nodes grouped into levels: every node in a level can run in parallel,
    # and every node in level N+1 depends on at least one node in level N or earlier.
    by_key = validate_and_index(nodes)
    in_degree = {key: len(node.depends_on) for key, node in by_key.items()}
    dependents: dict[str, list[str]] = {key: [] for key in by_key}
    for node in nodes:
        for dep in node.depends_on:
            dependents[dep].append(node.node_key)

    ready = deque(key for key, deg in in_degree.items() if deg == 0)
    levels: list[list[str]] = []
    visited = 0
    while ready:
        level = list(ready)
        levels.append(level)
        ready.clear()
        for key in level:
            visited += 1
            for dependent in dependents[key]:
                in_degree[dependent] -= 1
                if in_degree[dependent] == 0:
                    ready.append(dependent)

    if visited != len(by_key):
        stuck = [key for key, deg in in_degree.items() if deg > 0]
        raise InvalidDAG(f"cycle detected, nodes never became ready: {stuck}")
    return levels

Think of the planner as the general contractor drawing up the dependency schedule before a single trade shows up on site: it decides “electrical after framing” once, in one place, so that no individual trade has to negotiate ordering with every other trade it happens to interact with. Skip validation and hand raw planner output straight to the scheduler, and a single malformed node, one typo in a depends_on reference, silently strands part of the DAG forever, since nothing ever notices that a phantom dependency will never be satisfied.

The Agent Spawning Protocol

This component’s job is to turn a ready DAG node into a running, sandboxed sub-agent process within the latency budget, without ever letting the system spawn more concurrent agents than its resource budget or the session’s max_parallel_agents setting allows.

The tempting simplification is to provision a fresh sandbox from scratch for every spawn, which is simple to reason about but puts a full container or microVM cold-start, several hundred milliseconds at minimum, directly on the critical path of every single sub-agent, even the ones doing trivial work. The Spawner Pool instead maintains a small warm pool of pre-initialized sandboxes it hands out immediately, falling back to a cold provision only when the warm pool is empty, and it enforces the concurrency budget with a bounded semaphore rather than an unbounded goroutine or thread per spawn request.

// Agent spawn request/response contract between the Orchestrator and the Agent Spawner Pool
syntax = "proto3";
package orchestration.v1;

message SpawnRequest {
  string session_id = 1;
  string node_id = 2;
  string agent_type = 3;              // "research", "code", "retrieval", "critic"
  string instructions = 4;
  repeated string memory_read_scope = 5;  // namespaces this agent may read
  int32 timeout_seconds = 6;
  int32 max_tokens = 7;
}

message SpawnResponse {
  string spawn_id = 1;
  string sandbox_id = 2;
  SpawnStatus status = 3;
  string error_message = 4;
}

enum SpawnStatus {
  SPAWN_STATUS_UNSPECIFIED = 0;
  SPAWN_STATUS_ACCEPTED = 1;
  SPAWN_STATUS_REJECTED_AT_CAPACITY = 2;
  SPAWN_STATUS_REJECTED_INVALID = 3;
}

// Agent Spawner Pool: bounded concurrency spawn with a warm sandbox pool to cut cold-start latency
package spawner

import (
	"context"
	"errors"
	"time"
)

var ErrAtCapacity = errors.New("spawner pool at max parallelism")

type Sandbox struct {
	ID        string
	WarmSince time.Time
}

type Pool struct {
	sem       chan struct{} // bounds total concurrently running sub-agents
	warm      chan *Sandbox // pre-initialized sandboxes ready for immediate use
	provision func(ctx context.Context) (*Sandbox, error)
}

func NewPool(maxParallel, warmSize int, provision func(ctx context.Context) (*Sandbox, error)) *Pool {
	return &Pool{
		sem:       make(chan struct{}, maxParallel),
		warm:      make(chan *Sandbox, warmSize),
		provision: provision,
	}
}

// Spawn reserves a concurrency slot, then hands back a warm sandbox if one is queued,
// falling back to a cold provision only when the warm pool is empty.
func (p *Pool) Spawn(ctx context.Context) (*Sandbox, error) {
	select {
	case p.sem <- struct{}{}:
	case <-ctx.Done():
		return nil, ctx.Err()
	default:
		return nil, ErrAtCapacity
	}

	select {
	case sandbox := <-p.warm:
		return sandbox, nil
	default:
		sandbox, err := p.provision(ctx)
		if err != nil {
			<-p.sem // release the slot we reserved but never used
			return nil, err
		}
		return sandbox, nil
	}
}

// Release returns a concurrency slot after a sub-agent finishes, optionally requeuing
// its sandbox into the warm pool for reuse instead of tearing it down immediately.
func (p *Pool) Release(sandbox *Sandbox, reusable bool) {
	if reusable {
		select {
		case p.warm <- sandbox:
		default: // warm pool full, sandbox is torn down instead
		}
	}
	<-p.sem
}

A warm pool is the same trick an airport gate agent uses by keeping a jet bridge pre-positioned instead of driving it into place only after a plane has already parked: the expensive setup happens ahead of demand, so the moment demand actually arrives, the wait is nearly zero. Skip the concurrency semaphore entirely and let the Orchestrator spawn every ready node the instant it becomes ready, and a wide DAG level, 40 nodes becoming ready simultaneously, can spike sandbox creation load hard enough to starve every other session sharing the same fleet.

The Shared Memory Store

This component’s job is to give every agent in a session a single, consistent place to read accumulated facts from and write new facts to, so that agents coordinate through state rather than through direct conversation with each other.

The naive design lets any agent overwrite any key at any time, which works fine until two sub-agents genuinely running in parallel both try to update the same fact, for instance two research agents both writing a competitor_pricing summary, and one silently clobbers the other’s work with no record that a conflict ever happened. The Shared Memory Store instead requires every write to include the version it read, and rejects the write if that version is stale, the same optimistic concurrency control pattern databases use to protect against lost updates.

# Shared Memory Store: namespaced read/write with optimistic concurrency control
# Demonstrates: CAS-style writes via a versioned Lua script, atomic on the Redis server
import json
import redis

r = redis.Redis(host="shared-memory-0.internal", port=6379, decode_responses=True)

_CAS_SET = r.register_script("""
local current = redis.call('HGET', KEYS[1], 'version')
if current and tonumber(current) ~= tonumber(ARGV[1]) then
    return -1
end
redis.call('HSET', KEYS[1], 'value', ARGV[2], 'version', tonumber(ARGV[1]) + 1, 'written_by', ARGV[3])
redis.call('EXPIRE', KEYS[1], ARGV[4])
return tonumber(ARGV[1]) + 1
""")

def memory_key(session_id: str, namespace: str, key: str) -> str:
    return f"mem:{session_id}:{namespace}:{key}"

def read(session_id: str, namespace: str, key: str) -> tuple[dict | None, int]:
    raw = r.hgetall(memory_key(session_id, namespace, key))
    if not raw:
        return None, 0
    return json.loads(raw["value"]), int(raw["version"])

def write_cas(session_id: str, namespace: str, key: str, value: dict, expected_version: int, spawn_id: str, ttl_seconds: int = 86400) -> int:
    # Returns the new version on success, or -1 if expected_version is stale, meaning the
    # caller read an out-of-date value and must re-read before deciding whether to retry.
    new_version = _CAS_SET(
        keys=[memory_key(session_id, namespace, key)],
        args=[expected_version, json.dumps(value), spawn_id, ttl_seconds],
    )
    return int(new_version)

Exact-key lookups only cover half the problem: an agent frequently needs “anything relevant to pricing” rather than a fact stored under a key it already knows the name of, which is what the semantic recall index is for.

-- Semantic recall: find the 5 facts most relevant to a free-text query within this session
SELECT source_key, content_text, 1 - (embedding <=> $1::vector) AS similarity
FROM memory_vectors
WHERE session_id = $2
ORDER BY embedding <=> $1::vector
LIMIT 5;

A namespaced, versioned memory store is like a shared whiteboard in an incident response room where every entry is initialed and timestamped: anyone can read the whole board at a glance, but nobody can erase someone else’s line without first proving they saw the version they are overwriting. Skip the version check and use plain last-write-wins, and the failure is silent: two agents both believe their write succeeded, only one fact survives, and nothing in the system ever flags that a write was lost.

Inter-Agent Communication and Result Aggregation

This component’s job is to let a parent agent hand a direct instruction to a child it spawned without going through the broadcast memory store, and then to merge the outputs of sibling sub-agents into one coherent result once a DAG node’s dependents are ready to consume it.

Broadcast memory is the right tool for facts every agent might eventually want, but it is the wrong tool for a targeted instruction, “stop, I already found this, redirect to checking the enterprise tier instead”, meant for exactly one running agent. For that narrow case the system uses a per-spawn message channel, a lightweight stream distinct from the namespaced key-value store, so a direct message never collides with, or gets buried under, unrelated broadcast facts.

# Inter-agent messaging: a parent agent posts a direct instruction to a spawned child via a
# per-spawn Redis Stream, distinct from the shared memory blackboard used for broadcast facts
import redis
import json
import time

r = redis.Redis(host="shared-memory-0.internal", port=6379, decode_responses=True)

def send_message(spawn_id: str, sender: str, message_type: str, payload: dict) -> str:
    stream_key = f"chan:{spawn_id}"
    return r.xadd(stream_key, {
        "sender": sender,
        "type": message_type,
        "payload": json.dumps(payload),
        "sent_at": str(time.time()),
    })

def read_messages(spawn_id: str, last_id: str = "0", block_ms: int = 2000) -> list[dict]:
    stream_key = f"chan:{spawn_id}"
    response = r.xread({stream_key: last_id}, block=block_ms, count=50)
    if not response:
        return []
    _, entries = response[0]
    return [{"id": entry_id, **fields} for entry_id, fields in entries]

Once sibling sub-agents finish, their outputs rarely agree perfectly, two research agents may each report a slightly different price for the same tier, and picking the wrong one silently is worse than surfacing the disagreement.

# Result Aggregator: merge sibling sub-agent outputs for a DAG node's dependents,
# resolving conflicting claims with a confidence-weighted vote instead of last-write-wins
from collections import defaultdict

def aggregate_results(node_results: list[dict]) -> dict:
    # Each node_result: {"node_key": str, "output": dict, "confidence": float}
    claims: dict[str, list[tuple[float, object]]] = defaultdict(list)
    for result in node_results:
        for field_name, field_value in result["output"].items():
            claims[field_name].append((result["confidence"], field_value))

    merged: dict[str, object] = {}
    conflicts: dict[str, list[object]] = {}
    for field_name, votes in claims.items():
        distinct_values = {value for _, value in votes}
        if len(distinct_values) == 1:
            merged[field_name] = votes[0][1]
            continue
        # More than one distinct value for the same field: take the highest-confidence
        # claim but record the disagreement so the trace shows it was not unanimous.
        votes.sort(key=lambda v: v[0], reverse=True)
        merged[field_name] = votes[0][1]
        conflicts[field_name] = [value for _, value in votes]

    return {"merged": merged, "conflicts": conflicts}

Think of aggregation like a newsroom editor reconciling two reporters’ filed stories on the same event: rather than silently running whichever one landed first, the editor notes exactly where they disagree and decides, with that disagreement visible, which claim to run with or whether it needs a follow-up check. Skip conflict tracking and a downstream node inherits a confidently wrong fact with no trail explaining why two upstream agents ever disagreed in the first place.

Agent Trace, Observability, and Failure Propagation

This component’s job is to capture a complete, queryable record of everything every agent does, without ever making that capture a dependency an agent has to wait on, and to make sure a failing node’s dependents find out about the failure instead of starting on missing or corrupted inputs.

Synchronous logging, writing a trace row and waiting for the acknowledgment before an agent proceeds to its next action, seems like the safe choice, but at 13,000 concurrent sub-agents each emitting roughly 15 trace events over its lifetime, that adds a database round trip to every tool call and memory access across the whole fleet. The Trace Collector instead emits events into an in-process queue that a separate batch writer drains, so an agent’s own execution path never blocks on trace durability.

# Trace collector: async, non-blocking event emission so tracing never adds latency
# to an agent's hot path
import asyncio
import json
import time

class TraceCollector:
    def __init__(self, sink_queue: asyncio.Queue):
        self._queue = sink_queue

    def emit(self, session_id: str, node_id: str, spawn_id: str, event_type: str, payload: dict) -> None:
        event = {
            "session_id": session_id,
            "node_id": node_id,
            "spawn_id": spawn_id,
            "event_type": event_type,
            "payload": payload,
            "occurred_at": time.time(),
        }
        try:
            self._queue.put_nowait(event)
        except asyncio.QueueFull:
            # Dropping a trace event is preferable to blocking the agent's own execution;
            # a full queue signals the batch writer is falling behind and pages on its own.
            pass

async def batch_writer(queue: asyncio.Queue, db, batch_size: int = 200, flush_interval: float = 0.5) -> None:
    batch = []
    while True:
        try:
            event = await asyncio.wait_for(queue.get(), timeout=flush_interval)
            batch.append(event)
        except asyncio.TimeoutError:
            pass
        if batch and (len(batch) >= batch_size or queue.empty()):
            await db.executemany(
                "INSERT INTO trace_events (session_id, node_id, spawn_id, event_type, payload) "
                "VALUES ($1, $2, $3, $4, $5)",
                [(e["session_id"], e["node_id"], e["spawn_id"], e["event_type"], json.dumps(e["payload"])) for e in batch],
            )
            batch = []

When a sub-agent fails, whether it crashes, times out, or exhausts its token budget, the Orchestrator has to decide what happens to every node downstream of it, and “let them run anyway” is never the right default, since a dependent node built on a missing input produces an answer that looks complete but is quietly wrong. The Orchestrator marks the failed node’s entire downstream reachable set as blocked, the same way a general contractor halts drywall on a wall the framing inspector just rejected, rather than letting the next trade start on top of work that failed inspection.

Data Model

The durable state splits into five parts: orchestration sessions, DAG nodes, agent spawns, shared memory (key-value and vector), and the trace event log.

-- Orchestration session: one row per top-level task submitted to the system
CREATE TABLE orchestration_sessions (
    session_id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    goal_text            TEXT NOT NULL,
    status                TEXT NOT NULL DEFAULT 'planning'
                            CHECK (status IN ('planning', 'running', 'completed', 'failed', 'cancelled')),
    max_parallel_agents    SMALLINT NOT NULL DEFAULT 8,
    created_at              TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    completed_at              TIMESTAMPTZ,
    total_token_cost            BIGINT NOT NULL DEFAULT 0
);
CREATE INDEX ON orchestration_sessions (status) WHERE status IN ('planning', 'running');

-- DAG nodes: one row per sub-task the planner produced for a session
CREATE TABLE dag_nodes (
    node_id       UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    session_id     UUID NOT NULL REFERENCES orchestration_sessions(session_id),
    node_key         TEXT NOT NULL,        -- planner-assigned short id, e.g. "fetch_pricing"
    agent_type          TEXT NOT NULL,     -- 'research', 'code', 'retrieval', 'critic', ...
    instructions           TEXT NOT NULL,
    depends_on                 TEXT[] NOT NULL DEFAULT '{}',
    status                        TEXT NOT NULL DEFAULT 'pending'
                                     CHECK (status IN ('pending', 'ready', 'running', 'succeeded', 'failed', 'blocked')),
    result                           JSONB,
    created_at                          TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    started_at                             TIMESTAMPTZ,
    finished_at                               TIMESTAMPTZ,
    UNIQUE (session_id, node_key)
);
CREATE INDEX ON dag_nodes (session_id, status);

-- Agent spawns: one row per sub-agent process actually launched (a retry creates a new row)
CREATE TABLE agent_spawns (
    spawn_id       UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    node_id         UUID NOT NULL REFERENCES dag_nodes(node_id),
    attempt           SMALLINT NOT NULL DEFAULT 1,
    sandbox_id           TEXT,
    model                    TEXT NOT NULL,
    spawned_at                  TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    terminated_at                  TIMESTAMPTZ,
    exit_reason                       TEXT CHECK (exit_reason IN ('completed', 'timeout', 'crashed', 'cancelled')),
    token_cost                           INTEGER NOT NULL DEFAULT 0
);
CREATE INDEX ON agent_spawns (node_id, attempt DESC);

-- Shared memory: namespaced key-value facts, versioned for optimistic concurrency control
CREATE TABLE memory_entries (
    session_id     UUID NOT NULL REFERENCES orchestration_sessions(session_id),
    namespace        TEXT NOT NULL,   -- 'session', 'node:<node_key>', 'global'
    key                 TEXT NOT NULL,
    value                   JSONB NOT NULL,
    version                    INTEGER NOT NULL DEFAULT 1,
    written_by                    UUID REFERENCES agent_spawns(spawn_id),
    updated_at                        TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    PRIMARY KEY (session_id, namespace, key)
);

-- Semantic recall index: embeddings for free-text facts agents want to recall by similarity
CREATE TABLE memory_vectors (
    memory_vector_id     UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    session_id             UUID NOT NULL REFERENCES orchestration_sessions(session_id),
    source_key                TEXT NOT NULL,
    content_text                  TEXT NOT NULL,
    embedding                        VECTOR(1536) NOT NULL,
    written_by                          UUID REFERENCES agent_spawns(spawn_id),
    created_at                             TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX ON memory_vectors USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100);

-- Trace events: append-only log of every agent action, memory access, and tool call
CREATE TABLE trace_events (
    event_id      BIGSERIAL,
    session_id      UUID NOT NULL REFERENCES orchestration_sessions(session_id),
    node_id           UUID REFERENCES dag_nodes(node_id),
    spawn_id             UUID REFERENCES agent_spawns(spawn_id),
    event_type              TEXT NOT NULL
                               CHECK (event_type IN ('spawn', 'memory_read', 'memory_write', 'tool_call', 'llm_call', 'result', 'error')),
    payload                     JSONB NOT NULL,
    occurred_at                    TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    PRIMARY KEY (session_id, event_id)
) PARTITION BY HASH (session_id);

dag_nodes and agent_spawns are deliberately separate tables: a node can have multiple spawns over its lifetime (a timeout followed by a retry), and keeping the spawn history intact instead of overwriting it in place is what lets the trace show exactly which attempt produced which result. trace_events is hash-partitioned on session_id specifically because trace writes and trace reads are both always scoped to a single session, so partitioning by session avoids any partition becoming a hot spot for one especially chatty session while keeping every query about that session’s history within a single partition scan.

A DAG node moves through six states over its lifetime: pending, ready once dependencies clear, running once a sub-agent is spawned, and finally succeeded, failed, or blocked. A blocked node never had its own sub-agent fail directly, it inherited that status from an ancestor, which is exactly the distinction the trace needs to preserve so a debugging session can tell “this failed” apart from “this never got the chance to run.”

Key Algorithms and Protocols

Event-Driven Ready-Set Scheduling

The planner’s topological_levels function is useful for validating a DAG up front, but a production scheduler cannot afford to wait for an entire level to finish before starting the next one: if 9 of 10 nodes in a level finish in two seconds and the tenth takes two minutes, every dependent of those first 9 nodes should start immediately, not two minutes later. The Orchestrator instead tracks per-node remaining dependency counts and recomputes readiness the instant any single node completes.

# Orchestrator scheduler: event-driven readiness instead of static level-by-level barriers,
# so a fast node's dependents start the instant it succeeds, not at the next synchronized level
class Scheduler:
    def __init__(self, nodes: dict[str, TaskNode]):
        self.nodes = nodes
        self.dependents: dict[str, list[str]] = {key: [] for key in nodes}
        for key, node in nodes.items():
            for dep in node.depends_on:
                self.dependents[dep].append(key)
        self.remaining_deps = {key: len(node.depends_on) for key, node in nodes.items()}
        self.status = {key: "pending" for key in nodes}

    def initial_ready(self) -> list[str]:
        ready = [key for key, count in self.remaining_deps.items() if count == 0]
        for key in ready:
            self.status[key] = "ready"
        return ready

    def on_node_succeeded(self, node_key: str) -> list[str]:
        self.status[node_key] = "succeeded"
        newly_ready = []
        for dependent in self.dependents[node_key]:
            self.remaining_deps[dependent] -= 1
            if self.remaining_deps[dependent] == 0 and self.status[dependent] == "pending":
                self.status[dependent] = "ready"
                newly_ready.append(dependent)
        return newly_ready

Precomputing the dependents index once, up front, is what keeps on_node_succeeded at O(out-degree) per call instead of an O(V) scan of every node in the DAG on every single completion event; across a full DAG execution the scheduler does O(V + E) total work, the same bound as the planner’s own validation pass, just spread out as individual completion events instead of computed all at once.

Optimistic Concurrency for Shared Memory Writes

A read-modify-write pattern against shared memory, read the current value, compute a new one, write it back, is exactly where two concurrent agents racing on the same key lose an update if nothing checks that the value has not changed between the read and the write. The CAS-based write_cas function from the Shared Memory Store section is the primitive; the algorithm that uses it correctly is a bounded retry loop around that read-modify-write cycle.

# Shared memory write with CAS retry: read-modify-write pattern under optimistic concurrency
def update_with_retry(session_id: str, namespace: str, key: str, mutate, spawn_id: str, max_attempts: int = 5) -> dict:
    for attempt in range(1, max_attempts + 1):
        current_value, version = read(session_id, namespace, key)
        new_value = mutate(current_value)
        result = write_cas(session_id, namespace, key, new_value, version, spawn_id)
        if result != -1:
            return new_value
    raise RuntimeError(f"lost the CAS race on {namespace}:{key} after {max_attempts} attempts")

Each attempt is O(1) against the memory store, so the algorithm’s cost is O(k) where k is the number of contended retries, which is negligible under the common case of two or three agents occasionally touching the same key, but becomes a real problem for a single hot key many agents update constantly, like a shared progress counter. The edge case worth naming explicitly: for a purely commutative update like incrementing a counter, skip CAS entirely and use an atomic INCR, which needs no retry loop at all because the operation itself, not a read-then-write around it, is what the database makes atomic.

Cascading Failure Propagation

Once a node fails, every node that transitively depends on it, not just its immediate children, has to be marked blocked before the scheduler is allowed to consider spawning anything else in the DAG, or a grandchild two levels downstream could still be scheduled on the assumption that its (now-failed) ancestor’s output exists.

# Failure propagation: mark every downstream dependent of a failed node as blocked,
# instead of letting the scheduler start them on missing or corrupted inputs
from collections import deque

def propagate_failure(failed_node_key: str, dependents: dict[str, list[str]], node_status: dict[str, str]) -> list[str]:
    blocked: list[str] = []
    queue = deque([failed_node_key])
    while queue:
        current = queue.popleft()
        for dependent in dependents.get(current, []):
            if node_status.get(dependent) in ("succeeded", "failed", "blocked"):
                continue
            node_status[dependent] = "blocked"
            blocked.append(dependent)
            queue.append(dependent)
    return blocked

This is a breadth-first traversal of the dependents graph starting from the failed node, so it runs in O(V + E) in the worst case, visiting each downstream node exactly once because the status check prevents revisiting anything already marked. The edge case that matters here is a diamond dependency, two paths converging back on the same downstream node, where without the status guard the traversal would enqueue that node twice and do duplicate work, harmless for correctness but wasteful at DAG widths in the dozens.

Scaling and Performance

The Orchestrator scales horizontally by sharding sessions across control-plane nodes, hashed on session_id, since a single session’s scheduling state never needs to be visible to any other session’s scheduler, which means orchestrator nodes need no coordination with each other at all. The Agent Spawner Pool scales its warm pool size dynamically based on queue depth rather than a fixed schedule, since spawn demand is inherently bursty, a wide DAG level for one large session can momentarily dwarf the steady-state spawn rate of every other session combined. The Shared Memory Store is partitioned the same way sessions are, by a hash of session_id, so a single session’s read and write volume, however heavy, only ever lands on one shard rather than fanning out load across the whole fleet.

Capacity Estimation:

Given:
  - 500 concurrent orchestration sessions sustained, bursting to 800 in peak windows
  - Average 12 sub-agents spawned per session, peak complex sessions spawn up to 40
  - Each sub-agent performs ~6 shared memory reads, ~2 writes, and emits ~15 trace events
  - Trace event record: ~400 bytes; average sub-agent lifetime: ~45 seconds

Concurrent agents:
  Average: 500 sessions * 12 agents = 6,000 concurrently active sub-agents
  Peak: 800 sessions, 15% complex (40 agents) and 85% typical (12 agents)
    = (800 * 0.15 * 40) + (800 * 0.85 * 12) = 4,800 + 8,160 ~= 13,000 concurrent sub-agents

Shared memory throughput:
  Reads:  13,000 agents * 6 reads / 45s  ~= 1,733 reads/sec sustained,
          bursting to ~50,000 reads/sec when many sessions hit a readiness boundary together
  Writes: 13,000 agents * 2 writes / 45s ~= 578 writes/sec sustained,
          bursting to ~8,000 writes/sec during the same convergence windows

Spawn throughput:
  13,000 concurrent agents at ~45s average lifetime implies a spawn rate of
  13,000 / 45 ~= 290 spawns/sec sustained at peak; warm pool sized to absorb 3x that in bursts

Trace event volume:
  290 spawns/sec * 15 events ~= 4,350 trace events/sec ~= 1.7 MB/sec into the batch writer,
  trivial for a hash-partitioned append-only table

Storage (30-day trace retention):
  4,350 events/sec * 400 bytes * 86,400 s/day * 30 days ~= 4.5 TB over the retention window

Compute:
  290 spawns/sec at an 800ms p99 cold-start budget implies roughly 290 * 0.8 ~= 232 sandboxes
  provisioning concurrently at the worst moment; sized to 400 sandbox slots per region with
  headroom, plus a warm pool of 150 pre-initialized sandboxes to absorb burst without cold start

The read-to-write ratio on shared memory is heavily skewed toward reads, every sub-agent reads accumulated context before acting far more often than it writes new facts, which is exactly why the store leans on read replicas and an aggressive local cache of node_key -> status lookups at the Orchestrator, since polling readiness state is one of the highest-frequency operations in the whole system and changes on the order of once per node completion, not once per read.

Failure Modes and Recovery

Failure	Detection	Impact	Recovery
Sub-agent crashes mid-execution (unhandled exception, OOM)	Sandbox process exit is observed by the Spawner Pool; agent_spawns.exit_reason set to crashed	The owning DAG node fails; its transitive dependents are propagated to blocked	Orchestrator applies the retry policy for that agent_type; if attempts are exhausted, the node is marked failed and its blocked dependents are surfaced to the operator
Sub-agent hangs and never returns (infinite tool loop, stuck waiting on an LLM call)	Per-spawn timeout_seconds deadline expires without a completion event	The DAG node and its dependents stall until the timeout fires, delaying but not corrupting the session	Spawner forcibly terminates the sandbox at the deadline, exit_reason set to timeout, and the node is retried or failed per policy exactly as a crash would be
Shared memory shard for a session becomes unavailable	Client-side connection errors and elevated latency on the affected shard’s key range	Reads and writes for sessions hashed to that shard fail or stall; unaffected shards and sessions continue normally	Traffic fails over to the shard’s replica promoted to primary; in-flight writes that had not yet been acknowledged are retried by the calling agent once the shard recovers
Planner produces an invalid or cyclic DAG	topological_levels raises InvalidDAG before any node is scheduled	The session never starts executing; no sub-agent is ever spawned against a malformed plan	Session is marked failed at the planning stage with the specific validation error surfaced; the planner is retried with the error appended as additional context, or escalated to an operator
Orchestrator control-plane node crashes mid-session	Health check failure removes the node from the shard assignment; in-flight node status transitions for its sessions stop	Sessions pinned to that node stall; already-running sub-agents keep running but their completions are not yet observed	A standby orchestrator node picks up the session shard, reconstructs current DAG state from dag_nodes and agent_spawns, and resumes scheduling from persisted state, not from memory
Network partition between the Spawner Pool and the Shared Memory Store	Sub-agent reads and writes to memory start timing out while the sandbox itself remains healthy	Running sub-agents cannot see new facts or persist their own results, effectively stalling without failing outright	Sub-agents apply a bounded local retry with backoff on memory calls before self-reporting as failed, converting a silent stall into an explicit failure the Orchestrator can act on

Comparison of Approaches

Approach	Latency	Complexity	Failure Mode	Best Fit
Single agent, sequential tool loop, no orchestration	High for multi-part tasks, strictly serial	Low, one control loop	One stuck tool call blocks the entire task with no isolation	Simple tasks with few, mostly-sequential steps
Multi-agent with private per-agent context, no shared store	Low per agent, but duplicated discovery work across agents	Medium, agents are independent but blind to each other	Agents silently redo each other’s work or produce contradictory outputs with no way to detect it	Tasks that genuinely decompose into fully independent, non-overlapping sub-tasks
Multi-agent with a blackboard shared memory store (this design)	Low per agent, coordination overhead bounded by memory read/write latency	High, requires DAG validation, versioned memory, and a dedicated trace pipeline	A single hot key or a slow shared memory shard can bottleneck otherwise-parallel agents	Complex tasks whose sub-tasks depend on each other’s intermediate findings
Shared relational database as the only coordination mechanism, poll-based	Medium, bounded by polling interval rather than event latency	Medium, simpler than a dedicated memory service but couples every agent to direct DB access	Polling load scales with agent count regardless of actual event frequency, wasting capacity	Small deployments where a dedicated memory service is not yet justified
Static, hardcoded pipeline with no dynamic planner	Low, no planning latency at all	Low, a fixed sequence of stages	Any task that does not fit the hardcoded shape either fails or is forced into a poor approximation	Narrow, repeatable workflows where the decomposition never actually changes between runs

For a system that has to handle genuinely varied, multi-part goals where sub-tasks depend on each other’s intermediate output, and where a wrong final answer needs to be debuggable after the fact, the blackboard shared memory design is the right call specifically because its added complexity, DAG validation, versioned writes, an async trace pipeline, buys exactly the two properties a static pipeline or a private-context multi-agent setup cannot: dynamic decomposition that adapts to the actual task, and a shared, consistent, and fully traced view of what every agent has established so far. A team without that need, running the same fixed three-step workflow thousands of times a day, is almost always better served by the static pipeline; the orchestration machinery here only pays for itself once task shape genuinely varies and cross-agent dependencies are real.

Key Takeaways

• The DAG is the contract, not the planner’s prose: only a validated, cycle-free graph of nodes and dependencies is allowed to reach the scheduler, so a malformed planner output fails loudly at planning time instead of stranding part of a session silently.
• Shared memory is the only channel between agents: sub-agents never pass results to each other directly, which is what makes retries, replays, and post-hoc debugging possible against a single, versioned source of truth.
• Optimistic concurrency beats last-write-wins: a versioned CAS write turns a silently lost update into an explicit, retryable conflict, the same pattern DynamoDB and Cosmos DB expose as a first-class primitive.
• Scheduling is event-driven, not level-synchronized: a node’s dependents become ready the instant it succeeds, not at the next synchronized wave, which is what keeps one slow node from stalling every sibling that does not actually depend on it.
• Failure propagates transitively before the scheduler acts again: a failed node’s entire downstream subgraph is marked blocked in one traversal, so nothing downstream is ever scheduled against inputs that were never actually produced.
• Tracing must be asynchronous or it becomes the bottleneck it exists to help you debug: an in-process queue and a separate batch writer keep 13,000 concurrent agents from ever blocking on trace durability.
• A clean process exit is not proof of correct work: token-budget truncation and silently wrong reasoning both exit with status 0, so trustworthy verification checks what an agent actually wrote, not just how its process terminated.
• Conflicts are data, not noise to be resolved and discarded: recording disagreements between sibling agents, even ones the aggregator resolves automatically, is what makes a wrong final answer traceable back to its actual source.

The counter-intuitive lesson is that the hardest part of this system is not getting agents to reason well individually, that is a property of the underlying model. It is that coordinating multiple agents safely requires borrowing, almost unchanged, decades-old distributed systems primitives, optimistic concurrency control, event-driven scheduling, partitioned state, that have nothing to do with language models at all. The agents are new; the coordination problem underneath them is not.

Frequently Asked Questions

Q: Why not just give every sub-agent the full conversation history and let it read everything directly, instead of building a separate shared memory store?

A: A full history grows without bound as more agents run, and every sub-agent would pay the token cost of re-reading it on every turn, whether or not it is relevant to that agent’s specific task. A namespaced memory store lets an agent read only the facts scoped to what it actually needs, keeping context small and cheap, while the semantic recall index still makes the rest discoverable on demand rather than force-fed.

Q: Why not use a single long-running agent with tool calls instead of spawning separate sub-agents?

A: A single agent’s control loop is fundamentally sequential, so independent sub-tasks that could run in parallel are instead forced through one context window one at a time, directly multiplying wall-clock latency by the number of sub-tasks. Spawning specialized sub-agents also lets each one carry a narrower, task-specific prompt and toolset instead of one generalist agent trying to hold every possible instruction and tool in scope at once.

Q: Why version shared memory instead of just accepting that last-write-wins is good enough most of the time?

A: Most of the time is exactly the problem: an occasional lost update from two agents racing on the same key is rare enough to pass testing and then show up as an unexplainable wrong answer in production, with no record that a write was ever discarded. Versioning costs one extra round trip on write and turns a silent, rare failure into an explicit, retryable one that shows up in the trace.

Q: How do you prevent two sub-agents from being spawned for the same DAG node?

A: The status guard in on_node_succeeded and the equivalent guard in initial_ready only transition a node to ready from pending, never twice, so even if a completion event for a shared dependency is processed more than once, the second processing finds the dependent already past the pending state and does nothing. The database UNIQUE (session_id, node_key) constraint on dag_nodes is the second line of defense if two scheduler instances somehow race on the same node.

Q: Why not re-plan the entire DAG from scratch every time a sub-agent fails, instead of just blocking its dependents?

A: Re-planning discards everything the successful nodes already established, throwing away real, verified work to work around one node’s failure, and it re-introduces the planner’s own latency and cost on every single failure. Blocking only the failed subgraph, retrying just that node, and re-evaluating readiness from there preserves everything already accomplished and treats the failure as local rather than global.

Q: How do you bound cost when a planner can spawn agents dynamically rather than following a fixed pipeline?

A: Every session carries a max_parallel_agents ceiling and every spawn carries a max_tokens budget, so the planner’s freedom to decompose a task is bounded by hard limits the Orchestrator enforces regardless of what the planner requests. A session that would exceed its token or spawn budget is throttled or failed explicitly rather than allowed to silently run past its allocated cost.

Interview Questions

Q: Design the shared memory store’s concurrency model so two sub-agents running in parallel and writing to the same key never silently overwrite each other’s work.

Expected depth: Discuss optimistic concurrency control with a version-checked compare-and-swap, why last-write-wins fails silently rather than loudly, the retry-with-backoff pattern for the losing writer, and the specific case of a commutative update, like a counter, where CAS is unnecessary overhead compared to an atomic increment.

Q: Walk through what happens end to end when a sub-agent times out mid-task, including how the failure reaches every node that depends on it.

Expected depth: Cover deadline enforcement at the Spawner Pool, marking the node failed versus its dependents blocked, the breadth-first transitive propagation algorithm and why it must guard against revisiting already-blocked nodes, and the distinction between retrying the failed node versus re-planning the whole session.

Q: How would you decide the maximum parallelism for a session, and what should happen when a wide DAG level wants to exceed it?

Expected depth: Discuss the tradeoff between per-session budgets and shared fleet capacity, bounded semaphores versus unbounded spawning, queuing ready nodes that exceed the budget instead of rejecting them outright, and how a warm sandbox pool’s sizing should relate to the aggregate parallelism ceiling across all concurrent sessions.

Q: Design the tracing system so that debugging why a multi-agent run produced a wrong final answer takes minutes instead of hours.

Expected depth: Cover asynchronous, non-blocking trace emission so tracing never competes with agent latency, why every memory read and write needs to be captured, not just tool calls, how conflicting sibling outputs get recorded rather than silently resolved, and how a trace event stream lets an operator replay a specific spawn’s exact reasoning path.

Q: How would you extend this design to support a sub-agent that itself needs to spawn further sub-agents, recursive decomposition instead of a single flat DAG?

Expected depth: Discuss nested DAGs or sub-sessions with their own scoped memory namespace, how a parent node’s status should depend on its nested session’s completion, the risk of unbounded recursive spawning without a depth or total-agent-count ceiling, and how tracing needs to preserve the parent-child session relationship for the whole tree to remain debuggable.