A support agent that has been talking to the same customer for three weeks has a problem no retrieval system was designed to solve. The session has 400 turns. The transcript is 300,000 tokens. The customer just said “do that again but for the other account,” and every word that gives that sentence meaning is buried somewhere in turn 47, turn 180, and a preference the customer stated once, in passing, on day two.

Think of it like a duty desk at a hospital. The nurse coming on shift cannot read three weeks of charts before seeing the patient. What she gets is a handover: the last few hours verbatim, a running summary of the admission, a short list of standing facts (allergies, medications, do-not-resuscitate status), and the ability to pull a specific old note if something in the conversation points at it. That handover is not a compression of the chart. It is a different data structure, built continuously, optimized for the thirty seconds she has to read it.

The naive approach is to append every turn to a list and truncate from the left when the context window fills. This fails in a specific and expensive way. Truncation is uniform, but importance is not. The turn you drop is as likely to be the one where the customer gave you their account ID as it is to be small talk. Worse, at 30 million active sessions with an average of 60 turns each and a p50 of 8 turns per day of activity, you are re-sending the same history on every single turn. A 400-turn session at 8,000 tokens of packed history costs 3.2 million input tokens over its lifetime just to remember itself. At $3.00 per million input tokens that is $9.60 of pure recall overhead for one conversation.

The second naive approach, “just embed every turn and retrieve top-k,” fails differently. Retrieval over conversation turns has terrible precision because turns are short, highly anaphoric, and semantically near-identical to each other. Query “the other account” against 400 turns and you get 400 cosine similarities clustered between 0.71 and 0.78. There is no signal. And retrieval alone loses ordering, which conversations depend on absolutely: “no, the previous one” is meaningless without sequence.

So the forces in tension are these. We need recall across an unbounded history, precision tight enough that the packed context is mostly signal, coherence so that the model sees a consistent and non-contradictory view of what the user has said, latency low enough that memory assembly does not eat the time-to-first-token budget, and cost low enough that remembering is cheaper than re-asking. We need to solve for tiered storage, budgeted context assembly, and write-path fact extraction with contradiction resolution simultaneously.

Requirements and Constraints

Functional Requirements

• Store an append-only turn log per session with durable ordering and exactly-once turn IDs
• Maintain a rolling summary of the session that survives arbitrary length without unbounded growth
• Extract, store, and update durable facts about the user and the task (preferences, entities, commitments, constraints)
• Retrieve relevant historical turns and facts for the current query, blending semantic similarity with recency and explicit reference resolution
• Pack the retrieved material into a fixed token budget with a deterministic, cacheable prefix
• Support explicit memory operations: pin a fact, forget a fact, correct a fact, export all memory for a session
• Expire and tier-down cold sessions without losing the ability to resume them
• Enforce hard isolation so no session ever reads another session’s memory

Non-Functional Requirements

• Latency: memory assembly p50 under 15ms, p99 under 40ms. This is a hard budget because it sits in front of the model and adds directly to TTFT. Target end-to-end TTFT p99 under 900ms with a 200ms model prefill.
• Throughput: 12,000 turns/second at peak. Each turn is one read (assembly) plus one asynchronous write (turn append, fact extraction, summary update). Read amplification is roughly 4 backend calls per turn.
• Quality: memory recall@8 above 0.92 on a golden set of anaphora-resolution probes. Contradiction rate under 0.5% of assembled contexts. Summary faithfulness above 0.95 by LLM-as-judge.
• Cost: under $0.0009 per turn of memory overhead, including the extraction model, embeddings, and vector storage. Prefix cache hit ratio above 70% so the stable portion of the packed context is not re-prefilled.
• Capacity: 30 million active sessions, 1.8 billion stored turns, 900 million memory vectors at 768 dimensions. Roughly 2.7 TB of raw float32 vectors before quantization.
• Durability: no acknowledged turn is ever lost. Summaries and facts are derived state and may be rebuilt from the turn log.

Constraints

We assume a 128k-context model for generation but deliberately budget only 8,000 tokens for memory, because attention quality degrades in the middle of long contexts and because prefill cost scales with tokens sent. We assume a small, cheap model (8B class, self-hosted) for fact extraction and summarization on the write path, and a 768-dimension embedding model with a int8 quantized HNSW index for memory vectors. Out of scope: cross-session memory that spans different users, the generation model itself, and the retrieval of external documents (the RAG corpus) which is a separate subsystem that this memory store feeds alongside.

High-Level Architecture

Six components, split cleanly across a fast read path and a slower write path. The read path is the only thing in the user’s latency budget; everything else runs behind the response.

A turn arrives at the Session Gateway, which resolves the session, checks isolation, and hands the user’s message to the Context Packer. The packer issues three parallel reads: the last N verbatim turns from the Turn Log (a Redis list backed by Postgres), the current rolling summary and pinned facts from the Fact Store, and a top-k semantic lookup against the Memory Vector Index scoped to this session. It scores, deduplicates, and truncates the candidates into a fixed budget, emits the assembled prompt, and returns. Everything up to this point must complete in 40ms at p99.

The model streams its response. Only after the stream completes does the write path fire. The turn (user message plus assistant response) is appended to the turn log, enqueued for embedding, and handed to the Summary Engine, which decides whether the rolling summary needs a refresh, and to the Fact Extractor, which pulls structured claims out of the exchange and reconciles them against existing facts. Both of those run on a cheap model and both are idempotent on turn ID, so retries are safe.

The separation matters more than any individual component. The read path touches only pre-computed state. It never calls a model, never waits on extraction, and never blocks on a write. A memory store that does inference on the read path cannot hold a 40ms p99, no matter how small the model is.

The Turn Log

The turn log’s job is to be the single source of truth from which every other piece of memory can be rebuilt.

Everything else in this system is derived state. Summaries drift, facts get reconciled wrongly, embedding models get upgraded. When any of those go bad, you need a durable, ordered, immutable record to recompute from. That makes the turn log the one component where we trade latency for correctness.

The design is a two-tier write. Turns land in a Redis list keyed by session (sess:{id}:turns, capped with LTRIM at 64 entries) for fast verbatim reads, and are simultaneously written to a partitioned Postgres table for durability. The Redis copy is a cache of the tail; the Postgres copy is the log. A turn_seq monotonic counter per session, allocated by Redis INCR, gives us exactly-once semantics without distributed coordination: the writer allocates the sequence number before the model call, and the append is an upsert on (session_id, turn_seq).

# Appends a completed turn: fast tail cache plus durable log, idempotent on turn_seq.
import json
import asyncpg
import redis.asyncio as redis

TAIL_LEN = 64

async def append_turn(
    rdb: redis.Redis,
    pg: asyncpg.Pool,
    session_id: str,
    turn_seq: int,
    user_text: str,
    assistant_text: str,
    token_count: int,
) -> None:
    payload = {
        "seq": turn_seq,
        "user": user_text,
        "assistant": assistant_text,
        "tokens": token_count,
    }
    blob = json.dumps(payload, separators=(",", ":"))

    pipe = rdb.pipeline(transaction=True)
    pipe.rpush(f"sess:{session_id}:turns", blob)
    pipe.ltrim(f"sess:{session_id}:turns", -TAIL_LEN, -1)
    pipe.expire(f"sess:{session_id}:turns", 86400 * 7)
    await pipe.execute()

    async with pg.acquire() as conn:
        await conn.execute(
            """
            INSERT INTO session_turn
                (session_id, turn_seq, user_text, assistant_text, token_count)
            VALUES ($1, $2, $3, $4, $5)
            ON CONFLICT (session_id, turn_seq) DO NOTHING
            """,
            session_id, turn_seq, user_text, assistant_text, token_count,
        )

The LTRIM to 64 is the important number. Verbatim recent turns are the highest-value memory per token, but their value falls off a cliff. In our golden set, turns older than about 20 back are almost never the correct answer to an anaphoric reference that is not already captured in the summary or fact store. We keep 64 so that a burst of very short turns (one-word confirmations, which are common) does not push out substantive content.

If you simplified this to Postgres only, the tail read becomes an indexed range scan under contention from 12,000 writes/second on the same table, and your p99 assembly latency goes from 40ms to somewhere north of 200ms. If you simplified to Redis only, a node eviction silently truncates a customer’s history with no way to recover it.