AI Agent Memory: A Practical Design Guide (2026)

Who this is for: engineers and technical founders building an AI agent that needs to remember things across turns, sessions, or users - a support bot, a coding agent, a sales assistant - and are past the "just stuff everything in the prompt" phase.

AI agent memory is the single most underspecified part of most agent projects. Teams spend weeks tuning prompts and picking models, then bolt on "memory" as an afterthought - usually a vector database with default settings and a cron job that never runs. The result is an agent that forgets the user's name after five messages or, worse, retrieves the wrong fact with total confidence. This guide covers the actual architecture: what to store, where, for how long, and how to decide between the four or five real options instead of cargo-culting whatever framework you read about last week.

Why AI agent memory is not just a bigger context window

Claude's 1M-token context window (Sonnet 4.5 and Opus 4.1 both support it in beta) tempts people into thinking memory is solved: just keep appending. It isn't, for three concrete reasons.

• Cost. At $3/$15 per million input/output tokens (Claude Sonnet, mid-2026 pricing), a 200K-token conversation history re-sent on every turn costs roughly $0.60 per turn before the model writes a single word back. Do that 50 times a day per user and memory becomes your biggest line item, not compute.
• Recall degrades with length. Needle-in-a-haystack benchmarks show retrieval accuracy holding well above 90% for a single fact, but multi-hop reasoning across a long transcript drops noticeably past ~50K tokens on most models, Claude included. A bigger window does not mean the model reliably uses everything in it.
• Relevance, not volume, is the actual problem. A support agent does not need the full 400-message history with a customer - it needs the three facts that matter right now (plan tier, open ticket, last resolution). Dumping everything in is a retrieval failure disguised as generosity.

This is why every serious agent architecture - from Anthropic's own context management guidance to LangChain's memory modules to custom in-house systems - treats memory as a retrieval and curation problem, not a storage problem. See context windows explained for the mechanics of what actually happens inside the window.

The four types of AI agent memory

Most production systems combine at least three of these.

1. Working memory (the context window itself)

The current conversation, plus whatever you inject. This is fast, exact, and free of retrieval errors - and it's also the most expensive and the most volatile. It resets when the session ends unless you persist it somewhere.

2. Short-term / episodic memory

The last N turns or the current session, kept in a database (Redis, Postgres, or just your app's session store) and reloaded at session start. Cheap to implement, and it's what makes an agent feel "aware" of what just happened without re-sending the whole transcript. TTL of hours to a few days is typical for a chat widget; TTL of a full session is typical for a coding agent.

3. Long-term / semantic memory

Facts, preferences, and summaries that should survive across sessions and weeks - "this customer is on the enterprise plan," "this user prefers terse answers," "this codebase uses pnpm not npm." This is where vector search usually enters, but a plain SQL table with a user_facts column works fine for structured cases and is far easier to debug.

4. Procedural memory

Learned patterns about how to act - tool-use preferences, past corrections, style guides. Claude Code's own CLAUDE.md file is a real-world example of procedural memory: a human-curated file the agent reloads every session instead of re-learning conventions by trial and error. See writing Claude.md memory for how to structure one well.

Vector memory: what it's actually good for

Vector memory means embedding text into a numeric vector and retrieving by similarity instead of exact match. It is the right tool when memory is unstructured and you don't know the query shape in advance - "what has this user complained about before" - and the wrong tool when you have structured, known-shape data.

A minimal semantic memory write/read loop:

from anthropic import Anthropic
import chromadb

client = Anthropic()
memory_db = chromadb.PersistentClient(path="./agent_memory")
collection = memory_db.get_or_create_collection("user_facts")

def remember(user_id: str, fact: str):
    collection.add(
        documents=[fact],
        metadatas=[{"user_id": user_id, "ts": time.time()}],
        ids=[f"{user_id}-{hash(fact)}"],
    )

def recall(user_id: str, query: str, k: int = 5) -> list[str]:
    results = collection.query(
        query_texts=[query],
        n_results=k,
        where={"user_id": user_id},
    )
    return results["documents"][0]

# Injected into the system prompt, not the raw conversation:
facts = recall(user_id, "billing preferences")
system_prompt = f"Known facts about this user:\n" + "\n".join(facts)

Tip: always filter by user_id (or tenant) in the where clause. A vector store with no tenant isolation is a data leak waiting to happen - see AI privacy and security for Israeli businesses for the compliance angle.

For picking an actual vector database - Pinecone, Weaviate, pgvector, Qdrant, Chroma - see vector databases compared 2026. The short version: if you're already on Postgres, pgvector removes an entire service from your stack and is fast enough below ~10M vectors. Reach for a dedicated vector DB only past that scale or when you need hybrid search (BM25 + vector) out of the box.

When vector memory is the wrong choice

• Structured facts with a known schema (plan tier, signup date, region) - just use columns.
• Anything that needs exact match, not similarity - order numbers, ticket IDs.
• Small memory sets (under a few hundred facts per user) - a JSON blob loaded into the prompt is simpler, cheaper, and has zero retrieval error.

Don't reach for embeddings because it sounds sophisticated. A WHERE user_id = ? query that returns exact rows beats an ANN search that returns "probably relevant" rows every time the data is structured.

Summarization: compressing history without losing what matters

Summarization is the workhorse technique nobody talks about because it's unglamorous. The pattern: once a conversation crosses a token threshold, ask the model to compress the older portion into a dense summary, keep the summary plus the last few raw turns, and drop the rest.

SUMMARY_TRIGGER_TOKENS = 6000
KEEP_RAW_TURNS = 6

def maybe_summarize(history: list[dict]) -> list[dict]:
    if count_tokens(history) < SUMMARY_TRIGGER_TOKENS:
        return history

    to_compress = history[:-KEEP_RAW_TURNS]
    recent = history[-KEEP_RAW_TURNS:]

    summary = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=500,
        messages=[{
            "role": "user",
            "content": f"Summarize the key facts, decisions, and open "
                        f"questions from this conversation in under 200 words:\n\n{to_compress}"
        }]
    ).content[0].text

    return [{"role": "system", "content": f"Earlier context: {summary}"}] + recent

Two design decisions matter more than the code:

• Summarize on a token trigger, not a turn-count trigger. A 3-turn conversation with pasted logs can already be 8K tokens; a 30-turn back-and-forth of "yes/no" can be under 1K. Trigger on actual size.
• Never summarize the most recent turns. Summaries lose specifics (exact numbers, exact wording the user asked to preserve). Keep the last 4-8 turns raw and only compress what's older.

Claude's prompt caching changes the cost math here: if your summary + system prompt is stable across turns, caching it drops the repeated-read cost by ~90%, which sometimes means summarizing less aggressively is cheaper than the extra summarization calls.

Retrieval-augmented memory: the pattern that actually ships

In practice, "agent memory" in 2026 production systems is almost always retrieval-augmented generation (RAG) applied to a memory store instead of a document store. The pipeline is identical:

1. Embed the incoming query.
2. Retrieve top-k memory items filtered by user/tenant.
3. Re-rank if you have more than ~20 candidates (a cheap cross-encoder or even Claude Haiku as a re-ranker works).
4. Inject into the system prompt as structured facts, not raw transcript.
5. Optionally write new facts back after the turn completes.

The mistake I see most often: teams retrieve raw conversation turns instead of extracted facts. A raw turn like "yeah I guess Tuesday could work if the weather's ok" retrieved out of context six weeks later is nearly useless. An extracted fact - "user tentatively available Tuesdays" - is directly usable. Extract facts at write time, not read time. This is also where RAG for SMB and general RAG vs fine-tuning trade-offs overlap with memory design - the retrieval half of the stack is the same, only the corpus differs.

Forgetting: the feature nobody builds and everybody needs

Memory systems that only ever add rows degrade in three predictable ways: retrieval gets slower, retrieval gets noisier (old, contradicted facts compete with current ones), and storage cost creeps. Forgetting has to be a designed feature, not an accident of running out of disk.

Three forgetting strategies, roughly in order of effort:

• TTL expiry. Simplest: episodic memory expires after N days automatically. Works for support chat, fails for anything where "the user mentioned this once, three months ago, and it matters" is a real case.
• Contradiction overwrite. When a new fact conflicts with a stored one ("user is on Basic plan" vs. a new "user upgraded to Pro"), the write step should detect the conflict and replace, not append. This requires a small classification step - often just asking Claude "does fact B contradict fact A? yes/no" - before every write.
• Relevance decay. Score each memory by recency and access frequency; periodically prune the bottom percentile. This is closer to how human memory actually works and is worth the engineering only once you have real usage data showing retrieval noise is a problem.

Warning: never let an agent silently "forget" something the user explicitly asked it to remember. If a user says "remember I'm allergic to shellfish," that fact should be pinned - excluded from TTL and decay - not treated as an ordinary conversational fact.

Cost and design trade-offs

The honest answer for most SMB and mid-market agent builds: use structured SQL facts for anything with a schema, summarization for conversation history, and vector retrieval only for the genuinely unstructured long tail (support history, past complaints, free-text preferences). Reaching straight for a vector database because it's the "AI-native" answer is the single most common over-engineering mistake I see in agent projects - see common prompt-to-app mistakes for the wider pattern of this.

Also budget for the write side, not just reads. Every memory write that goes through an LLM call (fact extraction, contradiction checking, summarization) adds real per-turn cost. A rough estimate for a mid-volume support agent (5,000 conversations/month, average 15 turns): summarization + fact extraction typically adds $150-400/month in extra model calls on top of the primary conversation cost, depending on model choice. Using Haiku for extraction/summarization instead of Sonnet cuts that by roughly 80% with negligible quality loss for pure extraction tasks.

A minimal reference architecture

For a typical customer-facing agent (support bot, sales assistant, WhatsApp agent):

1. Session store (Redis, 24-48h TTL): last N raw turns, for continuity within a conversation.
2. User facts table (Postgres): structured, known-schema attributes, one row per fact, versioned by updated_at.
3. Semantic memory (pgvector or Chroma, only if you have real unstructured long-tail data): free-text facts extracted at end of session, filtered by user_id at query time.
4. System prompt assembly: session store + top-k semantic facts + relevant structured facts, assembled fresh every turn - never a growing accumulation.

This mirrors what a WhatsApp AI agent needs in practice, since WhatsApp sessions are inherently disjointed (users vanish for days, then reply to a message from last week) - memory design there is not optional polish, it's the difference between a usable agent and a broken one.

FAQ

What is AI agent memory?

AI agent memory is the set of mechanisms that let an AI agent retain and reuse information across turns or sessions instead of treating every message as the start of a fresh conversation. It typically combines short-term context (the current conversation), long-term storage (a database or vector store), and a retrieval step that decides what to inject into the prompt on each turn.

Do I need a vector database for agent memory?

Only if you have genuinely unstructured, high-volume memory data - free-text preferences, past complaints, unpredictable query shapes. If your memory is structured (plan tier, region, last order), a regular SQL table is simpler, cheaper, and more accurate than embeddings.

How much context should an agent keep versus summarize?

A common working rule: keep the last 4-8 raw turns for specificity, summarize everything older than that into a compact fact list, and trigger summarization on a token threshold (5,000-8,000 tokens is typical) rather than a fixed turn count, since message length varies wildly.

How does agent memory affect API costs?

Every token of retained history is re-sent (and re-billed) on every subsequent turn unless you use prompt caching or trim it. Summarization and selective retrieval typically cut per-turn token cost by 60-90% compared to appending full raw history, at the cost of some specificity.

Can Claude's 1M context window replace a memory system?

No - it removes one failure mode (hard truncation) but not the others: cost still scales with what you send, and multi-hop recall accuracy still degrades on very long transcripts. A 1M window is a safety margin, not a substitute for deciding what's actually relevant to inject.

Get help building this

If you're building an agent and the memory layer is the part that's stalling - or you inherited one that's already retrieving the wrong facts - we can look at it together. Book a free 30-minute call through the contact form or message us directly on WhatsApp.

Sources

• Anthropic engineering blog - context and memory management
• Anthropic Claude documentation - prompt caching
• Chroma documentation
• pgvector on GitHub
• LangChain memory concepts