Agent Landscape #4: Memory — Beyond Context Window Overflow

The Problem

You ask your agent to continue yesterday's research. It has no idea what you're talking about.

You've told it three times you prefer TypeScript over Python. It keeps suggesting Python.

It's been working on a complex task for 40 messages. The early context — including why it chose this approach — has been truncated. It contradicts its own earlier decisions.

These are all memory failures, and they happen because:

1. Context windows are finite. Even 200k tokens fills up after 50 tool calls with results.

2. There's no persistence between sessions. Close the tab, lose everything.

3. No retrieval mechanism. Even if facts are stored somewhere, the agent can't find them at the right moment.

4. No forgetting mechanism. Outdated facts pollute retrieval (your old address, a changed preference).

Three approaches exist. Each solves a different layer of this problem.

Research & Industry Context

Key Papers

MemGPT (Packer et al., 2023, UC Berkeley) — the foundational paper for agent memory. Key insight: treat LLM context like virtual memory — the agent itself manages what's "paged in" and what's "paged out" via explicit memory operations. Directly led to Letta.

Generative Agents (Park et al., 2023, Stanford) — simulated town of 25 agents with memory streams, reflection, and planning. Demonstrated that agents with structured memory (observation → reflection → plan) exhibit emergent social behaviors.

RAG (Lewis et al., 2020) — Retrieval-Augmented Generation. Instead of cramming everything into context, retrieve relevant chunks at query time. The foundation for long-term memory in all agent systems.

Voyager (Wang et al., 2023) — skill library as memory. Instead of remembering raw experiences, extract reusable procedures. This is the pattern Hermes and OpenClaw implement.

Sleep-Time Compute (Letta, 2024) — agents consolidate and reorganize memories during idle periods (between user interactions). Like how human sleep consolidates short-term into long-term memory.

Self-RAG (Asai et al., 2023) — agent decides when to retrieve, what to retrieve, and whether the retrieval is relevant. Adaptive retrieval rather than always-retrieve.

How Industry Solves This

Anthropic (Claude): Projects feature — user-configured persistent instructions and context loaded every session. Plus conversation compaction for within-session memory. Simple, effective, not agent-managed.

OpenAI (ChatGPT Memory): Automatic memory extraction — model identifies facts worth remembering, stores them, retrieves on subsequent conversations. User can view/edit/delete memories. Passive extraction, not agent-managed.

Google (Gemini): Gems — persistent instructions + style. Notebook LM for long-form context. No explicit agent memory system.

Claude Code: File-based memory system — CLAUDE.md files and per-project memory that persists across sessions. Agent reads these at conversation start. Hybrid: some auto-saved, some user-configured.

Cursor/Windsurf: Codebase indexing as implicit memory — the agent "remembers" the full codebase via embeddings and retrieval. Not conversational memory but functional equivalent for coding tasks.

Common production patterns:

• Tiered storage (fast in-context + searchable long-term + archival)

• Explicit memory operations as tool calls (not passive)

• Compression/summarization for old context

• User control (view, edit, delete what the agent remembers)

Open Source Solutions

Letta (formerly MemGPT) — 23k stars

What it is: A memory-first agent platform from UC Berkeley research. Core insight: give agents explicit control over their own memory, like an OS manages virtual memory.

Strengths:

• Most sophisticated memory system in any agent framework

• Agents explicitly manage their own memory (read, write, search as tool calls)

• Sleep-time compute — reorganize memories during idle periods

• Portable memory — agents migrate across machines with memory intact

Limitations:

• Requires running a Letta server (not serverless)

• No container orchestration or tool sandboxing

• Memory is the product — everything else is secondary

Key insight: Memory must be a first-class primitive that agents actively manage — not a passive append-only log that overflows.

Mem0 — 58k stars

What it is: A memory layer that bolts onto any existing framework. Simple API: add(), search(), get_all(). Handles embedding, storage, retrieval with automatic deduplication.

from mem0 import Memory
m = Memory()
m.add("User prefers dark mode and concise responses", user_id="u1")
results = m.search("user preferences", user_id="u1")

Strengths:

• Easiest integration (3 lines of code to add memory to any agent)

• Framework-agnostic (LangChain, CrewAI, custom)

• Automatic deduplication and conflict handling

• Managed service option (no vector DB ops)

Limitations:

• Memory-only — no execution, tools, or sessions

• Black-box management (you don't control what gets remembered)

• Not designed for session-scoped memory (global per user/agent)

Key insight: The API simplicity is the model to follow. But memory can't be external-only — it needs to integrate into the agent's execution loop.

Vector Databases (Chroma, Qdrant, LanceDB)

Storage engines for embedding vectors. The infrastructure layer for RAG and semantic search.

The relationship to agents: These are backends, not solutions. They store vectors and answer similarity queries. They don't know about agents, sessions, or memory lifecycle. An agent memory system uses a vector DB — it isn't one.

The Three-Layer Problem

Production agents need three kinds of memory that no single project fully addresses:

1. Working Memory (in context window)

• What the agent is currently doing

• Recent conversation turns, active task state

• Challenge: Context window is finite. What gets evicted? When?

2. Long-term Memory (searchable, persistent)

• User preferences and patterns

• Learned facts from past sessions

• Skills and procedures discovered over time

• Challenge: Retrieval quality. Stale memories. Conflicting facts.

3. Session Memory (per-conversation, structured)

• Full conversation history for a given task

• Tool call results and intermediate state

• Compressed summaries for replay

• Challenge: Grows linearly. Compression loses information. When to summarize?

Comparison

Key Takeaway

The memory problem is unsolved because it spans three layers:

1. What to remember (working memory management)

2. Where to store it (vector DB, KV store, event log)

3. When to recall it (retrieval strategy per turn)

Letta is the most complete answer but requires buying into their full platform. Mem0 solves storage/recall but not management. Vector DBs are plumbing, not solutions.

Production agents need a memory interface that supports all three layers, backed by pluggable storage, with agent-controlled read/write/forget operations.