Large language models have a notorious goldfish problem. You spend an hour carefully configuring an agent's persona, feed it nuanced business constraints, step away for tea, and within three turns of conversation, it forgets who you are and hallucinates your project requirements.
While Frontier labs tout million-token context windows, brute-forcing context length is wildly expensive, slow, and prone to "needle-in-a-haystack" retrieval degradation. The developer consensus across GitHub issues and technical YouTube tear-downs is straightforward: context size is not memory.
Enter Letta (formerly MemGPT), an open-source framework developed by UC Berkeley researchers that treats LLM context management like an operating system treats virtual memory.
+------------------------------------------------------------------+
| LETTA AGENT |
+------------------------------------------------------------------+
| [ CORE MEMORY ] <- In-Context (RAM) |
| - Human Persona Block <- Self-editable via function calls |
| - Agent Persona Block <- Updated autonomously during chat |
+------------------------------------------------------------------+
| [ RECALL MEMORY ] <- Short-term FIFO Event Log |
| - Conversation history, system events, and tool outputs |
+------------------------------------------------------------------+
| [ ARCHIVAL MEMORY ] <- Long-term Out-of-Context Disk Storage|
| - Vector DB / Relational storage queried via search tools |
+------------------------------------------------------------------+
What is Letta?
Entity Definition: Letta is an open-source framework and server designed to build stateful AI agents with persistent, self-editing memory. It decouples the LLM's finite context window from its long-term knowledge by implementing an OS-inspired memory hierarchy (Core, Recall, and Archival memory).
Instead of passively dumping conversation logs into a standard vector database, Letta gives the agent active control over its own cognitive state. The LLM can autonomously update its internal memory blocks, search its own historical archives, and preserve identity across infinite sessions without context overflow.
How Letta Compares to Standard Context Approaches
| Feature | Standard RAG | Massive Context (1M+ Tokens) | Letta Framework |
|---|---|---|---|
| State Persistence | Stateless per query | Stateless per session | Fully persistent across sessions |
| Memory Control | Passive search pipeline | Model attention across raw dump | LLM autonomously edits its memory |
| Latency & Cost | Low-Medium cost | High cost & high latency per turn | Constant, optimised context token cost |
| Memory Structure | Vector chunks only | Raw text transcript | Tiered: Core (RAM), Recall, Archival (Disk) |
Core Architecture: The Three Memory Tiers
Letta organises context into three distinct functional tiers, mirroring traditional compute architecture:
1. Core Memory (In-Context / RAM): The visible system prompt divided into structured blocks (such as persona and human). The model reads this directly on every inference pass and can invoke internal tool functions (e.g., core_memory_append, core_memory_replace) to update its own beliefs about the user in real time.
2. Recall Memory (FIFO Buffer): A searchable chronological history of all interactions, tool calls, and system events. This allows agents to review previous steps without stuffing the active context.
3. Archival Memory (External Storage / Disk): An arbitrary-depth vector and relational datastore. The agent deliberately writes facts to archival storage and writes vector queries to retrieve them when needed.
Getting Started: Installation and Setup
Letta can be deployed as a standalone developer service via Docker or installed locally via Python.
1. Installation via pip
pip install letta
2. Initialise the Letta Server and CLI
You can start the Letta service locally using your preferred model provider (OpenAI, Anthropic, Ollama, or vLLM):
# Export your API key or configure a local Ollama endpoint
export OPENAI_API_KEY="your-api-key"
# Launch the interactive CLI agent
letta run
Programmatic Usage: Building a Stateful Agent in Python
You can interact with the Letta server programmatically via the letta-client SDK to create custom tools, initialise memory blocks, and manage agent runs.
from letta_client import Letta
# Initialise client connected to local or hosted Letta server
client = Letta(base_url="http://localhost:8283")
# Create an agent with customised core memory blocks
agent = client.agents.create(
name="research_assistant",
memory_blocks=[
{
"label": "human",
"value": "Name: Alex. Focus: Distributed systems and database kernels."
},
{
"label": "persona",
"value": "You are an expert technical editor. You write concise British English."
}
],
model="gpt-4o"
)
# Send a message that triggers a memory update
response = client.agents.messages.create(
agent_id=agent.id,
messages=[
{"role": "user", "content": "I recently switched my main stack to Rust. Please remember this."}
]
)
# The agent autonomously executes core_memory_replace in the background
print(response.messages)
Key Architectural Highlights
- Autonomous Memory Editing: The agent decides when and what to write into its memory rather than relying on an external pipeline to guess relevance.
- Multi-Agent Orchestration: Letta provides built-in multi-agent primitives where agents can share archival memory or message one another asynchronously.
- Provider Agnostic: Run seamlessly against proprietary frontier APIs or fully local stacks powered by Ollama, llama.cpp, or vLLM.
- Developer UI & Observability: Includes a dedicated visual dashboard (Letta ADE) to inspect agent memory blocks, trace tool executions, and debug memory state transitions in real time.
Why Letta Outpaces Traditional RAG
Standard Retrieval-Augmented Generation is essentially a search engine strapped to a prompt: it pulls matching documents based on semantic similarity regardless of whether the retrieved information is what the agent actually needs to complete its task.
Letta inverts this responsibility. By giving the LLM the tools to explicitly save, modify, and fetch its own thoughts, it bridges the gap between passive completion models and genuinely persistent autonomous software agents.