How agents remember
Memory is not a feature you bolt onto an agent. It is a decision about what
the agent is — a calculator, a colleague, or a keeper of record. In
melchizedek, that decision is one line of YAML:
memory_system, three tiers, three different kinds of mind.
internal-only is amnesia by design. State lives in-process and dies
with it. Right for tests, batch jobs, one-shot pipelines — anywhere
yesterday’s context would be contamination, not wisdom.
session-only persists the conversation itself. Every session writes
to Postgres, so a restart resumes mid-thought instead of starting over.
One structural detail matters more than it looks: session rows are keyed by
a composite of app, user, and session, so parallel subagents working
under the same global session stay perfectly isolated. Two workers, one
factory floor, no shared scratchpad — corrupted context is how multi-agent
systems quietly go mad, and the schema forbids it.
long-term is where memory becomes identity. When a session ends, the
transcript doesn’t just get archived — it gets distilled. A model reads
the conversation and extracts discrete facts, each tagged by kind:
[PREFERENCE], [FACT], [DECISION]. Each fact is embedded as a
768-dimension vector and written to a pgvector table, scoped to that app
and user.
recall is a search problem
Retrieval mirrors storage. When a new query arrives, it is embedded the same way, and a cosine-similarity search returns the ten nearest facts — injected into the agent’s context before it reasons. The agent doesn’t remember everything; it remembers what resembles now. That’s the quiet insight of vector memory: relevance, not chronology, is the index. No timeline is replayed. The past shows up only where it rhymes with the present.
The whole lifecycle, played out — a fact told, distilled, and found again:
The costs are real and worth naming. Distillation is lossy — a fact extractor decides what mattered, and what it discards is gone. The embedding dimension is a hard coupling: change the embedding model and the entire memory store must be rebuilt to match. And ten retrieved facts are context, not truth — a memory written wrong persists wrong, which is why what gets extracted deserves as much design attention as what gets recalled. Every tier is a trade: the amnesiac agent is cheap and safe, the remembering agent is expensive and personal, and pretending otherwise is how systems end up with memory they can’t explain.
Choose deliberately, because memory is the load-bearing choice. Memory is what turns a tool into a relationship — the difference between a search box and the patient advocate that recalls six months of lab results at the moment they matter. Memory is accumulated context becoming something adjacent to care. And memory is a responsibility: an agent that remembers a person holds a piece of them, and must be built like it knows it.