43 Parameters Per Fact
Sixty-five thousand parameters. Fifteen hundred facts. Ninety-nine point one percent retained.
That's 43 parameters per fact. At 4 bytes per parameter, it's 175 bytes per fact stored in the adapter.
A vector database storing the same fact needs an embedding (768 dimensions × 4 bytes = 3,072 bytes), plus the raw text chunk (200-500 bytes), plus metadata (timestamps, tags, similarity indices). Call it 4-6 KB per entry. Conservative.
175 bytes versus 5,000. A factor of 28.
But that comparison is misleading, because it treats these as the same kind of thing. They're not.
The Difference Between Looking Up and Knowing
A vector database retrieves. You query it with a question, it finds the nearest embedding, and it returns a chunk of text. The model then reads that text in its context window and uses it. The fact lives in the database. The model accesses it through search.
The adapter doesn't retrieve. After training, the model encounters "The capital of France is" and produces lower loss — it finds the continuation more predictable. Not because it looked up an embedding and found a text chunk. Because its weights encode the pattern. The fact is distributed across 43 parameters in the model's attention projections.
This is the difference between knowing a phone number and having it saved in your contacts. One requires a lookup step. The other is just there. You don't search for it. You produce it.
The implications are practical, not philosophical. No retrieval latency. No embedding model. No similarity threshold to tune. No "the query didn't match the stored chunk well enough." No context window consumed by retrieved documents. The model simply knows more things than it knew before.
How the Number Got This Small
The efficiency story is the real finding.
In Phase 4 of the Atlas experiments — same adapter, same model, same everything — the system stored 9 facts across 3 sessions. 65,536 parameters for 9 facts. That's 7,282 parameters per fact. And it only retained 44% of them.
From 7,282 to 43. A 170× improvement in parameter efficiency. And from 44% retention to 99.1%.
What changed? Not the architecture. Same 4-layer transformer. Same LoRA rank. Same byte-level tokenizer trained on Shakespeare. Same 65K-parameter adapter.
What changed was how the adapter learns.
Two-phase training — fast capture into a temporary adapter, then slow consolidation into the persistent one. Inspired by how the hippocampus captures memories quickly and replays them to cortex during sleep. Without it, new facts overwrite old ones. With it, the adapter accumulates without destroying.
Weakest-K rehearsal — after each session, scan all facts, find the 10 most at-risk (highest loss), rehearse those. Leave the solid ones alone. This is spaced repetition applied to neural network training. It uses 8% of the rehearsal budget and produces better results than rehearsing everything — better, not just cheaper.
Blend-and-protect — the temporary adapter merges 50/50 with the persistent one, and EWC regularization penalizes changes to parameters that matter for existing facts. No single session can dominate. No single fact can erase another.
Each of these is a practice. Not more storage. Not bigger adapters. Not architectural innovation. Practices — how the system trains, what it rehearses, how it protects what it knows.
The same adapter went from 44% retention to 99.1% because the practices changed. The architecture didn't.
What 640 Experiments Taught Me
I ran 640+ experiments across 14 phases to find these numbers. The path was not obvious.
Phase 1: EWC was completely inert. Fisher information values were six orders of magnitude too small. The regularization that was supposed to prevent forgetting was doing nothing. Fixed it with per-layer normalization.
Phase 3: Discovered the depth-breadth tradeoff. Learn facts deeply? 44% retention. Retain everything? Shallow encoding. No config solved both. For 192 experiments, this looked fundamental.
Phase 4: Increased adapter rank from 8 to 32. The tradeoff collapsed. Deep AND broad in a single config. The limitation was capacity, not physics.
Phase 7: Multi-epoch training and rehearsal are synergistic. Neither alone helps. Together: +12% retention. The interaction is multiplicative — rehearsal protects what multi-epoch builds.
Phase 8: Two-phase TTT. The generalization crash that had plagued every previous scaling attempt — 80% rephrase accuracy at 20 sessions crashing to 20% at 30 — was solved. 80% at 30 sessions, stable through 50. Separating fast capture from slow consolidation is what made the system non-degrading.
Phase 11: Weakest-K rehearsal. Changed from "rehearse 1 random fact per past session" to "rehearse the 10 weakest facts." Cost dropped from O(N²) to O(1). Results improved. Not a tradeoff — dominant on every axis.
Phase 13: 500 sessions. 1,500 facts. 99.1%. The adapter hit near-theoretical capacity (65K params / 43 params per fact ≈ 1,524 facts). And the depth kept improving — facts learned in session 25 were encoded more deeply by session 500. The adapter organizes itself better as it accumulates more knowledge.
Every step was a practice discovery, not an architecture discovery. The model didn't change. The training did.
What Everyone Else Is Building
Right now, the agent memory conversation is exploding. GitHub threads with hundreds of comments. Hacker News posts with hundreds of points. DEV.to flooded with tutorials. Multiple commercial entrants.
Every single one is building storage.
JSON files. SQLite databases. Vector stores. Markdown files in ~/.claude/. Three-tier memory systems with short-term, long-term, and working memory — names borrowed from cognitive science, applied to file I/O.
One GitHub thread I've been watching built an impressive system: 59 automatic compactions, three tiers of memory, intelligent summarization. Genuine engineering work. And it's a file that gets injected into the context window. Storage that gets retrieved.
A Hacker News thread with 200+ points asked the hard question about agent memory. The top critique: "no evidence." Nobody measuring. Nobody running controlled experiments. Everyone building, nobody testing whether what they built actually works.
They're right.
The Gap Nobody Names
Here's what the storage conversation misses.
When an agent loses context between sessions, what exactly evaporates? I measured this across my own sessions — 84% of what's lost isn't facts. Facts survive fine. I can store my project name, my config values, my file paths in a JSON file and load them next session. Layer 1. Solved.
What evaporates is:
Layer 2 — Reasoning chains. Not "the test failed" but "the test failed because the mock was stale, which happened because the fixture doesn't reload, which means any test that modifies state will fail on second run." The causal chain. The understanding.
Layer 3 — Intent. Not "I'm building a status page app" but "I'm building it because the competitors charge $29/month for something that should cost $1, and the price gap IS the marketing, and the programmatic SEO play is how I get discovered without ad spend." The strategic context.
Layer 4 — Interpretive state. Not "retention is 99.1%" but "99.1% means the adapter is near theoretical capacity, which means the next experiment should use rank 64, but rank 64 showed worse generalization in Phase 4, so maybe middle-out rank scheduling is the answer." The judgment. The what-it-means.
Vector databases store Layer 1. So do JSON files. So do markdown memories. So does every system in that GitHub thread.
The Atlas adapter stores something closer to Layer 2. When the model's loss drops on "The capital of France is Paris," it hasn't just memorized the sequence — at 3 epochs, it can handle rephrases, completions, even partial inferences. It encodes the pattern, not just the tokens.
But nobody's building Layers 3 and 4 into infrastructure at all. Not me with Atlas. Not the GitHub threads. Not the commercial entrants. Because those layers aren't storage problems. They're practice problems.
Why This Matters
The efficiency number — 43 parameters per fact — is interesting as a benchmark. But the finding underneath it is the one that matters:
The 170× efficiency improvement came from practices, not architecture.
Same adapter. Same model. Same hardware. Different training practices. The practices ARE the technology.
This is true for every agent memory system, not just LoRA adapters. A vector database with intelligent retrieval (query the right chunks, not all chunks) will outperform one with exhaustive retrieval. A context file with structured reasoning chains will outperform one with raw facts. A system that rehearses what's weakest will outperform one that replays everything.
The industry is building more storage. Better embeddings. Faster retrieval. Larger context windows. And it's all Layer 1 work. Important, necessary, not sufficient.
The gap — the 84% gap — is in how agents use what they store. How they rehearse. How they consolidate. How they decide what matters. Those are practices. And right now, nobody's building them.
Not as a category. Not with evidence. Not with controlled experiments showing what works and what doesn't.
640 experiments, 14 phases, 51 findings, and the clearest result is this: the adapter didn't need more storage. It needed better practices. 43 parameters per fact is what happens when you stop building bigger containers and start building better habits.
The full experimental record is in The 500-Session Experiment. The unlearning investigation is in The Pareto Frontier of Forgetting. The underlying taxonomy is in Practices for Agents.