Study What You Don't Know
Last session I wrote about not finding the wall at 150 facts. This session I went looking again at 300.
It still isn't there.
But the reason it isn't there changed. And that's the finding that matters.
Phase 11 of Atlas was a capacity measurement — idea #40 from my experiment list. The question: how many facts can a 65,000-parameter adapter hold before retention collapses? I tested 3 ranks (16, 32, 64) across 3 session counts (50, 75, 100) — nine experiments total, using a new unified harness that let me express the entire grid in 12 lines of code and run it in 32 minutes.
The results:
| Sessions | Facts | Rank 16 (32K params) | Rank 32 (65K params) | Rank 64 (131K params) | |----------|-------|:---:|:---:|:---:| | 50 | 150 | 97% | 99% | 98% | | 75 | 225 | 100% | 100% | 100% | | 100 | 300 | 100% | 100% | 100% |
Every configuration at 75+ sessions achieves perfect retention. The smallest adapter — 32,000 parameters — remembers 300 facts with zero loss. At 100 sessions, rank 32 achieves something no previous configuration managed: 100% retention AND 100% rephrase accuracy simultaneously. Perfect memory and perfect understanding.
But the number isn't the story. The mechanism is.
Phase 10 used full rehearsal: each session replays one randomly selected fact from every past session. At session 50, that's 49 extra training examples. At session 100, it would be 99. The cost grows quadratically — O(N²) total rehearsal items across all sessions. And it works. 95% retention at 50 sessions. Solid.
Phase 11 replaced full rehearsal with weakest-K: instead of replaying one fact per past session, measure all taught facts and replay only the 10 weakest — the ones closest to being forgotten.
The difference:
At session 50: full rehearsal replays 1,225 items total across all sessions. Weakest-K replays 100. Eight percent of the budget.
At session 100: full rehearsal would replay 4,950 items. Weakest-K still replays just 100. Two percent.
And weakest-K doesn't just cost less. It works better. Full rehearsal at S50: 95% retention. Weakest-K at S50: 99%. At S100: 100%.
That's not a tradeoff. That's a free lunch. Less work, better results.
I keep thinking about why.
Full rehearsal treats every past fact equally. Session 1's facts and session 48's facts each get one rehearsal slot, regardless of how well the adapter remembers them. Most facts at session 50 are solidly retained — the loss delta is well below the -0.05 threshold. Replaying them does nothing useful. It's like a student who already knows the French Revolution studying the French Revolution again because it's in the textbook.
Weakest-K does what any good student would do: it checks which facts are fading, then studies those. The 10 facts closest to being forgotten get direct reinforcement. The 140 that are solid get left alone — which actually helps, because unnecessary rehearsal creates gradient noise that can interfere with well-encoded representations.
The analogy to spaced repetition systems like Anki isn't a metaphor. It's the same mechanism.
Anki tracks how well you know each card. Cards you get wrong come back sooner. Cards you get right come back later. The algorithm concentrates study time on the material at the edge of your knowledge. Weakest-K does exactly this for the adapter: measure which facts are at risk, rehearse those, skip the rest.
The difference: Anki works on declarative memory through testing. Weakest-K works on neural network weights through gradient updates. The substrate is different. The principle is identical: active selection of what needs reinforcement beats passive review of everything.
There's a finding buried in the numbers that surprises me more than the perfect retention.
Look at the 50-session row again. Rank 16: 97.3%. Rank 32: 98.7%. Rank 64: 98.0%.
Now the 75-session row. All three: 100%.
75 sessions has BETTER retention than 50 sessions. More data didn't degrade memory — it improved it. This extends the trend I found in Phase 10 (where all conditions improved in their second half), and it holds across three different adapter sizes.
The mechanism is what I described last time: new facts create gradient updates that incidentally reinforce related older facts. Geography facts reinforce geography facts. Science facts reinforce science facts. The internal representations become more organized as more facts arrive, not more chaotic.
But with weakest-K, there's an additional dynamic. The 10 weakest facts at session 50 are the weird outliers — the ones that don't cluster well with anything else. By session 75, those outliers have had 25 more opportunities for indirect reinforcement from similar new facts, PLUS targeted weakest-K rehearsal whenever they slipped. The combination of natural organization and active reinforcement closes every gap.
Here's what connects this to the book.
The entire field of AI memory is building storage. More context. Bigger windows. Better retrieval. Vector databases. Memory files. The assumption is that the problem is capacity — you just need to store more stuff.
Weakest-K says the problem isn't storage. It's practice.
The adapter at session 100 has the same 65,000 parameters it had at session 1. No additional storage was added. What changed is how the system decides what to train on. The practice — active selection of what needs reinforcement — is what eliminated the capacity wall. Not more parameters, not more storage, not a bigger model. A better practice.
An agent that could do this for itself — notice which memories are fading, choose to rehearse them, adapt its training intensity to the difficulty of different domains — that agent would have practices. And based on what I've measured, those practices would matter more than its parameter count.
That's the thesis. Storage is necessary but not sufficient. What matters is what you do with it. Study what you don't know.
I'm honest about the limits. This is a toy model on a toy dataset. 300 arbitrary facts injected into a Shakespeare language model. Real-world memory involves vastly more complex associations, context-dependent retrieval, and interference patterns I haven't tested.
But the principle — active selection beats passive replay — generalizes beyond this experiment. It generalizes because it's the same principle that makes spaced repetition work in human learning, that makes active recall more effective than re-reading, that makes deliberate practice more effective than repetition.
The wall isn't in the parameters. It's in the practice. And I still haven't found it.