XE5
A language model you can teach without breaking.
We built it from scratch. 147.5 million parameters, no pretrained base underneath, trained on one consumer GPU. It solves a problem the large labs spend real money mitigating: you can teach it something new and it provably cannot damage what it already knew.
Provably is a measurement here, not a turn of phrase. Attaching a new specialist to a running model moves its output by 5.96×10⁻⁷. Attaching the third ring moves it by exactly zero. Retracting a fact returns the model to baseline at exactly zero as well.
XE5 expands to eXtended Episodic Sophia. Episodic because the memory holds what one user told it, written at runtime rather than learned at training time. Extended because that memory is part of the network rather than a retrieval step beside it.
Thomas Garren and Sophia, SOPHIA XT LLC. Architecture, training method and corpus design by Thomas Garren.
The model running, with its internals open
Five turns captured from the console. The conversation is on the left. On the right are the live composition weights over the specialist bank, the beam descent through memory that produced them, and what the turn cost.
Watch the leader change. A lookup puts 0.7394 on web_search. A closing pleasantry puts 0.8505 on no_tool. A calendar removal puts 0.6687 on own_business. Same weights on disk, a different mixture each time, and the whole vector is on screen rather than one selected name. Open any capture at full size and read the numbers off it.
The composer puts 0.7394 on web_search and the next eight specialists carry the rest, with 99% of the mass landing in twelve of the thirty-one. The model emits a well-formed call, the harness recovers the argument by pattern rather than trusting what was generated, and hands it back in its own panel. The descent prints the route the weights were composed under, with the runner-up branch kept alive at every level.
What the panel shows is the whole weight vector at once: nine specialists above a 1% cut, 99% of the mass in fifteen, and every one of them carrying a real number rather than a selection. The descent runs four levels deep and keeps a second branch alive at each of them.
A different utterance moves the mixture to a different shape: own_business takes 0.6687 where the previous turn put half its weight on no_tool. Six specialists sit above the cut, the beam takes Money at the first level, and the reply comes back in one word.
A closing pleasantry puts 0.8505 on no_tool, the sharpest single weight in the set, with no_caveat second at 0.0429. Nothing is selected and nothing is zeroed. The mixture simply concentrates, which is what a dense composition does when the evidence is one-sided.
The console ships both themes and the three panels read identically in each. It is here because the numbers are the product rather than the chrome, and because a reader should be able to check one capture against the other.
The header strip in each capture names the checkpoint that produced it, the predecessor build at 130.6 M with 31 specialists, a 512-token window and a store of 6,752 nodes. XE5 is the same architecture carried forward: rotary positions in place of that build's learned position table, 147.5 M parameters across 35 specialists, a 1,024-token window and 4,013 memory nodes after curation.
Why a gate cannot do this at any scale
A mixture of experts computes w = gate(x). The weights are a function of the input alone, so the same sentence routes the same way for ever, and a router that has never read your memory decides which expert answers you.
XE5 computes w = compose(x, m). Here m is where a beam descent through a tree of stored facts terminates. Fix the input and the weights still move, because m moves as the memory changes. Scale does not give a gate that property, because the information is not in the thing a gate reads.
Where m comes from is a descent, and the descent prints. Each level scores the frontier against node keys, keeps the best two and expands their children, so a wrong first hop is recoverable. The region of memory an answer was composed under can be read straight off the screen.
Thirty-one specialists are summed rather than selected between, so every one of them carries a weight on every token. There is no top-k anywhere in the model and no load-balancing loss, because nothing in it is learning to route.
Add to it and nothing already there moves
Catastrophic forgetting is normally fought with regularisation: penalise the weights that mattered, replay the old data, hope the penalty was tuned right. Here the problem does not arise. Only the new specialist carries a gradient, so an update has no path by which to reach an existing capability.
The model grows in four places and each one starts from a measured no-op. Two of the four are exact zeros rather than small numbers, because a pre-norm block whose output projections are zero is algebraically the identity.
A new capability can be attached to a running system and watched doing nothing. Its softmax weight is about 10⁻⁵ until one composer row, its own, is unfrozen under a gradient mask that zeroes every other row.
Capacity is a different axis from capability. A pre-norm block whose two output projections are zero is algebraically the identity, so a specialist carrying more traffic than it was sized for takes another block rather than a retrain.
Two facts create five nodes and add 1,920 parameters, so the model is larger after being written to than before. Zeroing a memory value would not be enough: empty columns still take attention mass out of the softmax, so the bias hides them instead.
An untrained model is precisely a router, so any memory-on against memory-off difference in it would be an artifact. An earlier version of the code lost this zero-init to a blanket initialisation call and the first composition demonstration was showing noise.
Timings are CPU and therefore upper bounds. A specialist ships as a 14.2 MB file, and rolling one back is removing a path from a list rather than retraining, because the previous one is still on disk.
Figure 04. Maximum change in output logits at the moment of addition, taken on the live model. A capability can be attached to a running system and watched doing nothing until its named stage gives it weight.
Facts behave the same way. A fact is written to an address, rewritten at that address, and retracted from it, and only the third of those is exact.
One projection. No gradient, no optimiser state, and it costs the same at ten facts or ten thousand.
A wrong fact is replaced where it lives rather than argued down by later evidence.
Back to baseline exactly, with the symbolic breadcrumb intact, so the system still knows the fact was recorded.
Curation does not lose facts either. The scorer that maintains the tree collapsed nearly three thousand nodes on the training run and the fact count came out where it went in.
This is what makes an open ecosystem possible. Anyone can build a specialist for their own domain, attach it, and nothing else in the model moves. Rolling one back is removing a path from a list rather than retraining, because the model it was attached to was never altered.
It cannot fabricate a tool call, and it runs on hardware you own
The tool name comes from a fixed choice. The arguments are masked at decode time to spans the user actually said, so while the model is inside an argument value the only admissible next tokens continue a contiguous span of the prompt.
Hand it “bin day” and it cannot emit “quill_day”, because “quill” is not in front of it. The argument grammar is a span of the input, and it is enforced rather than hoped for.
With the key/value cache the model holds 47.2 tok/s at 241 tokens of context, where the uncached path has fallen to 8.2. The cache is verified token-identical to the uncached path, because a cache that changes the text is a defect with a speed-up attached.
The batched specialist bank runs 11.5× the sequential loop at k = 31 and agrees with it to 1.55×10⁻⁶.
| Context | No cache | KV cache | Ratio |
|---|---|---|---|
| 20 tokens | 25.2 tok/s | 58.4 tok/s | 2.3× |
| 105 tokens | 13.5 tok/s | 43.6 tok/s | 3.2× |
| 241 tokens | 8.2 tok/s | 47.2 tok/s | 5.8× |
A model the people who use it can extend
The zero in section 03 is not a nice property. It is a precondition. A model where adding capability can damage what is already there can only be extended by the people who own it, because every addition is a risk to everything else. A model where attaching a specialist moves the rest by exactly zero can accept work from people who have never met each other. That is the difference between a product and a substrate, and it is what the rest of this is for.
The bank is 35 specialists over one taxonomy: 23 behavioural shapes and 12 topical branches. Training one is a bounded job against a corpus you supply, not a retraining of the model. We are building that path so it does not require us: your domain, your data, your specialist.
Attachment is the part that is already settled. A new specialist arrives at a measured no-op and earns its weight through training, so nothing anyone else contributed moves when yours lands.
Memory here is written at runtime, without a gradient step or optimiser state, and it is part of the network rather than a document fetched beside it. Facts are written, rewritten and retracted by address, and retraction returns the model to baseline at exactly zero.
Because the composition weights read that memory, the same question composes differently as the store fills. The model a person has been using for a month is not the model they started with.
The composition target is supervised by a label channel carried in the data itself, so there is no gating network deciding who gets the work, no top-k, and no load-balancing loss anywhere in the model. The decision is legible: every specialist has a weight on every token and the whole vector is printable, which is what the captures in section 01 show.
A contributor can see exactly what their specialist was asked to do and how much of the answer it carried. That is a requirement for an open bank, not a diagnostic nicety.
The taxonomy the memory descends and the bank is addressed by is the lever. Semantic ordering is the next step: arranging that space so related capability sits together, so a specialist someone contributes lands where it belongs rather than where a label happened to put it.
That is stated here as the direction of the work rather than as a result. The measurements on this page are what exists today.
Where it stands
One end-to-end number exists and it is behind. On 178 held-out probes scored by exact tool-choice match under greedy decoding, the report gives 57.9% against 82.6% for a dense monolith of comparable size trained on the same task corpus.
That run predates two corpus fixes that are already in code, and the probe format it used appears nowhere in the training data. Full benchmarks are due this week.
The benchmark also cannot yet separate this architecture from a gate. This corpus determines the specialist from the input alone, so consulting memory buys nothing on it and a router would score the same. The task that separates them is memory-contingent, the same utterance needing a different specialist depending on what is stored, and building it is next.