SOPHIA XT
Open · Apache 2.0

CassandraT1

A 1.3B masked-diffusion language model. Open Apache 2.0.

Cassandra T1 is our open-released 1.3B-parameter masked-diffusion edge model. Released checkpoint (epoch 5, FP16) is SHA-256 verified and downloadable from GitHub. Trained from scratch on consumer hardware. Companion paper introduces the anchor-token-masking training methodology — empirically validated to produce a 1.67× OOD generalization advantage over standard masking on retrieval-grounded fine-tuning. For production inference at scale, we use Mercury 2 (Inception Labs) as the preferred provider. SOPHIA XT does not currently ship other open models.

What it is

Built for retrieval-grounded generation.

Masked-diffusion architecture

Generates text by parallel iterative unmasking over fixed-length sequences. Admits forced-anchor decoding for deterministic identifier preservation.

Anchor-token training methodology

Validated to produce 1.67× OOD generalization advantage over standard masking on retrieval-grounded fine-tuning. Mechanism causally tested via V/O ablation.

LTMi-XT-native

Built to consume the LTMi-XT retrieval format end-to-end — same data substrate for retrieval and fine-tuning.

Edge-deployable

1.3B parameters fit comfortably on consumer GPU for inference. Fine-tuning runs on a single RTX 3090 Ti in 30-100 seconds per LoRA.

Architecture

Specifications.

Cassandra T1 uses standard transformer primitives (GQA, RoPE, RMSNorm, SwiGLU) configured for the masked-diffusion paradigm. The architecture is deliberately conservative — the contribution is in the training methodology and the integration with retrieval-grounded inference, not in novel architecture.

Parameters1.3B
ArchitectureMasked-diffusion transformer
Layers28
Hidden dimension2,048
Attention heads16 query · 4 key-value (GQA 4:1)
Position encodingRoPE (θ = 5×10⁵)
NormalizationRMSNorm
ActivationSwiGLU
Vocabulary32,768 BPE
Max context128K
Sliding window4,096
Inference denoise steps12
Training cross-entropy (ep 5)2.2561
LicenseApache 2.0
Live · Mnema V1 router demo running on this site

MnemaV1

A compositional language model. Multi-LoRA, capsule-routed, edge-deployable.

Mnema V1 (codename Zenith) introduces the NCN-Fold compositional training framework. A small shared base model is paired with many small LoRA specialists, and a coordinate-bearing capsule lattice picks the right specialist for each query at inference time. Adding a new domain takes one LoRA — about $0.50 and 30 minutes on a consumer GPU. The 360M Edge variant is deployed live on this site for the interactive demo; the 1.7B production size is in training.

Compositional, not monolithic

One small base + many small specialists. Each new domain is an additional small file (~17 MB LoRA), not a retrained model. Routing decisions are visible and inspectable per query.

Capsule lattice routing

A coordinate-bearing structure routes each query to the nearest specialist capsule via cosine retrieval on the pooled query embedding. 90% routing accuracy on the canonical 10-prompt benchmark.

CodenameZenith
CategoryCompositional Language Model (CLM)
Base (Edge variant deployed)SmolLM2-360M-Instruct warm-start
Production target1.7B (training in progress)
ArchitectureCompositional transformer + NCN-Fold routing
Specialists6 NCN Modules: arch · general · diag · math · code · dialog
File format.ncn (proprietary, rank-16 adapter)
RoutingCapsule lattice — cosine retrieval on 6-capsule lattice (v4)
Routing accuracy5 / 6 canonical (N=6 production); 80.2 % at N=32 off-line via GRC discriminant
Edge footprint~850 MB (base + 6 .ncn + lattice)
Inference (this VPS, CPU)~3-4 tok/s streamed, 4 vCPU AMD EPYC 9354P (fp32 N=6)
Training toolingComposer — ~$0.50 + 30 min per new Specialist on a 3090
Live demosophiaxt.com/mnema/ (this site)
Direct lineage

Where XE5 came from.

Mnema V1 selects among six NCN Module specialists with capsule routing. XE4-S is the paper that measured where that approach stops working and specifies what replaced it. XE5 is the architecture that diagnosis produced. The chain is one continuous line of work, and each step is published including the one that failed.

  1. Predecessor
    Mnema V1

    Six .ncn specialists, rank-16 LoRA deltas, chosen by cosine retrieval over a capsule lattice. Selection, not combination. Live on this site.

    Open the demo
  2. Inherits from Mnema
    XE4-S

    Takes the capsule-routed architecture Mnema runs and measures it to its limit. Selection held at two specialists and fell to 23.3 percent top-1 by three. The answer it specifies is dense composition over thirty-one specialists, selecting nothing.

  3. Current
    Sophia XE5

    Combines all thirty-five densely and conditions the weights on memory: w = compose(x, m) where a mixture of experts computes w = gate(x).

New architecture · first of its kind

SophiaXE5

A composer, not a gate. Specialist weights conditioned on memory the user writes at runtime.

A mixture of experts computes its weights as w = gate(x). Sophia XE5 computes w = compose(x, m), where m is where a beam descent through a hierarchical memory of runtime-written facts terminates. Because m depends on stored state the input does not determine, fixing the input does not fix the weights. That is a class of model a router cannot express, and it is the reason XE5 exists.

Cassandra T1 keeps its architecture deliberately conservative and puts the contribution in the training methodology. XE5 is the opposite model. Here the architecture is the contribution: a 147.5 M-parameter decoder-only transformer trained from scratch on one consumer GPU, in which all 31 specialists are combined on every token rather than selected. There is no top-k, no gating network, and no load-balancing loss anywhere in the system, because the assignment was fixed in the data instead of learned by a router.

Total and resident parameter counts come apart, which is what makes the design scale: the paper projects a 7B configuration at 6.98 B total and 1.95 B resident, where going from 57 capabilities to 120 at the same width moves the first number and leaves the second alone. The full technical report gives the measurement behind every design decision.

A composer, not a gate

Two linear maps and a softmax, 24,638 parameters, the smallest component in the system by an order of magnitude. Its one distinction is that the memory context vector m sits in its argument list. The memory term is zero-initialised, so an untrained model is exactly a router and the conditioning path has to earn every unit of its contribution.

Dense over all 31 specialists

Nineteen behavioural shapes and twelve topical branches carry weight at the same time, on every token. Across 178 probes the twelve specialists still sitting at initialisation take a mean 0.030% of the mixture mass and at most 0.122% on any single probe, so untrained capacity stays out of the forward pass.

Memory that is part of the network

6,752 nodes over a tree of derived keys, descended at beam width 2. Retrieved nodes enter the residual stream above the specialists and prepend to the keys and values of every block. Each node owns a learned value vector, so writing a fact adds parameters, and the descent prints as a path you can read.

Growth is a measured no-op

Attach a specialist and the output moves 5.96×10⁻⁷. Deepen one in place and it moves exactly 0.000×10⁰. Write a fact, 2.645×10⁻⁵. The composer's memory term, zero by construction. Four growth points, one rule, each with a named training stage that lets it earn weight.

CategoryCompositional language model, memory-conditioned
Composition rulew = compose(x, m) · dense over all 35 specialists
Parameters147.5 M at 35 specialists, trained from scratch
ArchitectureDecoder-only autoregressive transformer, pre-norm blocks
Trunk6 blocks, d = 384, 17.1 M, trained once and frozen
Attention heads6 heads of width 64
Feed-forward4d inner width, GELU
Position encodingRotary, base 10,000 · no learned table, so the window is not a parameter shape
Specialists35 × 3.55 M = 124.3 M · 23 behavioural shapes + 12 topical branches
Shared reconciliation stack2 blocks at d = 384, 3.55 M, zero-initialised at both residual writes
Composer24,638 parameters · two linear maps and a softmax
Output head6.29 M, weight-tied to the token embedding
Memory4,013 nodes after curation, derived keys, beam width 2 · +384 parameters per node
Memory value vectors4,013 × 384 = 1.54 M, written at runtime
Vocabulary16,384 byte-level BPE, full byte alphabet
Training context1,024 tokens
Routing lossNone · no top-k, no gating network, no load balancing
Specialist bankBatched matmuls · 11.5× the Python loop at k = 31, agreeing to 1.55×10⁻⁶
KV cacheToken-identical to the uncached path · 47.2 tok/s held flat at 241 tokens of context
Training hardwareOne RTX 3090 Ti · no pretrained base
Build cost~6.5 h elapsed, ~3.8 h of it GPU-active
General corpus2.107 B tokens, 1.500 B words of FineWeb-Edu
Technical reportPDF, 40 pages

Every figure above is read off the XE5 technical report, 19 September 2026. The full evaluation, ablations and the eleven-arm ordering search live in the paper and on the research page.