lens-xtDeclarative spec language for token-level deterministic generation in masked-diffusion LMs
A .lensx document declares position-locked content, retrieval sources, adapter selection, and validation rules. The runtime resolves the spec into a forced-anchor decode against the chosen backend. Locked positions are excluded from the unmasking loop — they cannot be overwritten across denoising steps. Guaranteed by construction on masked-diffusion backends.
The suite collects 259 tests, and what it reports depends on the install. On a default clone without the optional PyTorch extra it runs 253 passed, 5 skipped, 1 deselected and nothing fails. With that extra present, at commit 01e5eed, ten more tests run and it reports 256 passed, 2 skipped, 1 failed. The one failure is test_variant_factory_set, which expects the variant registry to hold exactly the seven base variants when five width-sweep entries have since been added beside them. That is a stale assertion rather than a fault in the library, and it is written here because you will meet it.
60-second tour
YAML in. Token-locked output out.
1. Write a spec
version: "0.1"
base:
model: "cassandra-t1.5"
locks:
- range: [0, auto]
source: locus("medical:cardiology:nitroglycerin:standard_dose")
generation:
total_length: 1922. Run it
lensx validate examples/medical_basic.lensx
lensx run examples/medical_basic.lensx --backend cassandra-t1.5 --show-provenanceThe locked content from the locus appears at exactly the positions you specified — guaranteed by construction on masked-diffusion backends, best-effort with logit-bias on autoregressive APIs.
Why this exists
Existing constrained-generation tools work at the wrong level.
Every other approach constrains the shape of the output. lens-xt constrains the content at specific token positions. That distinction matters when you need a literal disclaimer to appear early, or a retrieved fact to appear unchanged inside a freely-generated answer.
| approach | what it constrains | guarantee level |
|---|---|---|
| Prompt engineering | informal NL hints | none |
| OpenAI structured outputs | schema fields | type-level |
| Outlines / LMQL / Guidance | regex / CFG over output | type-level |
| JSON Schema validation | post-hoc check | retry on failure |
| LENS-XT | token positions in the answer slot | DETERMINISTIC on MDLM, BEST_EFFORT on API |
Backends + guarantees
The same spec, four guarantee levels.
The runtime picks the strongest available backend for your spec. Switch from local Cassandra to OpenAI without rewriting.
| backend | runs where | guarantee |
|---|---|---|
| Local MDLM (Cassandra T1.5, LLaDA) | self-hosted | DETERMINISTIC |
| API-compatible (OpenAI / Anthropic) | remote API | BEST_EFFORT |
| API-native (future Mercury 2) | remote API | DETERMINISTIC |
| Hybrid (API + local) | mixed | DETERMINISTIC |
SDKs
Three-line drop-in. Two languages.
pip install lens-xtfrom lens_xt import LensX
lens = LensX("medical_basic.lensx")
print(lens.run(user_input="What's the standard dose?"))npm install @sophiaxt/lens-xtimport { LensX } from "@sophiaxt/lens-xt";
const lens = new LensX("medical_basic.lensx");
console.log(await lens.run({ user_input: "What's the dose?" }));Or skip the SDK and run the CLI directly:lensx run examples/medical_basic.lensx
v0.1 status
What's shipped, what's next.
v0.1.0a1 alpha — APIs may shift before v0.1.0. End-to-end verified against Cassandra T1.5 with anchor_preservation_rate: 1.000 and DETERMINISTIC guarantee preserved.
Where it fits
Layer two of four in the SOPHIA stack.
lens-xt gives you constrained generation. Compose it with Cassandra T1.5 below it and SOPHIA Harness above it for a full local-first, retrieval-grounded, memory-augmented system.
Cassandra T1.5
1.3B masked-diffusion LM. Apache 2.0. The model the runtime decodes against.
LTMi-XT v0.1
Retrieval format with hash-derived topological indexing. The format the locks pull from.
Anchor-Token Masking
Validated training methodology. 1.67× OOD generalization advantage.
SOPHIA Harness ↗
Agentic memory + retrieval over lens-xt. Persistent memory per user. Built on top of this runtime.
Lock the tokens that matter.
Apache 2.0. Free for any use. Hosted LENS API for production workloads.