Cassandra T1.7 — training tonight
Continue-trained from SmolLM2-1.7B-Instruct (HuggingFace, Apache 2.0, exact size match), under our anchor-token masking training discipline, using Unsloth + Liger Kernel on a single RTX 3090. First usable checkpoint in 24-72 hours. Sequential vision + reasoning bolt-ons follow. No new framework needed — proven tools, applied to our published methodology.
Stack
Built from proven tools, not hand-rolled CUDA
Base model
SmolLM2-1.7B-Instruct
HF Apache-2.0 · matched size · Unsloth 4-bit
Training framework
Unsloth + Liger Kernel
Triton kernels · 4× memory · 2-5× speed
Adapter method
QLoRA · r=64
4-bit base + LoRA adapters · 22 GB VRAM ceiling
Objective
Anchor-token masking
Zenodo 10.5281/zenodo.20496783 · V/O-direction load-bearing
Honest engineering math
Why this works in days, not months
From-scratch training of a 1.7B model on a single 3090 takes ~165 days at 100% utilization. Reality. The win comes from continue-training a strong base model (50-200× compute savings) and using Triton-kernel-based stacks (Unsloth + Liger) instead of vanilla PyTorch (2-5× speed, 60-70% less memory). All independently benchmarked.
1.7B from scratch on 3090
~165 days
100% util · unachievable in days
Continue-train + Unsloth + QLoRA
24-72 hrs
first usable adapter checkpoint
Speed vs vanilla HF Transformers
2-5×
Unsloth confirmed (independent benchmarks)
Memory vs FP16 baseline
60-70% less
QLoRA 4-bit base + adapters
Training order
Sequential, not simultaneous
Doing text + vision + reasoning in one training run is what blows training budgets. Sequential ships.
Phase 1 — Continued text training
Days 1-3SmolLM2-1.7B base + LoRA adapters. Train on diagbuddy appliance corpus (curated structured data) + conversation set. Loss target: < 1.6 (Mercury Coder Small convergence point). Eval every 500 steps with pooled corpus overlap against C1-C7 corpora.
Phase 2 — Vision encoder bolt-on
Week 2Frozen SigLIP-base (400M, Google, Apache-2.0) for the visual modality. Connect to the LM via a small projection layer (the only trainable surface). Vision data: appliance fault-mode photos + service-manual diagrams.
Phase 3 — Reasoning fine-tune
Week 3-4DPO or RFT on a small reasoning dataset (~5K pairs). Aligned to diagnostic-reasoning trace format. Goal: structured chain-of-thought for appliance troubleshooting + cross-domain QA.
Training + evaluation corpora
Real data, deterministic paths
The training corpus is the curated diagbuddy appliance knowledge set. Evaluation runs against the canonical C1-C7 corpora used in the V1-V6 lens-audit pipeline — corpus_hash references file paths for deterministic reproducibility.
| ID | Path | Use |
|---|---|---|
| appliance | diagbuddygo/corpus/curated/appliance-knowledge-v1.jsonl | Curated structured appliance corpus — breadcrumb + statement + parts + confidence + year-range. |
| c1 | cassandra-eval/C1.json | Cold storage domain · foundation-v0 series eval. |
| c2 | cassandra-eval/C2.json | Architecture spec · T2 + v2_triple eval. |
| c3 | cassandra-eval/C3.json | Mixed memo · T2 + v2_triple. |
| c5 | cassandra-eval/C5.json | Medical · cross-domain validation. |
| c6 | cassandra-eval/C6.json | Legal · cross-domain. |
| c7 | cassandra-eval/C7.json | Cooking · cross-domain · V1-V6 running. |
Fund the T1.7 run + cluster time for T2 v0.1
T1.7 runs locally on existing hardware. T2 v0.1 (the first investable milestone per the internal training plan) is a 3B-param continuation, ~50B tokens, ~$5-15K cloud spend on H100s. We have the data, the methodology, the eval suite, and the training scripts. We need the cluster time.