You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
hslm: retrain the HSLM model from t27 -- full-model training binary, pinned corpus, held-out PPL (cell 25) #10372
Part of #10024 (our own model), cell 25 "t27c --backend ml": a training binary over the whole HSLM model, not one head slice.
Boundary
New specs only: specs/nn/hslm_tokenizer.t27 (the 729-token byte-level BPE of trinity-training), specs/ml/hslm_corpus.t27 (text file -> token file, train and held-out), specs/ml/train_hslm.t27 (main() trains nn/hslm end to end through use, writes an MSLH checkpoint, reports held-out loss).
Edits to specs/nn/hslm.t27 only where the training binary needs a hook (an update step on the weights), with its tests kept green.
No hand-written foreign code. Building runs on the Railway lab, never on a Mac.
Scenarios
The best HSLM run, R5 (PPL 2.96 at ctx 27, gHashTag/trinity docs/experiments/FOUND_EXPERIMENTS_SUMMARY.md), was wiped by EXP-002. Reading its sources shows that 2.96 was exp(loss) of ONE training minibatch at about 32K steps, last position only, best of many seeds (the same config ranged 2.96 to 464), on a TinyStories file whose bytes were never recorded. It cannot be checked as stated.
So "restore" means: retrain the same architecture from t27, on a pinned corpus, and report a number that can be checked.
Requirements
Corpus pinned by sha256: TinyStoriesV2-GPT4 (roneneldan/TinyStories on Hugging Face, CDLA-Sharing-1.0), a fixed train prefix and the official valid file as held-out.
Tokenizer: the 469 merges of trinity-training src/hslm/bpe_merges.zig @ 3e9648c1817a97c5f9b8c727b0f8e2291928eead, as t27 data; tests pin encodings that the Zig tokenizer also produces.
Training: R5's golden config (gHashTag/trinity docs/lab/papers/hslm/golden-config.md): ctx 27, batch 66, lr 1e-3 with 2000 warmup and cosine decay, grad clip 1.0, weight decay 0.01, a pinned seed. Loss on the last position as R5 did, and also full-sequence next-token loss on held-out data.
Every number published comes from a log or a file whose sha256 is in the spec.
Success Criteria
t27c compile specs/ml/train_hslm.t27 --backend ml builds a binary on the lab that trains the 3-block model, writes a checkpoint and reads it back.
A run of at least the step count R5 reached reports held-out loss and PPL, with the train-minibatch figure next to it so it can be compared to 2.96.
The model and a .t27 model card are published on Hugging Face; own_model cell 25 cites them.
Closed by the issue-shape gate (specs/policy/issue_shape.t27, #7498): an issue must have the template's sections. Edit the body; an edit that passes reopens it.
missing: ## Requirements -- numbered obligations, FR-001: ... MUST ...
Activity
github-actions commentedon Oct 18, 2026
Closed by the issue-shape gate (specs/policy/issue_shape.t27, #7498): an issue must have the template's sections. Edit the body; an edit that passes reopens it.
FR-001: ... MUST ...spec(hslm): run the layout checks on variables so none is vacuous (Refs
spec(hslm): a lone newline is an empty story, a blank line is text (R…
spec(ml): train_hslm -- drop lone ; lines (Refs #10372)
spec(ml): train_hslm -- no duplicated bodies: read float params from …
Merge origin/master into feat/train-hslm (Refs #10372)
Merge origin/master into feat/train-hslm: types ratchet fix #10390 (R…
train_hslm: train on the loss of all 27 positions of a window (Refs #…
t27b ledger: name specs/ml/train_hslm.t27 (blocked, ExprCall(undeclar…
Merge origin/master into feat/train-hslm
t27b ledger: cap 131 = master's 130 plus specs/ml/train_hslm.t27
Merge origin/master into feat/train-hslm
t27b ledger: cap 133 = master's 132 plus specs/ml/train_hslm.t27
t27b ledger: cap 135 = master's 134 plus specs/ml/train_hslm.t27
t27b ledger: cap 137 = master's 136 plus specs/ml/train_hslm.t27
t27b ledger: cap 139 = master's 138 plus specs/ml/train_hslm.t27
spec(ml): train_hslm -- main() trains the whole 3-block HSLM with STE…
spec(hslm): nn/hslm_tokenizer and ml/hslm_corpus -- the 729-token BPE…
ml: ngram_hslm -- unigram, bigram and trigram baselines on HSLM's hel…
spec(ml): eval_hslm and ngram_hslm -- HSLM held-out loss at step 8000…