Skip to content

hslm: retrain the HSLM model from t27 -- full-model training binary, pinned corpus, held-out PPL (cell 25) #10372

Closed as not planned

Description

@gHashTag

Part of #10024 (our own model), cell 25 "t27c --backend ml": a training binary over the whole HSLM model, not one head slice.

Boundary

  • New specs only: specs/nn/hslm_tokenizer.t27 (the 729-token byte-level BPE of trinity-training), specs/ml/hslm_corpus.t27 (text file -> token file, train and held-out), specs/ml/train_hslm.t27 (main() trains nn/hslm end to end through use, writes an MSLH checkpoint, reports held-out loss).
  • Edits to specs/nn/hslm.t27 only where the training binary needs a hook (an update step on the weights), with its tests kept green.
  • No hand-written foreign code. Building runs on the Railway lab, never on a Mac.

Scenarios

  • The best HSLM run, R5 (PPL 2.96 at ctx 27, gHashTag/trinity docs/experiments/FOUND_EXPERIMENTS_SUMMARY.md), was wiped by EXP-002. Reading its sources shows that 2.96 was exp(loss) of ONE training minibatch at about 32K steps, last position only, best of many seeds (the same config ranged 2.96 to 464), on a TinyStories file whose bytes were never recorded. It cannot be checked as stated.
  • So "restore" means: retrain the same architecture from t27, on a pinned corpus, and report a number that can be checked.

Requirements

  • Corpus pinned by sha256: TinyStoriesV2-GPT4 (roneneldan/TinyStories on Hugging Face, CDLA-Sharing-1.0), a fixed train prefix and the official valid file as held-out.
  • Tokenizer: the 469 merges of trinity-training src/hslm/bpe_merges.zig @ 3e9648c1817a97c5f9b8c727b0f8e2291928eead, as t27 data; tests pin encodings that the Zig tokenizer also produces.
  • Training: R5's golden config (gHashTag/trinity docs/lab/papers/hslm/golden-config.md): ctx 27, batch 66, lr 1e-3 with 2000 warmup and cosine decay, grad clip 1.0, weight decay 0.01, a pinned seed. Loss on the last position as R5 did, and also full-sequence next-token loss on held-out data.
  • Every number published comes from a log or a file whose sha256 is in the spec.

Success Criteria

  • t27c compile specs/ml/train_hslm.t27 --backend ml builds a binary on the lab that trains the 3-block model, writes a checkpoint and reads it back.
  • A run of at least the step count R5 reached reports held-out loss and PPL, with the train-minibatch figure next to it so it can be compared to 2.96.
  • The model and a .t27 model card are published on Hugging Face; own_model cell 25 cites them.

Activity

  1. added
    shape-refusedClosed by the issue-shape gate (specs/policy/issue_shape.t27); a passing edit reopens it
    on Oct 18, 2026
  2. github-actions commented on Oct 18, 2026

    @github-actions
    Contributor

    Closed by the issue-shape gate (specs/policy/issue_shape.t27, #7498): an issue must have the template's sections. Edit the body; an edit that passes reopens it.

    • missing: ## Requirements -- numbered obligations, FR-001: ... MUST ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    shape-refusedClosed by the issue-shape gate (specs/policy/issue_shape.t27); a passing edit reopens it

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions