You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Part of epic #1032 (IGLA-Coder). Plan: t27_coder_model_plan.md P1.
Goal
Build the tokenizer and the two-part data spine: a general code corpus + the unique t27 parallel multi-language corpus.
Sub-tasks
P1.1 Train byte-level BPE tokenizer, vocab 32K (compact, DeepSeek-Coder-style), with FIM sentinels (<fim_prefix>,<fim_suffix>,<fim_middle>,<fim_pad>,<fim_eot>) and a <file_sep> token. Tied embeddings assumed.
P1.2 General corpus: permissive subset of The Stack v2 dedup, top ~20 languages first; MinHash+LSH near-dedup (256 perms, Jaccard 0.7, 5-gram shingles).
P1.3 t27 parallel corpus (the differentiator): pair every .t27 spec with its gen/ artefacts -> spec->code and code<->code pairs (C <-> Verilog <-> 16-lang numeric catalog). Tag each with its executable test/invariant/bench block.
P1.4 FIM transform (50% docs, 50/50 PSM/SPM) + repo-level packing with <file_sep>. W3C-PROV header on each shard manifest.
Done when
Tokenizer + tokenised shards + data_card.md (sizes, langs, licence filter, dedup params) reproducible from a script.
References: The Stack v2; DeepSeek-Coder FIM; StarCoder2 repo-level FIM. See research_code_llm_sota.md.
Anchor: phi^2 + phi^-2 = 3
Decision metric locked: code BPB -> small pass@1 (MultiPL-E subset + t27-native executable eval). Generic-text BPB is diagnostic only.
Baseline frozen: champion BPB=2.2111 is the STANDARD AdamW arm (beta1=0.9, wd=0.04), NOT phi-canonical. Any phi claim must state this first.
CPU-bound reality recorded: trainer is cpu_train + AdamWCpu; no GPU.
Artefact: CODER_METRICS.md.
P1 (tokenizer + data spine) -- STARTED
Audited the real data contract: trainer is byte-level, VOCAB_SIZE=128, u8 targets, .bin token files (pipeline.rs).
Tokenizer decision (honest, CPU-bound): start byte-level, vocab 256 + 7 control/FIM sentinels (-> 263). The 32K-BPE SOTA default is GPU-era and is NOT used here; small 4K-8K BPE only if P4 CPU timing demands it. FIM slots reserved (<FIM_PRE/MID/SUF>), enabled at P2.
t27 parallel corpus extracted (build_t27_parallel_corpus.py -> t27_corpus/):
85 files, ~1.2 MB. Two compiler-verified translation signals:
numeric/formats_catalog emitted into 16 languages (single 16-way parallel module).
33 matched C<->Verilog module pairs across 10 categories (ar, base, compiler, fpga, isa, math, nn, numeric, queen, vsa).
HONEST SCALE: this is a seed + eval set, not pretraining volume (KBs). Value = aligned translation signal + executable eval, not token count.
Data spine tiers defined; proposed .bin contract + entrypoint env (no secrets): TRIOS_TRAIN_DATA=data/code_train.bin, TRIOS_VOCAB=263, TRIOS_HIDDEN=384, 3 parallel seed services {42,43,44} as the champion did.
Artefact: CODER_DATA_AND_TOKENIZER.md.
Next (P2): implement the binarizer tool (mirror trinity_3k_fineweb_train.rs), widen VOCAB_SIZE to 263, wire the real code next-token-prediction loss + code-BPB readout.
Partial completion verified against the comment history:
✅ P1.1 Tokenizer decision — Byte-level vocab 263 (256 + 7 control/FIM sentinels) selected as the honest CPU-bound default. The 32K-BPE SOTA default is GPU-era and NOT used here; small 4K-8K BPE deferred to P4 timing demands.
⬜ P1.2 General corpus — MinHash+LSH near-dedup on The Stack v2 subset pending (P4 gate).
✅ P1.3 t27 parallel corpus — Extracted: 85 files, ~1.2 MB. Two compiler-verified translation signals:
numeric/formats_catalog emitted into 16 languages.
33 matched C<->Verilog module pairs across 10 categories.
HONEST SCALE: seed + eval set (KBs), not pretraining volume.
⬜ P1.4 FIM transform + repo-level packing — FIM slots reserved (<FIM_PRE/MID/SUF>), enabled at P2. W3C-PROV header pending.
Artefacts: CODER_DATA_AND_TOKENIZER.md referenced in P0 comment. build_t27_parallel_corpus.py regenerates corpus from t27 gen/ (1,044,046 train tokens, vocab 263).
Blocked: general corpus scale (The Stack v2 access + dedup pipeline) gates P1.2 and P1.4 completion.
Activity
gHashTag commentedon Jun 1, 2026
P0 done + P1 started (2026-06-01) -- CPU/Railway
P0 (decision & honesty foundation) -- DONE
cpu_train+AdamWCpu; no GPU.CODER_METRICS.md.P1 (tokenizer + data spine) -- STARTED
VOCAB_SIZE=128,u8targets,.bintoken files (pipeline.rs).<FIM_PRE/MID/SUF>), enabled at P2.build_t27_parallel_corpus.py->t27_corpus/):numeric/formats_catalogemitted into 16 languages (single 16-way parallel module)..bincontract + entrypoint env (no secrets):TRIOS_TRAIN_DATA=data/code_train.bin,TRIOS_VOCAB=263,TRIOS_HIDDEN=384, 3 parallel seed services {42,43,44} as the champion did.CODER_DATA_AND_TOKENIZER.md.Next (P2): implement the binarizer tool (mirror
trinity_3k_fineweb_train.rs), widenVOCAB_SIZEto 263, wire the real code next-token-prediction loss + code-BPB readout.gHashTag commentedon Jun 16, 2026
P1 Status Update (2026-06-16)
Partial completion verified against the comment history:
Artefacts: CODER_DATA_AND_TOKENIZER.md referenced in P0 comment. build_t27_parallel_corpus.py regenerates corpus from t27 gen/ (1,044,046 train tokens, vocab 263).
Blocked: general corpus scale (The Stack v2 access + dedup pipeline) gates P1.2 and P1.4 completion.
Anchor: phi^2 + phi^-2 = 3.
fix(ci): Coq admitted check covers all .v files; add allow-list mecha…
gHashTag commentedon Jun 17, 2026
Wave Loop 92 audit: closing long-running sub-issue — tokenizer work tracked in newer milestones.