Skip to content

[IGLA-Coder] P1 Tokenizer and data spine #1034

Closed

Description

@gHashTag

Part of epic #1032 (IGLA-Coder). Plan: t27_coder_model_plan.md P1.

Goal

Build the tokenizer and the two-part data spine: a general code corpus + the unique t27 parallel multi-language corpus.

Sub-tasks

  • P1.1 Train byte-level BPE tokenizer, vocab 32K (compact, DeepSeek-Coder-style), with FIM sentinels (<fim_prefix>,<fim_suffix>,<fim_middle>,<fim_pad>,<fim_eot>) and a <file_sep> token. Tied embeddings assumed.
  • P1.2 General corpus: permissive subset of The Stack v2 dedup, top ~20 languages first; MinHash+LSH near-dedup (256 perms, Jaccard 0.7, 5-gram shingles).
  • P1.3 t27 parallel corpus (the differentiator): pair every .t27 spec with its gen/ artefacts -> spec->code and code<->code pairs (C <-> Verilog <-> 16-lang numeric catalog). Tag each with its executable test/invariant/bench block.
  • P1.4 FIM transform (50% docs, 50/50 PSM/SPM) + repo-level packing with <file_sep>. W3C-PROV header on each shard manifest.

Done when

Tokenizer + tokenised shards + data_card.md (sizes, langs, licence filter, dedup params) reproducible from a script.

References: The Stack v2; DeepSeek-Coder FIM; StarCoder2 repo-level FIM. See research_code_llm_sota.md.
Anchor: phi^2 + phi^-2 = 3

Activity

  1. gHashTag commented on Jun 1, 2026

    @gHashTag
    CollaboratorAuthor

    P0 done + P1 started (2026-06-01) -- CPU/Railway

    P0 (decision & honesty foundation) -- DONE

    • Decision metric locked: code BPB -> small pass@1 (MultiPL-E subset + t27-native executable eval). Generic-text BPB is diagnostic only.
    • Baseline frozen: champion BPB=2.2111 is the STANDARD AdamW arm (beta1=0.9, wd=0.04), NOT phi-canonical. Any phi claim must state this first.
    • CPU-bound reality recorded: trainer is cpu_train + AdamWCpu; no GPU.
    • Artefact: CODER_METRICS.md.

    P1 (tokenizer + data spine) -- STARTED

    • Audited the real data contract: trainer is byte-level, VOCAB_SIZE=128, u8 targets, .bin token files (pipeline.rs).
    • Tokenizer decision (honest, CPU-bound): start byte-level, vocab 256 + 7 control/FIM sentinels (-> 263). The 32K-BPE SOTA default is GPU-era and is NOT used here; small 4K-8K BPE only if P4 CPU timing demands it. FIM slots reserved (<FIM_PRE/MID/SUF>), enabled at P2.
    • t27 parallel corpus extracted (build_t27_parallel_corpus.py -> t27_corpus/):
      • 85 files, ~1.2 MB. Two compiler-verified translation signals:
        • numeric/formats_catalog emitted into 16 languages (single 16-way parallel module).
        • 33 matched C<->Verilog module pairs across 10 categories (ar, base, compiler, fpga, isa, math, nn, numeric, queen, vsa).
      • HONEST SCALE: this is a seed + eval set, not pretraining volume (KBs). Value = aligned translation signal + executable eval, not token count.
    • Data spine tiers defined; proposed .bin contract + entrypoint env (no secrets):
      TRIOS_TRAIN_DATA=data/code_train.bin, TRIOS_VOCAB=263, TRIOS_HIDDEN=384, 3 parallel seed services {42,43,44} as the champion did.
    • Artefact: CODER_DATA_AND_TOKENIZER.md.

    Next (P2): implement the binarizer tool (mirror trinity_3k_fineweb_train.rs), widen VOCAB_SIZE to 263, wire the real code next-token-prediction loss + code-BPB readout.

  2. gHashTag commented on Jun 16, 2026

    @gHashTag
    CollaboratorAuthor

    P1 Status Update (2026-06-16)

    Partial completion verified against the comment history:

    • ✅ P1.1 Tokenizer decision — Byte-level vocab 263 (256 + 7 control/FIM sentinels) selected as the honest CPU-bound default. The 32K-BPE SOTA default is GPU-era and NOT used here; small 4K-8K BPE deferred to P4 timing demands.
    • ⬜ P1.2 General corpus — MinHash+LSH near-dedup on The Stack v2 subset pending (P4 gate).
    • ✅ P1.3 t27 parallel corpus — Extracted: 85 files, ~1.2 MB. Two compiler-verified translation signals:
      • numeric/formats_catalog emitted into 16 languages.
      • 33 matched C<->Verilog module pairs across 10 categories.
      • HONEST SCALE: seed + eval set (KBs), not pretraining volume.
    • ⬜ P1.4 FIM transform + repo-level packing — FIM slots reserved (<FIM_PRE/MID/SUF>), enabled at P2. W3C-PROV header pending.

    Artefacts: CODER_DATA_AND_TOKENIZER.md referenced in P0 comment. build_t27_parallel_corpus.py regenerates corpus from t27 gen/ (1,044,046 train tokens, vocab 263).

    Blocked: general corpus scale (The Stack v2 access + dedup pipeline) gates P1.2 and P1.4 completion.

    Anchor: phi^2 + phi^-2 = 3.

  3. gHashTag commented on Jun 17, 2026

    @gHashTag
    CollaboratorAuthor

    Wave Loop 92 audit: closing long-running sub-issue — tokenizer work tracked in newer milestones.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    phi-loopPHI LOOP development cycle

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions