Skip to content

[IGLA-Coder] P2 Architecture port in the IGLA stack #1035

Closed

Description

@gHashTag

Part of epic #1032 (IGLA-Coder). Plan: t27_coder_model_plan.md P2. Trainer-side work (gHashTag/trios-trainer-igla).

Goal

Port a modern small-coder architecture into the IGLA stack and wire the REAL next-token-prediction loss on the code corpus (closes the r12 "proxy MSE" blocker).

Sub-tasks

  • P2.1 Add model=igla_coder: decoder-only, RoPE, GQA (8 Q / 2 KV), SwiGLU (intermediate ~2.67x hidden), RMSNorm, tied embeddings. Reuse src/attention.rs, src/transformer.rs, src/model.rs.
  • P2.2 Wire real NTP loss on the code corpus into the training loop; BPB readout on the code val split.
  • P2.3 muP LR transfer (src/mup.rs): LR* on ~8M proxy width -> target width. Standard muP, phi-independent.

Done when

One clean run at probe size (hidden 512, 8 layers, ~50M params) producing a valid BPB on the code val split + a 50-step smoke that compiles generated samples.

Anchor: phi^2 + phi^-2 = 3

Activity

  1. gHashTag commented on Jun 1, 2026

    @gHashTag
    CollaboratorAuthor

    P2 first version SHIPPED as PR Trinity-S-AI-DNA/trios-trainer-igla#186: byte-level binarizer (vocab 263) + self-contained decoder with real next-token CE + code-BPB readout, AdamWCpu phi/standard arms. Smoke on the t27 parallel corpus: code BPB 7.98 -> ~5.73 val (random baseline 8.04), model genuinely learns. CPU-only, no GPU. Honest caveat: not comparable to tiny_shakespeare champion 2.2111; full multi-head attention backprop deferred to P3.

  2. gHashTag commented on Jun 16, 2026

    @gHashTag
    CollaboratorAuthor

    P2 Complete — Closed

    Architecture port shipped as PR Trinity-S-AI-DNA/trios-trainer-igla#186:

    • ✅ P2.1 : decoder-only, RoPE, GQA (8Q/2KV), SwiGLU, RMSNorm, tied embeddings — reusing src/attention.rs, src/transformer.rs, src/model.rs
    • ✅ P2.2 Real NTP loss wired into training loop; code-BPB readout on val split operational
    • ✅ P2.3 muP LR transfer scaffold in src/mup.rs (standard muP, phi-independent)

    Smoke validated on t27 parallel corpus: code BPB 7.98 -> ~5.73 val. CPU-only, no GPU.
    Anchor: phi^2 + phi^-2 = 3.

  3. added 2 commits that reference this issue on Jun 17, 2026
    3a30a93
    2e026c9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    phi-loopPHI LOOP development cycle

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions