Skip to content

[IGLA-Coder] P4 Pilot pretraining at 50-200M #1037

Open

Description

@gHashTag

Boundary

Pilot pretraining of the best P3 arm at 50–200M params for IGLA-Coder (epic #1032, part of the
#10024 own-model map). In scope: CPU continued training from an existing published checkpoint
(tern-tc-9m) and any budget-gated scale-up decision. Out of scope: GPU purchases, cloud
training, paid API calls, direct provider access — the DSH Ladder free rungs and local CPU are
the only compute paths (owner decision 2026-11-11). Quality baseline to beat: tern-tc-500m
HumanEval pass@1 = 0.95% (2/164). Training follows the continual-learning law
specs/igla/cl_loop.t27 (branch cl-spike): replay mix, forgetting threshold, adapter-step clip.

User Scenarios

  • An operator approves a bounded CPU training run (corpus slice, steps, checkpoint interval);
    the agent resumes from the published checkpoint, trains, evaluates pass@1 on held-out specs
    through the [IGLA-Coder] P5 Multi-language evaluation harness #1038 harness, and reports with receipts.
  • A training run forgets old capability; the law rolls the adapter back (threshold exceeded)
    and reports the regression instead of silently shipping a worse model.
  • The pilot quality is measured honestly (MultiPL-E pass@1 on >=6 languages) and compared
    against the 0.95% baseline before any larger spend is requested.

Requirements

  • FR-001: The agent MUST restore and verify the HSLM Zig trainer on current master
    (specs/igla/, tri/ pipeline) before any training run; no hand-written trainer outside the repo.
  • FR-002: The agent MUST use the continual-learning law specs/igla/cl_loop.t27
    (replay 5:3, forgetting threshold 1, LoRA step cap) for every update.
  • FR-003: The M1 corpus MUST be 1k real .t27 specs drawn from THIS repository, filtered as:
    every specs/**/*.t27 containing at least one test block (the measured pool is 2500 of 3183
    specs as of commit b339f8a; find specs -name '*.t27' + grep -l '^test '), deduplicated by
    module name, and split by commit date -- everything before the corpus cutoff trains, the newest
    10% is held out for pass@1. The full 50-200B-token corpus is M2-only.
  • FR-004: The agent MUST record tokens/sec, seconds/step, steps/hour, memory and pass@1 at every
    checkpoint; scoring MUST go through the [IGLA-Coder] P5 Multi-language evaluation harness #1038 harness.
  • FR-005: Checkpoints MUST follow the Catalog42 protocol (seeds, frozen config, sha256 of outputs).

Success Criteria

  • M1: pass@1 >= 5% on the held-out .t27 spec split within 8 CPU-hours. THE BENCHMARK IS FIXED:
    HumanEval-style pass@1 scored by the [IGLA-Coder] P5 Multi-language evaluation harness #1038 harness executing the generated spec with t27b test
    (the same benchmark protocol the 0.95% tern-tc-500m baseline was measured with -- comparison
    is only valid on the same benchmark); the [IGLA-Coder] P5 Multi-language evaluation harness #1038 t27-native eval runs as a supplementary report,
    not as the M1 gate. Receipt files and the checkpoint sha256 are committed to the branch. A flat
    pass@1 over 3 consecutive checkpoints, or > 1 replay regression twice, exits non-zero for the
    milestone and stops the run.
  • M2: MultiPL-E pass@1 on >= 6 languages at 50-200M params, strictly above the 0.95% baseline
    of tern-tc-500m, with the reproducibility capsule (seeds, config, sha256) attached.

Plan (from the original body, preserved)

  • P4.1 Pilot pretrain on the P1 corpus, best arm from P3.
  • P4.2 Two-stage schedule: main pretrain (WSD/cosine) + annealing last 10–15% on a high-quality
    mix (verified-execution exercises + the t27 parallel corpus).
  • P4.3 Reproducibility capsule: seeds + frozen config + sha256 of outputs (Catalog42 protocol).

Part of epic #1032 and the #10024 own-model map; the continual-learning correction and stage
separation are in the comments (2026-11-11). Blocked on: nothing structural — the compute target
is CPU continued training; the budget line (M2 scale-up) is a separate owner decision.
Anchor: phi^2 + phi^-2 = 3

Activity

  1. added
    needs-boundaryNo '## Boundary' section, so the Queen can reserve nothing for it and no bee can take it
    on Sep 23, 2026
  2. gHashTag commented on Oct 7, 2026

    @gHashTag
    CollaboratorAuthor

    Reopened 2026-10-07: the bulk close of issues without a ## Boundary section was reverted by the owner. Nothing to do here; this comment replaces the earlier close notice.

  3. added
    needs-shapeIssue lacks Boundary/Scenarios/Requirements/Success criteria
    on Oct 7, 2026
  4. removed
    needs-shapeIssue lacks Boundary/Scenarios/Requirements/Success criteria
    on Oct 7, 2026
  5. tri-claw commented on Oct 9, 2026

    @tri-claw

    The Queen cannot take this issue yet. A bee starts only on an issue whose body has every section below, and this one lacks:

    • missing: ## Boundary -- list every file this task may touch, one per line (- specs/x.t27); without it the Queen reserves nothing and no bee can take the issue
    • missing: ## User Scenarios -- at least one Given/When/Then scenario
    • missing: ## Requirements -- numbered obligations, FR-001: ... MUST ...
    • missing: ## Success Criteria -- an outcome a machine can settle: a number with a unit, a backticked file, exit 0, or a backticked command

    She tried to write them 2 times and could not (last: not a task: No candidate paths exist in the repository to define a boundary for this training task; the issue references external plans (t27_coder_model_plan.md) and protocols (Catalog42) not present as files.). Add them and the next round takes it. The rule is specs/policy/issue_shape.t27.

  6. gHashTag commented on Nov 11, 2026

    @gHashTag
    CollaboratorAuthor

    DRAFT — P4 budget request package (ORDER IGLA_CODER_CL_SPIKE_V1, Wave 4; owner approval required before any training spend)

    Measured on this bench (commands in receipts):

    • t27b oracle loop on a real spec: build median 16.9 ms, test median 14.1 ms → 31 ms/iteration, ~116k oracle runs/hour single core (t27b build <spec> --macho + t27b test <spec> --quiet, 20 runs each, median).
    • CL loop end-to-end at toy scale (law: specs/igla/cl_loop.t27, 6 tests green; branch cl-spike @ 24f933e, local-only): 3 old tasks learned (6 oracle iterations), new task learned (2 iterations), replay held (0 regressions, threshold 1), accepted by law; 7 ms CPU for the whole dry run, params ~0.1M << 10M cap. Receipts: /tmp/cl-spike/receipts.json.

    Extrapolation to the real pilot (#1037, 50���200M):

    • The oracle loop stays ~free: the compiler cost is per-evaluation, independent of model size.
    • Training cost is dominated by forward/backward on the model, not the oracle. For 50M params on CPU-only: expect ~0.1–0.5 s/step at batch 8 (order of magnitude, to be measured in the pilot's first hour) → a 10k-step pilot ≈ 0.5–1.5 wall-clock hours/epoch; a meaningful pilot (10–50 epochs over a spec corpus) ≈ 5–75 CPU-hours. RAM: 50M params fp32 ≈ 200 MB weights + optimizer ≈ 600 MB — fits this Mac.
    • Honest unknown: sample efficiency of spec-driven rejection sampling on a 50M model is NOT yet measured; the pilot's first milestone must measure pass-rate vs steps before any larger run.

    Proposed milestones (each stoppable):

    1. M1 (no budget change, CPU-only, ≤ 8 CPU-hours): train 50M on 1k spec corpus with LoRA+replay; measure pass@1 on the held-out set with the [IGLA-Coder] P5 Multi-language evaluation harness #1038 harness. Stop if pass@1 at M1 < 5%.
    2. M2 (the actual P4 spend decision, separate approval): scale corpus and steps only if M1 shows a learning curve.

    Stop conditions: forgetting above threshold twice in a row; pass@1 flat over 3 checkpoints; CPU-hour cap exceeded.

    Ask: approve M1 (CPU-only, ≤ 8 CPU-hours) as a budget line item inside P4; M2 comes back as a separate approval with M1 measurements attached.

  7. gHashTag commented on Nov 11, 2026

    @gHashTag
    CollaboratorAuthor

    CORRECTION to the draft above (owner-identified errors, 2026-11-11)

    1. CPU training is not a hypothesis — it already happened. tern-tc-9m and tern-tc-500m
      are trained AND published (see epic: our own model -- IGLA CODER, HSLM in t27 and the harness, one 27-part map #10024 map); the HSLM Zig trainer, forward/backward,
      checkpoint I/O and optimizers (SGD/Adam/LAMB) are verified. The GPU-first framing of the
      draft above is wrong. The real measured defect of the existing CPU path is QUALITY:
      tern-tc-500m HumanEval pass@1 = 0.95% (2/164).

    2. Stage separation the draft missed:

      Stage Measures Proves
      CL spike, Wave 3 (DONE: branch cl-spike @ 24f933e, law specs/igla/cl_loop.t27, 6 tests) loop scaffold: replay mix 5:3, forgetting threshold, adapter update, rollback the CL mechanics work
      M1 (next, conditionally approved) code generation: pass@1 on held-out real .t27 specs via the [IGLA-Coder] P5 Multi-language evaluation harness #1038 harness the model learns to generate code, not memorize constants
      M2 scale-up 50–200M, full corpus deployable quality

      The Wave-3 dry run (constant-guessing) is a PREREQUISITE, not evidence for pass@1.

    3. Revised M1 plan (CPU, existing pipeline):

      • Start from the existing tern-tc-9m checkpoint (published), not from scratch.
      • Restore/verify the Zig training path on current master first (unbreak check).
      • Corpus: 1k real .t27 spec slices (not toy constants); held-out split for pass@1.
      • Continue training with LoRA/adapters + replay per the law in specs/igla/cl_loop.t27.
      • Measure: tokens/sec, seconds/step, steps/hour, memory, pass@1 via [IGLA-Coder] P5 Multi-language evaluation harness #1038 harness.
      • Stop conditions unchanged (forgetting x2, flat pass@1 over 3 checkpoints, cap exceeded).
    4. P4 reframed: not "buy GPU" — a plan and acceptance of CPU continued training:
      corpus slice, steps, checkpoint interval, eval gate, stop conditions. GPU only decided
      AFTER CPU quality is measured (the 0.95% number is the baseline to beat).

    No training spend has been made; the next spend requires the owner's approval of this revised M1.

  8. added
    shape-refusedClosed by the issue-shape gate (specs/policy/issue_shape.t27); a passing edit reopens it
    on Nov 11, 2026
  9. removed
    needs-boundaryNo '## Boundary' section, so the Queen can reserve nothing for it and no bee can take it
    on Nov 11, 2026
  10. github-actions commented on Nov 11, 2026

    @github-actions
    Contributor

    Closed by the issue-shape gate (specs/policy/issue_shape.t27, #7498): an issue must have the template's sections. Edit the body; an edit that passes reopens it.

    • missing: ## Requirements -- numbered obligations, FR-001: ... MUST ...
    • missing: ## Success Criteria -- an outcome a machine can settle: a number with a unit, a backticked file, exit 0, or a backticked command
  11. removed
    shape-refusedClosed by the issue-shape gate (specs/policy/issue_shape.t27); a passing edit reopens it
    on Nov 11, 2026
  12. github-actions commented on Nov 11, 2026

    @github-actions
    Contributor

    The shape check passes now (specs/policy/issue_shape.t27); reopened.

  13. gHashTag commented on Nov 11, 2026

    @gHashTag
    CollaboratorAuthor

    Shape fixed (4-heading, FR-numbered obligations, machine-settleable criteria); branch with the CL law: cl-spike @ 24f933e (specs/igla/cl_loop.t27). M1 awaits owner approval per the correction above.

  14. gHashTag commented on Nov 11, 2026

    @gHashTag
    CollaboratorAuthor

    ADDENDUM (owner, 2026-11-11): continuous-learning serving loop + baseline correction

    • Baseline correction: HumanEval 0.95% (tern-tc-500m) is NOT the baseline for held-out .t27
      tasks. Before the first adapter update, the STARTING checkpoint must be measured on the SAME
      .t27 eval the candidate will be scored on. The M1 gate compares candidate vs that measured
      starting number, on the same benchmark.
    • Serving/learning separation: inference runs a stable checkpoint in a separate CPU process;
      training prepares the NEXT version offline (never mutate weights inside a request).
    • Data: opt-in only; the stored example is the tuple prompt -> allowed context -> proposed patch
      -> test results -> user verdict. An unverified model answer is not a training label. Secrets and
      cross-tenant context are stripped at collection.
    • Law work: data-selection, replay, promotion and rollback rules go into .t27 (no new
      handwritten .rs protocol logic; the loader stays IO-only per the vps: ladder_protocol_map.t27 -- law of the protocol adapter in front of dsh-ladder #13605 pattern).
    • Publication: versioned adapter, canary with limited traffic, fast rollback; promotion requires
      measured improvement plus regression/privacy gates.
    • Measured serving numbers (time-to-first-token, tokens/sec, p95, concurrent clients) must be
      measured before any realtime claim; inference gets priority over training under load.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    phi-loopPHI LOOP development cycle

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions