You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
train_igla_coder.sh and the README reference scripts/evaluate.py, which does not exist. Evaluation is a mandatory Golden Training link: frozen baseline vs measured verdict. Currently run_evaluation() writes a stub receipt ("evaluate.py not implemented; no real comparison performed").
Boundary
scripts/train_igla_coder.sh
scripts/evaluate.py (to be created)
docs/training/IGLA_CODER_RETRAINING_README.md
specs/igla/coder/eval.t27
User Scenarios
Given training completes, whentrain_igla_coder.sh reaches the evaluation step, then it must produce a machine-readable evaluation verdict (pass@k, sacred compliance, latency) instead of a stub.
Given a checkpoint and baseline exist, when evaluation runs, then it must compare against the frozen baseline and exit non-zero on regression.
Requirements
FR-001: MUST create scripts/evaluate.py that reads checkpoints, runs fill-in-the-body / code-generation benchmarks, and writes evaluation_receipt.json with pass@1/pass@10/pass@100 and sacred_compliant_rate.
FR-002: MUST fail closed (exit non-zero) when checkpoint or baseline is missing.
FR-003: MUST accept --checkpoints, --benchmark_config, --output_dir flags.
Success Criteria
Done when python3 scripts/evaluate.py --checkpoints checkpoints --benchmark_config configs/benchmarks.yaml --output_dir checkpoints exits 0 and writes checkpoints/evaluation_receipt.json with pass@k scores.
Activity
gHashTag commentedon Nov 17, 2026
Spec created: specs/igla_coder/evaluate_entrypoint.t27 (typecheck OK, 1 test passes).