Skip to content

IGLA CODER Missing evaluate.py Evaluation Script #12915

Closed

Description

@gHashTag

IGLA CODER Missing evaluate.py Evaluation Script

train_igla_coder.sh and the README reference scripts/evaluate.py, which does not exist. Evaluation is a mandatory Golden Training link: frozen baseline vs measured verdict. Currently run_evaluation() writes a stub receipt ("evaluate.py not implemented; no real comparison performed").

Boundary

  • scripts/train_igla_coder.sh
  • scripts/evaluate.py (to be created)
  • docs/training/IGLA_CODER_RETRAINING_README.md
  • specs/igla/coder/eval.t27

User Scenarios

Given training completes, when train_igla_coder.sh reaches the evaluation step, then it must produce a machine-readable evaluation verdict (pass@k, sacred compliance, latency) instead of a stub.

Given a checkpoint and baseline exist, when evaluation runs, then it must compare against the frozen baseline and exit non-zero on regression.

Requirements

  • FR-001: MUST create scripts/evaluate.py that reads checkpoints, runs fill-in-the-body / code-generation benchmarks, and writes evaluation_receipt.json with pass@1/pass@10/pass@100 and sacred_compliant_rate.
  • FR-002: MUST fail closed (exit non-zero) when checkpoint or baseline is missing.
  • FR-003: MUST accept --checkpoints, --benchmark_config, --output_dir flags.

Success Criteria

Done when python3 scripts/evaluate.py --checkpoints checkpoints --benchmark_config configs/benchmarks.yaml --output_dir checkpoints exits 0 and writes checkpoints/evaluation_receipt.json with pass@k scores.

Activity

  1. gHashTag commented on Nov 17, 2026

    @gHashTag
    CollaboratorAuthor

    Spec created: specs/igla_coder/evaluate_entrypoint.t27 (typecheck OK, 1 test passes).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions