Skip to content

[GDPVal] Final stage emits certified metrics after exhausting retries, on any coverage #3341

Open

Description

@agronskiy

What happened

A GDPval baseline run completed with status: complete and emitted both
certified keys — comparison/stage_1/eval_elo 1585.06,
comparison/stage_1/normalized_elo 0.5425 — fitted on 86 of 220 tasks
(39%)
. Nothing in the run surfaced that.

Had it not been inspected by hand, that number would have been ingested as the
model's first-ever GDPval result. It is a model with no prior cell to
sanity-check against, and the fit also sits above the reference ladder's top
anchor, so there was no downstream signal that would have caught it either.

Mechanism

  1. 134 tasks each accumulated 3 failed attempts, exhausting
    NEMO_GYM_MAX_ROLLOUT_ATTEMPTS=3.
  2. 427 of those 428 failures were timeout_exceeded — the 4h Slurm
    partition reap, hit three times, on long trajectories that cannot finish
    inside one leg.
  3. With attempts exhausted the orchestrator stopped re-dispatching, declared the
    stage complete, and fitted on whatever carried evidence: 86 rows.
  4. mean/expected_stage_row_count reads 220.0 against 86 actual. The
    discrepancy is recorded and not acted on.

Two separable defects

(a) timeout_exceeded consumes rollout attempts. It reflects how much
walltime the leg had left, not anything about the task. Burning the 3-attempt
budget on reaps is what turned a slow run into a truncated one.
_is_terminal_failure() already treats the class as non-terminal; the attempt
cap should agree. A reaped rollout should be re-dispatched on the next leg
without charge.

(b) The final stage has no coverage floor. Non-final stages carry
partial_completion with min_success_fraction: 0.9, and that gate would have
rejected this outright. The final stage's plan is only
{num_tasks: 220, num_models: 4}, so a 39% sample and a 99% sample emit
identically-shaped, catalogable metrics. The stage-scoped keys are exactly the
ones leaves certify on, because the flat comparison/normalized_elo requires
220/220 — so the strict gate exists on the key nobody uses and no gate exists on
the key everybody uses.

Suggested direction

For (a): exclude timeout_exceeded from the attempt counter, or count attempts
only for classes _is_terminal_failure() considers terminal.

For (b): either allow partial_completion on the final stage, or refuse to emit
comparison/stage_N/* when the fitted row count falls materially short of
expected_stage_row_count — the values are already both present at emit time.
Failing loudly is clearly better than publishing a 39% fit that looks identical
to a complete one.

Evidence

HSG, 2026-09-12/13, invocation 9016a2e60ff492cb, four 4h legs. Attempt
histogram 1:142, 2:134. Failure classes 427 timeout_exceeded, 1 other.
A second run in the same sweep (c71abd45d04b4210) is on the same trajectory
with 134 timeout_exceeded and 90/220 cached, so this is reproducible rather
than a one-off.

Related: #3336 (a walltime reap landing in a non-waivable class by chance).
Both come from the same place — reap-shaped failures being treated as though
they said something about the rollout.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions