You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[GDPVal] Walltime-teardown rollouts are classified 'legitimate', which is non-waivable #3336
During a full GDPval baseline sweep, one run reported legitimate: 152 on a
stage-2 batch. It was not 152 broken tasks. It was a single 4h Slurm walltime
reap whose in-flight rollouts Ray could not serialise; every one of them landed
in the legitimate bucket.
They recovered, but only incidentally: the rows were also non-terminal, so the
next auto-requeued leg re-dispatched them (the retry batch scoped correctly to
153 rows rather than redoing the stage).
Why it matters
legitimate is deliberately the one class the partial-completion policy will
never waive — waivable_failure_classes is [timeout_exceeded, transient], and
the comments in refs.yaml are explicit that "a real task failure must still
stop the stage". That is the right design.
The problem is that a purely infrastructural event can present as that class. If
a run had exited cleanly with those rows outstanding, the coverage gate would
have rejected a non-final stage over something that was never a task failure —
and per the orchestrator's own comment, a coverage rejection "requires an
explicit policy/data change", so nel resume would not have helped.
Note the sibling path is already handled correctly: the same reap, when it
surfaces as TaskPerAttemptTimeoutError, classifies to timeout_exceeded,
which is waivable.
The function matches TaskPerAttemptTimeoutError → timeout_exceeded, TaskSampleSkipError → skipped, the Ray kill family → kill_shaped, and RayTaskError → kill_shaped when there is no user-code frame. Everything else
falls through to return "legitimate" at the end.
A serialisation failure raised while the worker is being torn down at walltime
reaches that fallback. Semantically it belongs with kill_shaped or timeout_exceeded: the rollout did not fail on its own merits, the allocation
ended underneath it.
Suggested direction
Make the fallback conservative for teardown-shaped exceptions rather than
defaulting them to the one non-waivable class. Either recognise the
serialisation error explicitly, or treat "exception raised after the walltime
deadline has passed" as timeout_exceeded regardless of type. The current
default optimises for not silently waiving real failures, which is correct, but
it means every unrecognised infrastructure fault inherits the strictest class.
Evidence
Observed 2026-09-12 on HSG, qwen-3.8-flash-next, invocation c71abd45d04b4210, across legs 7100053 → 7104442 → 7104454. Found while running
a seven-model GDPval sweep for #3333.
Direct A/B evidence from the same sweep, which makes this sharper than the
original report: the same physical event produced different waivability on
two runs, by chance.
Both runs hit a 4h Slurm walltime reap mid-stage-2:
run
classified as
waivable?
glm-5.3-flash
timeout_exceeded: 141
yes
qwen-3.8-flash-next
legitimate: 145
no
Same cluster, same leg boundary, same stage, same config. The only difference is
whether Ray managed to serialise the in-flight rollouts before the worker went
away. When it does, the reap surfaces as TaskPerAttemptTimeoutError and the
classifier returns timeout_exceeded. When it loses that race, the exception
falls through _classify_rollout_failure to the terminal return "legitimate".
So waivability of a walltime reap is currently decided by a nondeterministic
teardown race, not by anything about the rollout. Both recover identically when
a subsequent leg re-dispatches them — only the label differs, and the label is
what the partial-completion gate reads.
That is the failure mode to fix: a run that exits cleanly while holding the legitimate variant blocks its stage on a coverage rejection, which nel resume
cannot clear, while an otherwise identical run holding the timeout_exceeded
variant is waived and proceeds.
Activity
agronskiy commentedon Sep 12, 2026
Direct A/B evidence from the same sweep, which makes this sharper than the
original report: the same physical event produced different waivability on
two runs, by chance.
Both runs hit a 4h Slurm walltime reap mid-stage-2:
glm-5.3-flashtimeout_exceeded: 141qwen-3.8-flash-nextlegitimate: 145Same cluster, same leg boundary, same stage, same config. The only difference is
whether Ray managed to serialise the in-flight rollouts before the worker went
away. When it does, the reap surfaces as
TaskPerAttemptTimeoutErrorand theclassifier returns
timeout_exceeded. When it loses that race, the exceptionfalls through
_classify_rollout_failureto the terminalreturn "legitimate".So waivability of a walltime reap is currently decided by a nondeterministic
teardown race, not by anything about the rollout. Both recover identically when
a subsequent leg re-dispatches them — only the label differs, and the label is
what the partial-completion gate reads.
That is the failure mode to fix: a run that exits cleanly while holding the
legitimatevariant blocks its stage on a coverage rejection, whichnel resumecannot clear, while an otherwise identical run holding the
timeout_exceededvariant is waived and proceeds.