Skip to content

[GDPVal] Walltime-teardown rollouts are classified 'legitimate', which is non-waivable #3336

Open

Description

@agronskiy

What happened

During a full GDPval baseline sweep, one run reported legitimate: 152 on a
stage-2 batch. It was not 152 broken tasks. It was a single 4h Slurm walltime
reap whose in-flight rollouts Ray could not serialise; every one of them landed
in the legitimate bucket.

They recovered, but only incidentally: the rows were also non-terminal, so the
next auto-requeued leg re-dispatched them (the retry batch scoped correctly to
153 rows rather than redoing the stage).

Why it matters

legitimate is deliberately the one class the partial-completion policy will
never waive — waivable_failure_classes is [timeout_exceeded, transient], and
the comments in refs.yaml are explicit that "a real task failure must still
stop the stage". That is the right design.

The problem is that a purely infrastructural event can present as that class. If
a run had exited cleanly with those rows outstanding, the coverage gate would
have rejected a non-final stage over something that was never a task failure —
and per the orchestrator's own comment, a coverage rejection "requires an
explicit policy/data change", so nel resume would not have helped.

Note the sibling path is already handled correctly: the same reap, when it
surfaces as TaskPerAttemptTimeoutError, classifies to timeout_exceeded,
which is waivable.

Where

responses_api_agents/stirrup_agent/app.py, _classify_rollout_failure.

The function matches TaskPerAttemptTimeoutError → timeout_exceeded,
TaskSampleSkipError → skipped, the Ray kill family → kill_shaped, and
RayTaskError → kill_shaped when there is no user-code frame. Everything else
falls through to return "legitimate" at the end.

A serialisation failure raised while the worker is being torn down at walltime
reaches that fallback. Semantically it belongs with kill_shaped or
timeout_exceeded: the rollout did not fail on its own merits, the allocation
ended underneath it.

Suggested direction

Make the fallback conservative for teardown-shaped exceptions rather than
defaulting them to the one non-waivable class. Either recognise the
serialisation error explicitly, or treat "exception raised after the walltime
deadline has passed" as timeout_exceeded regardless of type. The current
default optimises for not silently waiving real failures, which is correct, but
it means every unrecognised infrastructure fault inherits the strictest class.

Evidence

Observed 2026-09-12 on HSG, qwen-3.8-flash-next, invocation
c71abd45d04b4210, across legs 7100053 → 7104442 → 7104454. Found while running
a seven-model GDPval sweep for #3333.

Activity

  1. agronskiy commented on Sep 12, 2026

    @agronskiy
    ContributorAuthor

    Direct A/B evidence from the same sweep, which makes this sharper than the
    original report: the same physical event produced different waivability on
    two runs, by chance.

    Both runs hit a 4h Slurm walltime reap mid-stage-2:

    run classified as waivable?
    glm-5.3-flash timeout_exceeded: 141 yes
    qwen-3.8-flash-next legitimate: 145 no

    Same cluster, same leg boundary, same stage, same config. The only difference is
    whether Ray managed to serialise the in-flight rollouts before the worker went
    away. When it does, the reap surfaces as TaskPerAttemptTimeoutError and the
    classifier returns timeout_exceeded. When it loses that race, the exception
    falls through _classify_rollout_failure to the terminal return "legitimate".

    So waivability of a walltime reap is currently decided by a nondeterministic
    teardown race, not by anything about the rollout. Both recover identically when
    a subsequent leg re-dispatches them — only the label differs, and the label is
    what the partial-completion gate reads.

    That is the failure mode to fix: a run that exits cleanly while holding the
    legitimate variant blocks its stage on a coverage rejection, which nel resume
    cannot clear, while an otherwise identical run holding the timeout_exceeded
    variant is waived and proceeds.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions