Skip to content

Budget breaker: a 404 from a delete the operator already made turns a successful resume or roll-back into a red run with a wrong message, and the resume notice is never posted #2329

Open

Description

@don-petry

Problem

Found by an exhaustive interleaving check of the hand-pause procedure (#2314) against the breaker's own read and write sequence. Neither case leaves the fleet in a wrong pause state; both leave a wrong record of what happened.

  1. Resume. A resume writes AGENTS_PAUSED=false, then deletes AGENTS_PAUSE_SOURCE, then AGENTS_PAUSE_UNTIL (scripts/lib/budget-breaker-writer.sh, the resume plan). If an operator has deleted the source in between (which the hand-pause procedure does), the breaker's delete answers 404. The run then stops as failed, goes red with an error that names a missing permission (scripts/budget_breaker.sh, the apply failure path), and exits before posting the "resumed" notice. The fleet was in fact resumed, and nothing on the tracking issue says so.
  2. Trip roll-back. When the pause guard rolls a trip back, it deletes the source it wrote only if it still holds that value. An operator delete landing between the guard's read and the roll-back's delete makes that delete a 404; the roll-back is reported as failed, AGENTS_PAUSE_UNTIL is left behind, and the error says the source "is STILL SET" when it is gone.

Scope

A DELETE that answers 404 means the variable is already absent, which is the state the step wanted. Treat it as done (log it as "already absent"), carry on with the remaining steps, and post the notice the run owes. Keep every other failure as it is today.

Acceptance criteria

  1. A test where the source disappears between the resume's PATCH and its delete: the run is green, the until is deleted, and the "resumed" notice is posted.
  2. A test where the source disappears between the guard's read and the roll-back's delete: the run is green, the until is deleted, and no error claims the source is still set.
  3. A 403 or 500 on the same calls still fails the run exactly as today.

The model checker that found these is saved with the session's notes, not in the repository; the traces are in the comment on #2314.

Activity

  1. github-actions commented on Oct 14, 2026

    @github-actions
    Contributor

    Thanks for the detailed report, and for tracing these interleavings so carefully. The scope and acceptance criteria are clear. Treating a 404 on delete as 'already absent' makes sense, and 403/500 still fail as today. One question before work starts: should the 'already absent' log line apply to every delete in the writer (source and until, in both the resume and roll-back paths), or only the two cases you describe?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugBug reports

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions