Skip to content

Retries from #572 ship disabled: same_agent_attempts defaults to 1, and docs/config.md says 3 #697

Closed

Description

@aviggiano

Summary

#572 added bounded error-agnostic agent retries and was closed completed. The shipped
ultrafuzz.toml sets the budget to 1, which means the feature is off by default and the exact
failure #572 was filed to prevent still ends runs:

# ultrafuzz.toml (shipped)
[retry]
same_agent_attempts = 1
agents = []

same_agent_attempts includes the first attempt, so 1 is "no retry". #572's own motivating incident
was a node dying at 94% completion with (attempt 1/1) on a stochastic provider error that succeeded
on a manual re-run
. A default of 1 reproduces that class exactly.

The docs and the shipped default disagree

docs/config.md presents the retry section as:

[retry]
same_agent_attempts = 3
agents = ["sol-xhigh", "gpt55-xhigh"]

while docs/reference/configuration.md and the shipped ultrafuzz.toml both use 1. A reader of the
configuration guide would reasonably believe 3 is normal. Whatever the intended default, the three
places should not disagree.

Observed cost on a recent long run

A 264-node run took ~42 hours of wall clock. Ten distinct nodes hit Attempt 1 of 1. Splitting them
honestly by whether a retry could have helped at all:

class nodes would a retry help?
agent output rejected by a semantic gate 5 sometimes — see caveat
deterministic environment failure (capture overflow, ENOBUFS) 5 no

Caveat, stated because it cuts against this issue: I re-ran one of the five agent-output nodes and it
produced non-conforming output a second time, which is why I filed the underlying prompt/gate mismatch
separately. So a larger budget is not a substitute for fixing gate specifications, and I am not claiming
it would have saved all five. The claim is narrower: with a budget of 1, a stochastic or borderline
failure is never even retried, and every one of those five became terminal for a multi-day run on its
first attempt.

Why the default matters more than the knob

The knob already exists and works. The issue is that the safe-by-default value is the one that loses
runs, and the failure only materialises hours in, on the nodes furthest from the start. Unattended
evaluation is the primary use case for this tool, and it is the use case least able to absorb "resume it
by hand".

Suggested acceptance criteria

  • Decide the intended default explicitly and make ultrafuzz.toml,
    docs/reference/configuration.md and docs/config.md agree on it.
  • If the default stays 1, say so where an operator will see it, with the tradeoff: a single
    stochastic agent failure ends the run.
  • Consider decoupling the budget for deterministic failures (which should not be retried at all)
    from stochastic ones, so raising the default does not multiply the cost of a capture overflow
    that will fail identically every time.

Activity

  1. aviggiano commented on Aug 20, 2026

    @aviggiano
    CollaboratorAuthor
    • default should be 3
    • thorough should be 5
    • smoke/low-cost should be 1
  2. added this to the v0.1.0 milestone on Aug 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    In ProgressClaimed and actively worked on; has an assignee. Remove if work stops.bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions