Add bounded error-agnostic agent retries with backoff and optional model fallback #572

Scope: generic and error-agnostic. This was observed during the #531 paired Aave v4 evaluation, but the product gap is a configurable retry policy for ordinary agent failures, not refusal detection.

Problem

A stochastic model-provider failure can terminate a whole run. The observed failure happened on run aave-v4-v056-gpt55-xhigh-default-20260813-control at 94% complete (223 of 237 nodes finished, with no other failures):

node:dynamic-strategy-generator#0 retries-exhausted:
  All retries exhausted. Last error: Error: This content was flagged for possible
  cybersecurity risk. If this seems wrong, try rephrasing your request. To get
  authorized for security work, join the Trusted Access for Cyber program:
  https://chatgpt.com/cyber  (attempt 1/1)

The failure was stochastic rather than deterministic. Resuming with --retry-failed re-ran the same prompt and the node succeeded. In an unattended evaluation, requiring this manual intervention can lose a row and a day of compute.

Decision: dumb retries, not refusal parsing

Ultrafuzz must not inspect, parse, or classify provider error text for this feature. Any ordinary failure that Smithers already considers retryable should consume the same bounded retry policy, regardless of whether it is a refusal, transient provider error, CLI failure, or another thrown agent error. Existing Smithers handling of deterministic non-retryable configuration failures and quota parking remains unchanged; Ultrafuzz adds no new message matcher or failure taxonomy.

Every automatic retry must start a fresh model generation with the original prompt unchanged. In particular, it must not route through retryFailureAwareArgs, append the previous error, rephrase the request, or resume/continue the failed agent session. Retrying may continue to reset attempt-owned artifact outputs so files from different model attempts cannot be combined.

TOML configuration

Add a project-level policy whose agents entries are ordered references to existing model-profile IDs:

[retry]
same_agent_attempts = 3
agents = ["sol-xhigh", "gpt55-xhigh"]

[models.sol-xhigh]
agent = "CodexAgent"
model = "gpt-5.6-sol"
reasoning = "xhigh"

[models.gpt55-xhigh]
agent = "CodexAgent"
model = "gpt-5.5"
reasoning = "xhigh"

sol-xhigh and gpt55-xhigh are arbitrary profile IDs; Ultrafuzz must resolve them through their [models.<id>] tables. It must not infer the model or reasoning effort from the profile name.

same_agent_attempts is the total number of attempts on the primary profile, including the first attempt. It defaults to 1. agents[0] is the primary profile and each later entry is an opt-in fallback that receives one attempt after the primary budget is exhausted. If agents is absent, the node's normally selected model profile is the sole primary and model fallback is disabled. Node/group max_attempts remains the more-specific override for the primary attempt count.

Smithers projection

Stay on the currently pinned smithers-orchestrator@0.32.0 for this issue. It already supplies:

• <Task retries={N}>, where N is the number of additional attempts;
• exponential retry backoff through retryPolicy={{ backoff: "exponential", initialDelayMs }};
• ordered fallback agents through agent={[primary, fallback, ...]};
• durable attempt rows containing the selected chain index, agent ID, model, error, response, and timing.

Smithers chooses chain rung N-1 for attempt N. Therefore three primary attempts followed by one fallback must be projected as:

agent={[primary, primary, primary, fallback]}
retries={3}

For more fallbacks, append each profile once and set the task retry budget to expandedChain.length - 1. This preserves same-model retries before model fallback without parsing the failure.

Use Smithers' existing exponential backoff for the first implementation. Its delay is internally capped at five minutes and currently has no jitter or configurable maximum. Ultrafuzz currently emits maxDelayMs: 30000, but that field is not part of Smithers 0.32's RetryPolicy and is ignored; remove it from the Smithers projection rather than presenting it as effective configuration. Custom jitter or a configurable maximum delay is outside this issue.

The upstream changes described as Smithers 0.33.1 do not add a generic retry primitive, jitter, a configurable delay cap, or same-model-before-fallback behavior. There is no installable smithers-orchestrator@0.33.1; those changes are included in the renamed smthrs@0.34.0 line and focus on classified session-loss, quota/preflight failover, and detached recovery. Upgrading would also require the smithers-orchestrator to smthrs package/scope migration, so it is not part of this work.

Fallback provenance

Model fallback is off by default because it changes which model produced a node's output. Every configured and used fallback must be projected into Ultrafuzz run provenance and per-node metadata. report.json and eval provenance must expose the planned chain, failed primary attempts, and actual producing profile/model so consumers can exclude or annotate mixed-model runs. Benchmark and paired-evaluation lanes must be able to forbid fallback outright and fail closed if fallback is configured or observed.

Secondary recovery defect

The observed node could not be recovered with --reset-node because the command reported that its latest attempt was absent. Smithers 0.32 inserts an attempt row before agent selection/generation, so this likely lies in Ultrafuzz's linked-attempt/recovery projection. A node that failed before Ultrafuzz can resolve an attempt should still be resettable by node ID, or the diagnostic should name the recovery flag that does apply.

Acceptance criteria

• ultrafuzz.toml accepts a bounded primary-attempt count and an optional ordered list of model-profile IDs, validating that every referenced profile exists.
• A simulated ordinary agent failure on the first attempt is retried without inspecting or classifying its error text and can succeed without operator action.
• Each retry sends the original prompt unchanged in a fresh generation/session and does not append prior-failure feedback.
• Attempts use Smithers' bounded exponential backoff and stop at the configured budget; a persistent failure fails closed with the attempt count and original diagnostic.
• Same-model attempts are exhausted before an optional fallback runs; with three primary attempts and one fallback, the observed order is primary, primary, primary, fallback.
• Model fallback is disabled by default, and profile IDs such as gpt55-xhigh resolve explicitly to their declared agent, model = "gpt-5.5", and reasoning = "xhigh".
• Every attempt and fallback is durable, and report.json/eval provenance names the configured chain and actual producing profile/model; benchmark and paired-evaluation lanes can forbid fallback.
• The ineffective Smithers maxDelayMs projection is removed; no jitter or custom maximum-delay behavior is claimed by this issue.
• --reset-node recovers a node whose linked latest attempt cannot be resolved, or its diagnostic names the applicable recovery command.

Related: #531 (the evaluation that exposed this), #494.