Agent-failure retries replay a byte-identical prompt, so a deterministic agent decision burns the whole retry budget #376

Summary

When a durable agentic node fails with agent-failure / executor-error, the runtime retries it up to
max_attempts times — but the retried agent receives a byte-identical prompt and no indication that a
previous attempt existed or why it failed. For a deterministic failure caused by an agent decision, every
retry reproduces the same decision. The retry budget is spent as a replay.

R53 is a clean reproduction: four attempts, four identical failures, ~35 minutes of node execution, run
dead. See #368 for the underlying capture bug and
this comment for the
per-attempt table.

Evidence

aave-v4-v0012-main-issue357fix-r53-invariant-only, node stateful-invariant-handlers, retry_count: 3.
From attempts.jsonl — all four outcome: failed / executor-error:

Each attempt independently created recon-corpus-deep/ and echidna-deep/ and overflowed the workspace
capture buffer the same way. The fourth wrote more into them than the first (echidna-deep 37453 B / 15
files → 50561 B / 20 files). The run finished failed at 23:25:05Z with 8 downstream nodes never started.

Root cause

The prompt is rendered once, at plan time, before any execution, and is immutable thereafter.

• packages/runtime/src/plan-run.ts:209 → renderPromptsForPlan (plan-run.ts:790-870) →
  renderPrompt (packages/prompts/src/render.ts:229), written to <artifactDir>/prompt.rendered.md
  (render.ts:280, render.ts:356-360).
• SUPPORTED_TEMPLATE_VARIABLES (packages/prompts/src/render.ts:8-31) contains attempt_index and
  strategy_loop_index but no last_error / previous_failure / retry_reason. render.ts:194-197
  hard-errors on unknown variables, so a prompt author cannot reference the prior failure even
  deliberately.
• Execution-time assembly, packages/runtime/src/templates/smithers/workflows/workflow.tsx:4795:
  authorizedDefensiveSecurityContext + untrustedContentBoundary + task.runtimeContext +
  operatorPrompt + promptForTask(...). Every piece is a constant or the plan-time file;
  promptForTask (workflow.tsx:156-171) just reads prompt.rendered.md and rewrites path prefixes.
  Nothing consults run state, last_error, or the attempt ledger.
• resetTaskArtifactsForRetry (workflow.tsx:250-298, called at workflow.tsx:208-210) deletes the
  previous attempt's artifact dir (preserving only prompt.rendered.md), so the agent cannot infer the
  prior outcome from its own outputs either.
• The failure text is captured — packages/runtime/src/workflow-sync.ts:1836 writes last_error into
  run state, and packages/dashboard/src/index.ts:1844 shows it to humans. It is simply never routed back
  to the thing that has to do better next time.

The only cross-attempt signals reaching the agent are the SMITHERS_ATTEMPT env var (a number) and, on
resume paths, its own pre-failure CLI transcript. The latter does not help: the failure is thrown after
agent.generate() returns, at verifyArtifacts(task) (workflow.tsx:219-243), so the reason was never in
that conversation.

Note this is not the engine's fault. schemaRetry (@smithers-orchestrator/engine) does feed the
validation error back on intra-attempt structured-output correction — the durable node retry is the path
with no feedback channel.

Why this is worth fixing independently of #368

#368's fix makes this particular decision harmless. It does not change the fact that any
deterministically-wrong agent decision consumes the whole retry budget:

• The retry budget currently only helps non-deterministic failures (transient API errors, sandbox
  deaths, flaky tool calls). For anything the agent decided, max_attempts: N costs N× the wall clock and
  buys nothing.
• Cost is real and load-bearing: 35 minutes of paid model execution here, plus the run-level deadline
  consumed, for one decision.
• It also silently degrades every future fix of this shape. Each time we harden a capture/contract against
  one agent behaviour, the next unhandled behaviour still burns 4 attempts before surfacing.

Proposed design (not presuming it is the right one)

Thread the prior attempt's failure into the retried agent's context, inside the existing untrusted-content
boundary:

1. On retry, read the previous attempt's terminal error from the attempt ledger / run state (already
   persisted at workflow-sync.ts:1836) and expose it as a new execution-time preamble section in the
   workflow.tsx:4795 concatenation — not as a template variable, since the prompt file is plan-time
   immutable and content-addressed by digest (plan-run.ts:431-447); rendering it per-attempt would break
   the snapshot digest contract.
2. Place it inside untrustedContentBoundary. Error text can contain agent-authored path names and tool
   output; it must not be able to issue instructions.
3. Preserve the artifact wipe (resetTaskArtifactsForRetry) — the goal is telling the agent why, not
   letting it resume broken state.
4. Bound the injected text (the R53 message is short, but a stack trace or captured stderr is not) and run
   it through the existing redaction path.

Open questions I do not want to presume on:

• Whether this should be opt-in per node (retry_feedback: true in topology) or unconditional.
• Whether verifier/contract failures should be fed back as-is, or summarized — the raw
  WORKSPACE_PATCH_SCHEMA_INVALID-style text may be more confusing than useful to the agent.
• Whether feeding the failure back risks the agent "fixing" the symptom destructively (e.g. deleting its
  own deliverable to shrink the diff), which is exactly the failure mode that got Exclude oversized untracked roots by size, and record what was left out #372 closed.

Acceptance

• A node that fails deterministically on attempt 1 receives, on attempt 2, text naming the prior
  failure.
• The injected text sits inside the untrusted-content boundary and is bounded in size.
• The plan-time prompt.rendered.md digest is unchanged by the injection (snapshot contract intact).
• A regression test asserts attempt 2's assembled prompt differs from attempt 1's only by the
  failure section.

Note for whoever picks this up

The fix lands in packages/runtime/src/templates/smithers/workflows/workflow.tsx, which currently has two
long-lived branches over it (#165, #366). I have deliberately not opened a PR to avoid re-conflicting them
mid-flight — worth coordinating on merge order rather than racing it.