Repository navigation
Selectively convert high-yield strategy nodes to NoFuzz bug-search framing #591
Description
Context
The Aave v4 comparison in #531 exposed an important prompt/topology confound:
- NoFuzz strategy nodes were framed as "find bugs associated with X".
- The corresponding Ultrafuzz nodes were framed as "author focused Foundry tests for X" and required
ultrafuzz/generated-tests@3output.
That difference pushed the Ultrafuzz treatment toward broad test production rather than source-backed bug-hypothesis discovery. It also made the comparison unable to isolate the value of autonomous stateful fuzzing. NoFuzz can still author deterministic multi-call PoCs, so this task is about node objective and output contract—not about treating NoFuzz as single-transaction-only.
Goal
Investigate which strategy nodes, if any, should be converted from test-authoring prompts into NoFuzz-style bug-search nodes, then selectively convert the nodes supported by that investigation. Retain test-authoring framing for nodes whose job is to build harnesses, invariants, or differential tests.
Do not convert every strategy node globally. Identify candidates as part of this task and document why each converted node is a better fit for bug search than test authoring.
Proposed changes
- Inventory strategy nodes by primary objective: bug-hypothesis discovery, harness/invariant construction, differential testing, or another documented category.
- Investigate their prompts, topology roles, expected outputs, and prior finding yield to identify candidate nodes.
- Define explicit criteria for when a node should use bug-search framing.
- For selected nodes, replace "author focused Foundry tests for X" with an explicit objective equivalent to "find bugs associated with X".
- Restore the false-positive guard: "A property that holds is not a finding."
- Preserve the findings output and the requirement for a concrete, source-backed violation.
- Remove mandatory
ultrafuzz/generated-tests@3output only from selected bug-search nodes. - Allow tests/PoCs as optional evidence, rather than making test generation the node's objective.
- Keep harness, invariant, and differential-construction nodes test-authoring oriented.
- Consider a serial design in which a bug-hypothesis node emits candidates and a later generated-harness/fuzzing node consumes those candidates.
Acceptance criteria
- All current strategy nodes are inventoried and classified by primary objective.
- Candidate nodes are identified through this investigation, rather than assumed in advance.
- The selection criteria and per-node conversion rationale are documented.
- Only reviewed bug-search nodes are converted; test-construction nodes retain their existing role.
- Every selected prompt has an explicit bug-search objective and the false-positive guard.
- Mandatory generated-test output is removed only for selected nodes; findings output and source-backed violation requirements remain.
- Topology/prompt regression tests prove that selected nodes use bug-search framing, do not require generated tests, and still emit findings.
- A bounded blinded/paired prompt A/B reports token use, generated test LOC, and candidate/confirmed-finding yield before broad rollout.
- A rollback path is documented.
Non-goals
- Globally replacing generated-test authoring.
- Removing stateful fuzzing or invariant campaigns.
- Treating every model-authored multi-call PoC as equivalent to autonomous sequence search.
Follow-up to #531.
Activity
aviggiano commentedon Aug 14, 2026
Consolidating this work into #5, the v0.1.0 human-review gate for all strategy prompts.
#5 now carries this issue’s node inventory and classification work, bug-search selection criteria, per-node conversion rationale, output-contract guardrails, paired A/B validation, and rollback requirements.
No prompt or output-contract conversion is authorized by this consolidation. Implementation remains blocked until the current prompt/topology set has been manually reviewed and a human maintainer has explicitly approved each proposed conversion.
Closing this issue as consolidated into #5.