Skip to content

Selectively convert high-yield strategy nodes to NoFuzz bug-search framing #591

Closed as not planned

Description

@aviggiano

Context

The Aave v4 comparison in #531 exposed an important prompt/topology confound:

  • NoFuzz strategy nodes were framed as "find bugs associated with X".
  • The corresponding Ultrafuzz nodes were framed as "author focused Foundry tests for X" and required ultrafuzz/generated-tests@3 output.

That difference pushed the Ultrafuzz treatment toward broad test production rather than source-backed bug-hypothesis discovery. It also made the comparison unable to isolate the value of autonomous stateful fuzzing. NoFuzz can still author deterministic multi-call PoCs, so this task is about node objective and output contract—not about treating NoFuzz as single-transaction-only.

Goal

Investigate which strategy nodes, if any, should be converted from test-authoring prompts into NoFuzz-style bug-search nodes, then selectively convert the nodes supported by that investigation. Retain test-authoring framing for nodes whose job is to build harnesses, invariants, or differential tests.

Do not convert every strategy node globally. Identify candidates as part of this task and document why each converted node is a better fit for bug search than test authoring.

Proposed changes

  • Inventory strategy nodes by primary objective: bug-hypothesis discovery, harness/invariant construction, differential testing, or another documented category.
  • Investigate their prompts, topology roles, expected outputs, and prior finding yield to identify candidate nodes.
  • Define explicit criteria for when a node should use bug-search framing.
  • For selected nodes, replace "author focused Foundry tests for X" with an explicit objective equivalent to "find bugs associated with X".
  • Restore the false-positive guard: "A property that holds is not a finding."
  • Preserve the findings output and the requirement for a concrete, source-backed violation.
  • Remove mandatory ultrafuzz/generated-tests@3 output only from selected bug-search nodes.
  • Allow tests/PoCs as optional evidence, rather than making test generation the node's objective.
  • Keep harness, invariant, and differential-construction nodes test-authoring oriented.
  • Consider a serial design in which a bug-hypothesis node emits candidates and a later generated-harness/fuzzing node consumes those candidates.

Acceptance criteria

  • All current strategy nodes are inventoried and classified by primary objective.
  • Candidate nodes are identified through this investigation, rather than assumed in advance.
  • The selection criteria and per-node conversion rationale are documented.
  • Only reviewed bug-search nodes are converted; test-construction nodes retain their existing role.
  • Every selected prompt has an explicit bug-search objective and the false-positive guard.
  • Mandatory generated-test output is removed only for selected nodes; findings output and source-backed violation requirements remain.
  • Topology/prompt regression tests prove that selected nodes use bug-search framing, do not require generated tests, and still emit findings.
  • A bounded blinded/paired prompt A/B reports token use, generated test LOC, and candidate/confirmed-finding yield before broad rollout.
  • A rollback path is documented.

Non-goals

  • Globally replacing generated-test authoring.
  • Removing stateful fuzzing or invariant campaigns.
  • Treating every model-authored multi-call PoC as equivalent to autonomous sequence search.

Follow-up to #531.

Activity

  1. added this to the v0.1.0 milestone on Aug 14, 2026
  2. aviggiano commented on Aug 14, 2026

    @aviggiano
    CollaboratorAuthor

    Consolidating this work into #5, the v0.1.0 human-review gate for all strategy prompts.

    #5 now carries this issue’s node inventory and classification work, bug-search selection criteria, per-node conversion rationale, output-contract guardrails, paired A/B validation, and rollback requirements.

    No prompt or output-contract conversion is authorized by this consolidation. Implementation remains blocked until the current prompt/topology set has been manually reviewed and a human maintainer has explicitly approved each proposed conversion.

    Closing this issue as consolidated into #5.

  3. removed this from the v0.1.0 milestone on Aug 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions