Skip to content

Revise default and exhaustive profiles toward stateful fuzzing #1130

Closed

Description

@aviggiano

Revise default and exhaustive audit profiles to shift effort from repeating ordinary strategies toward stateful fuzzing. Split from #1128.

Reviewed baseline: 6b1a253cfc3290bf04d9e530554a46192611bf19.

The objective is to reduce repetition of ordinary agent strategies and increase stateful fuzzing. The intended profile behavior is:

Control New default New exhaustive
Ordinary strategy loops 2, down from the effective 3 3, down from 5
Stateful invariant pipeline Included Included
Invariant properties selected High priority High and medium priority
Final stateful campaign fuzzing duration 1 hour 4 hours (14,400 seconds)
Invariant pipeline repetitions One execution of each stage One execution of each stage

These are defaults; deliberate project and one-run overrides retain their documented precedence. Ordinary strategy loops are separate from failure retries and from the fuzzer's internal execution of many randomized call sequences.

  • Add the existing five-stage stateful pipeline to the packaged default topology and its matching editable repository copy: stateful-invariant-setup, stateful-invariant-handlers, stateful-invariant-coverage, stateful-invariant-implement-properties, and stateful-invariant-campaign. Carry over the required commands, artifacts, specialist group defaults, and dependencies. Feed campaign results into deduplication, triage, and final reporting. Retain each stage's explicit loops: 1.
  • Set default's ordinary strategy group to two loops, and exhaustive's profile to three. Keep the topology's fallback and displayed effective loop count consistent with actual graph expansion. Preserve exhaustive's additional differential and dynamic specialist lanes. Avoid changing unrelated retry, model, and concurrency settings as a side effect.
  • Add profile-specific invariant property thresholds: default high; exhaustive medium, which includes high and medium. Update profile types, validators, resolved configuration schemas, configuration-origin/override reporting, and generated documentation together.
  • Fix scaffolding and default handling so automatically generated settings do not accidentally pin high priority or default loop counts when the user switches to exhaustive. Preserve explicit settings in existing projects. Account for the root TOML's current dual use as packaged defaults and editable project configuration; do not silently reinterpret an explicit user override.
  • Make normal default/exhaustive property selection obey the stated priority thresholds. reference_expectations are identifiers linking properties to externally expected checks; preserve these tags as reporting/provenance metadata, but do not let a tag silently force a low-priority property into these normal audit profiles. Keep mandatory expected-check handling explicit for benchmark-specific execution, including the existing invariant-only profile. An excluded expected check must remain visible as unselected, never recorded as implemented or fulfilled.
  • Update the implementation prompt, runtime artifact gates, shared property-provenance validation, and generated-workflow/final-report reconstruction to use the same resolved selection policy. Preserve tag validation and propagation for selected properties.
  • Set exhaustive's invariant_testing_fuzzer_timeout_seconds to 14400. This is four hours of supervised Recon fuzzing for one campaign over the shared selected property suite, not four hours per property. Setup, the deployment smoke, shutdown, and artifact finalization are additional time.
  • Give the exhaustive campaign node a recommended timeout_seconds: 16200 (4 hours 30 minutes), without increasing every specialist node. The current minimum is 600 seconds for smoke + 14,400 for fuzzing + 300 shutdown grace + 300 artifact finalization = 15,600 seconds; the recommendation leaves another ten minutes of headroom. Retain the runtime's insufficient-budget validation. Existing retry/deadline validation rejects a blanket five-hour node under five attempts, so validate the complete budget rather than only changing the fuzzer setting.
  • Align local and manual Modal execution resources, sandbox lifetimes, and enclosing deadlines with the resolved campaign budget. The cloud resource timeout is separate from the agent task timeout: allocate at least 16,200 seconds for the campaign resource, with Modal's additional 1,800-second lifecycle reserve producing a five-hour sandbox. Review the public full benchmark row's current 15,000-second limit, which is shorter than the new campaign minimum: supported manual four-hour runs must receive a compatible envelope or fail clearly before execution. Keep intentionally short smoke lanes explicitly bounded.
  • Fix generated Modal workspace configuration that currently hardcodes high priority and a one-hour fuzzer timeout (packages/modal/src/workspace-config.ts). It must inherit resolved profile values or declare an intentional benchmark override, rather than silently replacing exhaustive's medium/four-hour settings.
  • Preserve the overall workflow deadline and handle insufficient remaining time explicitly. Five existing attempts at a 4h30m campaign-node timeout could occupy 22h30m, leaving little of the 24-hour workflow budget for other stages. Verify cancellation and partial/timeout reporting; do not promise every retry fits or report an abbreviated campaign as complete.
  • Update and run focused tests for profile resolution, overrides, fresh-init profile switching, actual expanded loop counts, all five invariant stages reaching reporting, strict high/high-plus-medium selection, benchmark-specific expected-check handling, timeout arithmetic, and final coverage reconstruction. Regenerate profile documentation and run build, lint, type, and relevant release-validation checks using fixtures, without paid inference or cloud benchmark launches.

Relevant starting points: packages/config/audit-profiles.yml, packages/config/topologies/default.yml, packages/config/topologies/exhaustive.yml, .ultrafuzz/topology.yml, packages/config/src/resolve.ts, packages/config/src/audit-profiles.ts, packages/runtime/test/runtime.test.ts, and .ultrafuzz/prompts/strategies/invariants/implement-properties.md.

Acceptance: new default has two ordinary loops, high-priority stateful properties, and a one-hour final campaign; exhaustive has three ordinary loops, high-plus-medium properties, and a four-hour final campaign with compatible timeouts. Explicit overrides and benchmark-specific mandatory expectations remain correctly handled.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions