Skip to content

Bind Aave v4 benchmark ground truth to the ScFuzzBench subject #453

Closed

Description

@aviggiano

Evidence

The read-only R60 preflight for #404 fails before any model work:

EVAL_GROUND_TRUTH_SUBJECT_MISSING

benchmarks/private-ground-truth/aave-v4-scfuzzbench/findings.yml still has a legacy top-level source and no schema_version or subject. #423 deliberately made the Modal/eval path require an exact subject repository and revision, so the file is no longer launchable as benchmark ground truth.

It would be incorrect to bind those 12 labels to upstream aave/aave-v4 at 6959e3219b5506bf2acae18551cbb2a68a5b8fba. The checked-in case study records that they belong to the injected ScFuzzBench fork. The verified exact subject is:

  • repository: https://github.com/scfuzzbench/aave-v4-scfuzzbench
  • revision: edd6c82721512540c8c90e7a36a4a8e19fd7bdf3
  • tree: 3b3910d0657e4a639888ccae45822af680de6ef6

The revision is reachable and differs from upstream by the injected contracts/harness (tests/recon/** among 26 changed files), matching the semantics of the 12 labels.

Acceptance

  • Migrate the checked-in document to ultrafuzz.eval-ground-truth.v1 with the exact ScFuzzBench repository/revision as subject.
  • Remove or rename the legacy identity-bearing source field so it cannot conflict with subject; retain truthful non-identity provenance if useful.
  • Preserve all 12 scored bugs and the two canaries byte-for-byte in meaning.
  • readGroundTruthDocument(..., { requireSubject: true }) and assertGroundTruthSubject pass only for the exact fork/revision and fail for upstream Aave or another revision.
  • Regression covers the real checked-in file, not only an in-memory fixture.
  • The Handoff: Aave v4 invariant-only end-to-end run — state, blockers, and next steps #404 invariant-only config targets the exact fork/revision, retains the 27 exclusions and strategy_loops: 3, and validates to 32 executable nodes at the current 16-agent/32-node concurrency ceiling.
  • No Modal run launches until this tracked migration is on its exact candidate SHA.

Refs #404, #405, #423, #413.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions