[aviggiano]

Summary

• organize benchmarks under evmbench, ultrafuzzbench, and scfuzzbench roots
• move cohorts, policies, labels, history, and archived results to their owning benchmark family
• update runtime, CLI, CI publication, cleanup, documentation, and test path consumers
• require canonical family-scoped policy paths with no root-layout fallbacks

Verification

• pnpm -w format:check
• pnpm -w lint
• pnpm -w typecheck
• pnpm -w benchmark:check:prebuilt
• pnpm -w docs:check
• pnpm -w test:ci-scripts (66 tests)
• pnpm --filter @ultrafuzz/evals test (392 tests)
• focused publication, launch, history, and manifest tests (64 tests)

Closes #513

Greptile Summary

The PR reorganizes benchmark manifests, labels, history, and archived results under benchmark-family directories and updates their runtime, CLI, CI, documentation, and test consumers.

• Moves EVMBench, UltrafuzzBench, and SCFuzzBench artifacts beneath family-specific roots.
• Updates benchmark preparation, Modal execution, history publication, workflow filters, and CLI defaults to use the new paths.
• Migrates checked-in history publication URLs and public result archives.

Confidence Score: 4/5

The PR is not yet safe to merge because cleanup and history publication for already-running pre-refactor candidates still fail on missing legacy policy paths.

The existing thread reply claims the cleanup path was fixed with a shared current-or-legacy resolver, but current HEAD hardcodes only family-scoped manifests in launch validation, publication preparation, and history scope validation, providing a concrete counterexample for legacy candidates.

Files Needing Attention: scripts/ci/validate-modal-benchmark-launch.mjs, scripts/ci/prepare-eval-history-publication.mjs, packages/evals/src/history.ts

Important Files Changed

Filename	Overview
scripts/ci/validate-modal-benchmark-launch.mjs	Switches launch and cleanup policy loading to canonical family-scoped manifests, but the previously reported legacy-candidate cleanup failure remains at HEAD.
scripts/ci/prepare-eval-history-publication.mjs	Uses the new family-scoped policy files consistently, while still rejecting legacy candidate checkouts during publication preparation.
packages/evals/src/history.ts	Updates public-generation validation to family-scoped manifests but provides no legacy candidate resolution for in-flight pre-refactor runs.
packages/modal/src/public-worker.ts	Updates public worker cohort, lane, and ground-truth paths consistently with the new benchmark layout.
scripts/ci/publish-eval-history-cas.mjs	Moves the CAS publisher’s permitted history path to the new UltrafuzzBench location.

_{Reviews (4): Last reviewed commit: "docs(evals): use canonical SCFuzzBench p..." | Re-trigger Greptile}

Context used:

• Knowledge Base — Evals
• Knowledge Base — Modal remote benchmark execution

[aviggiano]

@aviggiano please review