ci(bench): run every benchmark harness module in the CI Gate (#1679)

[DecisionNerd]

Closes #1679

What changes

• All 41 benchmarks/tests modules now run in the CI Gate. Before this PR, 8 ran, from an explicit list in Rust Tests. benchmarks/tests is now a package, and a new Benchmark Harness Tests job runs unittest discover -v -s tests -t . -p 'test_*.py' from benchmarks/. A new module runs without being wired anywhere.

  • The job follows rust_tests, because the four GDC suites build Rust runners against graphforge-api.
  • It runs in parallel with Rust Tests, the CI Gate's longest job, instead of adding to it.
  • Every benchmarks/ path except benchmarks/algorithms/ already falls through the classifier to enable_all, and no harness module reads benchmarks/algorithms/. New classifier assertions pin this for test, harness and schema edits.

• Policy (scripts/ci/test-ci-storage-policy.py) requires four things, and negative fixtures cover each one:

  • exactly one lane runs unittest from benchmarks/, and it runs the exact discovery command, not a module list;
  • that lane is gated on rust_tests;
  • that lane feeds CI Gate;
  • every test_*.py under benchmarks/tests can be reached through packages.

  The dependency-cache inventory moves from 11 to 12 for the new job's registry cache.

• test_smoke: test_checked_in_fixtures_are_discoverable failed on g500-ladder-qualification.schema.json. docs: move evidence out of the tree, prune orphan dev docs, give ADRs a lifecycle (#1625) #1628 moved that schema from docs/ into benchmarks/schemas/, next to its only validator (progressive_storage_qualification.py). The move itself was right, and moving the file back would break docs: stop committing per-issue evidence to the docs tree, prune unreferenced development docs, and give ADRs a lifecycle #1625's docs-tree rule. Every other document in benchmarks/schemas/ has a top-level graphforge-*-schema/N identity, and discover_fixtures requires one, so this PR adds "schema": "graphforge-g500-ladder-qualification-schema/4". JSON Schema 2020-12 treats an unknown keyword as an annotation, so validation doesn't change, and test_progressive_storage_qualification and test_native_ladder_controller still pass. make -C benchmarks smoke failed for the same reason and now passes.

• test_measurement_tools: this was the only pytest module in a unittest-only locked environment. It is rewritten as unittest subtests covering the same 16 cases, with no new dependency. The conversion exposed a second bug that pytest had been masking. The stub sudo runs cat >/dev/null, and run-f1.sh calls it without redirecting stdin, so the fio case hung whenever the process inherited an open stdin. Every tool invocation now passes stdin=subprocess.DEVNULL.

• make -C benchmarks smoke-python now uses the same package discovery. Its old -s tests form loaded modules twice, as test_x and as tests.test_x.

Evidence

Exact CI command, run from a fresh worktree with no benchmarks/target, CARGO_BUILD_JOBS=4, ext4 TMPDIR:
PYTHONPATH=harness uv run --locked python -m unittest discover -v -s tests -t . -p 'test_*.py'
Result: Ran 528 tests in 263.539s, OK, 41 modules, 0 skipped. The 528 is the issue's 453, plus 69 from test_tck_perf (#1680), 1 new in test_benchexec_authority, and 5 more from test_measurement_tools (6 methods instead of 1 import error). The per-module sum matches the discovery total, so nothing is loaded twice.

Mutations:

• a new benchmarks/tests/test_zz_mutation_probe.py with a failing test was discovered: Ran 529, FAILED (failures=1), rc=1;
• benchmarks/tests/probe/test_probe.py in a directory without __init__.py fails the policy;
• removing tests/__init__.py fails both the policy and discovery;
• going back to a module list fails the policy;
• dropping the CI Gate edge fails the policy;
• removing the watchdog restore in run-perf.sh fails all 6 test_measurement_tools watchdog subtests.

Gates: make pre-push-fast, python3 scripts/ci/test-ci-storage-policy.py, bash scripts/ci/test-classify-changes.sh, make gate-registry-check, scripts/check-workflows.sh, bash scripts/ci/test-require-gates.sh, python3 scripts/ci/benchmark-measurement-policy.py and python3 scripts/ci/test-benchmark-measurement-policy.py all pass.

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: CurateLabs/graphforge/.coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 958e6fc5-3428-494b-a9e5-ec251c187137

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.

Warning

Billing warning: we have not been able to collect payment for this subscription for more than 72 hours. Please update the payment method or pay any pending invoices in Billing to avoid service interruption.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}