Repository navigation
Description
Reopens the substance of #2288 and #2289, which were closed by ce6ea628b on a measurement that was taken on the wrong tree.
What happened
ce6ea628b added a cargo test -p t27c --release step to corpus-ratchet.yml, justified by "1221 passed, 0 failed, 0 ignored". That measurement was real, and it was taken in the wrong working tree.
The host checkout at /Users/playom/t27 sits on feat/wave-547/host-heapsort, not master. The two trees are not close:
feat/wave-547/host-heapsort |
origin/master |
|
|---|---|---|
bootstrap/src/compiler.rs |
22,142 lines | 36,970 lines |
#[test] in bootstrap/src |
1,781 | 1,622 |
cargo test -p t27c main bin |
895 passed, 0 failed | 1602 passed, 13 failed, 2 ignored |
So the step went red on its very first run on master: https://github.com/gHashTag/t27/actions/runs/32330080992
The step is removed again in the PR that references this issue. It was live on master for about 34 minutes.
The 13 failing tests on master
Measured on ce6ea628b, ubuntu-latest, cargo test -p t27c --release:
compiler::tests_compiler_rejects::lowers_only_first_of_two_modules_characterization
compiler::tests_phase40_coverage::test_for_range_loop_unroll
compiler::tests_phase40_coverage::test_parse_for_range
compiler::tests_phase40_coverage::test_parse_for_range_c
compiler::tests_phase40_coverage::test_parse_for_range_nested
compiler::tests_phase40_coverage::test_parse_for_range_rust
compiler::tests_phase40_coverage::test_parse_for_range_verilog
compiler::tests_phase40_coverage::test_parse_for_range_with_expr
compiler::tests_w458::local_array_named_after_a_verilog_keyword_is_escaped
compiler::tests_w458::tests_w459::array_param_bound_from_test_block
compiler::tests_w458::tests_w459::test_block_emits_real_function_call
lex_conform::tests::lexer_matches_its_conformance_table
parse_conform::tests::parser_matches_its_conformance_table
This list is the main binary only and is therefore a lower bound. cargo test stops at the first failing target, so the 21 test binaries in bootstrap/tests/ never ran on master. Their pass/fail state is still unmeasured. A ratchet baseline must be built with --no-fail-fast.
What should happen next
Re-land it as a ratchet, not a gate:
- Run
cargo test -p t27c --release --no-fail-faston a clean master checkout to get the complete failing set (the 13 above plus whatever the other 21 binaries contribute). - Record those names in a baseline file next to
docs/reports/suite_expectations.json. - Fail only when a test outside the baseline fails, or a baselined test starts passing without the baseline being updated in the same commit.
- Print the failing count openly in the step output, so the number is visible rather than hidden inside an exit code.
The underlying defect from #2288 is unchanged and still real: eight workflows on master run cargo build --release -p t27c, cargo build does not compile #[cfg(test)] modules, and formal-yosys.yml — which does run the tests — has never landed on master. 1,622 #[test] attributes on master have never run in CI, and at least 13 of them do not pass.
Two things this episode also exposed
corpus-ratchet is not a required check, so this step could never have gated anything.
Corrected 2026-08-20 — see #2294. An earlier version of this issue said master had no branch protection, citing
GET /repos/gHashTag/t27/branches/master/protection→404 Branch not protected. That endpoint reports only legacy branch protection and 404s for a repository protected by rulesets. That claim was wrong.
Master is protected, by the active ruleset t27-master-protection: PRs required, deletion and non-fast-forward blocked, and four required status contexts —
check-now-freshness
validate
check
check-linked-issue
corpus-ratchet is not one of them, and neither are coverage or withdrawn-live — which is exactly why all three sit red on master while PRs keep merging. PR #2291 auto-merged at 03:57:13Z while its own corpus-ratchet run started at 03:57:16Z: three seconds before the gate it was adding began to run.
This sharpens the finding rather than softening it: the step was added to a workflow outside the required set, so even had it been green it could never have blocked a merge. It was wrong on its numbers and wrong in its placement. corpus-ratchet.yml's W632 (BLOCKING) header stays aspirational until the context is added to the ruleset — which must follow the ratchet, not precede it, since requiring a currently-failing check would block every PR in the repo.
corpus-ratchet was already failing on master before any of this — Run the corpus ratchet has failed on eeb779e4c, 57a53005a, 2255e4c32, bffd38982, and 400850702. Adding a step before it made things actively worse for 34 minutes: the unit-test failure short-circuited the job, so the pre-existing ratchet failure stopped being reported at all.
Still open, unaffected by the revert
- bootstrap/src/math_compare.rs is compiled by nothing; its plateau test contradicts its own goldens #2290 —
bootstrap/src/math_compare.rsis compiled by nothing; its plateau test contradicts its own goldens. bootstrap/Cargo.toml's[profile.release]is silently discarded because the workspace root has no[profile]section.
Boundary
bootstrap/Cargo.tomlbootstrap/src/compiler.rsbootstrap/src/math_compare.rs
User Scenarios
- Given a clean master checkout, when running
cargo test -p t27c --release --no-fail-fast, then the complete failing test set is captured (the 13 known failures plus any from the 21 bootstrap/tests binaries). - Given the baseline file recorded next to
docs/reports/suite_expectations.json, when a test outside the baseline fails or a baselined test starts passing without the baseline being updated in the same commit, then the ratchet step fails. - Given the ratchet step runs in
corpus-ratchet.yml, when it executes, then the failing count is printed openly in the step output so the number is visible rather than hidden inside an exit code.
Requirements
- FR-001: The workflow MUST run
cargo test -p t27c --release --no-fail-faston a clean master checkout to enumerate all failing tests across all 22 test binaries. - FR-002: A baseline file MUST be created next to
docs/reports/suite_expectations.jsoncontaining the full list of currently failing test names. - FR-003: The ratchet step MUST fail only when a test not in the baseline fails, or when a baselined test passes without the baseline being updated in the same commit.
- FR-004: The ratchet step MUST print the current failing count (baselined + new) to stdout so the number is visible in CI logs.
- FR-005: The
corpus-ratchet.ymlstep MUST be a ratchet (warn-on-new-failures) not a gate (hard-fail), until the ruleset adds its context as required.
Success Criteria
cargo test -p t27c --release --no-fail-fastruns to completion on master and reports a total failing test count (e.g., "13 + N failures") exit 0- A baseline file is created at
docs/reports/t27c_test_baseline.json(or similar name next tosuite_expectations.json) containing the enumerated failing test names exit 0 - The ratchet step in
corpus-ratchet.ymlexecutes, prints the failing count (e.g., "t27c ratchet: 13 baselined failures, 0 new failures"), and exits 0 when no new failures exist exit 0 - The ratchet step exits non-zero and prints "t27c ratchet: 13 baselined failures, 2 new failures" when a test outside the baseline fails exit 0
- 5 tests in
bootstrap/tests/that previously never ran in CI now execute under--no-fail-fastand their pass/fail state is recorded 5 tests
Activity
fix(corpus-ratchet): remove the t27c test step; it was measured on th…
docs(now): retract "master has no branch protection" -- wrong endpoint (
fix(compiler): stop dead-store elimination deleting writes to module …
fix(gen-verilog): a field of an array element of a struct resolves as…
fix(gen-verilog): an unannotated local keeps the type it was copied f…
fix(verilog): escape keyword-named identifiers where they are USED (#…
fix(fpga-build): the module set is the directory, not a literal (#2452)
feat(tri): read the one subset of open issues that re-measures itself (…
gHashTag commentedon Sep 23, 2026
A boundary drafted from the paths this issue already names, so a bee can take it. Nothing is reserved until this is in the issue body - the Queen reads the body, not comments. Check the paths first: a wrong boundary reserves files the work does not own.
Boundary
bootstrap/Cargo.tomlbootstrap/src/compiler.rsbootstrap/src/math_compare.rsdocs/reports/suite_expectations.json