Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 67 additions & 1 deletion .github/workflows/corpus-ratchet.yml
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,66 @@ jobs:
- name: Build t27c
run: cargo build --release -p t27c

# Closes #2288. `cargo build` does not compile `#[cfg(test)]` modules, so
# eight workflows on master build t27c and not one of them ever compiled,
# let alone ran, a single test in it. Measured 2026-08-20: 1221 tests
# exist and 1221 of them had never executed in CI. They all pass -- so
# this lands as a plain gate, not a ratchet.
#
# It goes here because the crate is already built one step above, so the
# marginal cost is compiling the test targets and running them: measured
# 1m43s cold / 1.75s warm, against this job's 45-minute ceiling and its
# ~314s ratchet run.
#
# Scope, stated openly -- this runs what `cargo test -p t27c` REACHES,
# which is not every test in the tree. 913 `#[test]` attributes live in
# bootstrap/src; 895 compile into this run. The missing 18:
# * bootstrap/src/math_compare.rs -- 10 tests, and no `mod math_compare;`
# exists anywhere in the crate, so rustc never opens the file. Built
# standalone it is 9 pass / 1 fail (see #2288 follow-up); the failure
# is a bad assertion, not a bad implementation, so it is NOT silenced
# here and NOT wired in here -- wiring it in would land this gate red.
# * bootstrap/src/proxy.rs -- 8 tests behind
# `#[cfg(all(test, feature = "server"))]`. No workflow passes
# `--features server`, so they are stripped before type-checking.
# Neither hole is created by this step; both are named so the green tick
# is not read as "every test in bootstrap/src ran".
- name: Unit tests (cargo test -p t27c)
id: unit_tests
run: |
set -o pipefail
cargo test -p t27c --release 2>&1 | tee unit_tests.log
rc=$?
totals=$(grep -E '^test result:' unit_tests.log \
| awk '{p+=$4; f+=$6; i+=$8} END {printf "%d passed, %d failed, %d ignored", p, f, i}')
bins=$(grep -cE '^test result:' unit_tests.log)
echo ""
echo "t27c unit tests: ${totals} across ${bins} test binaries."
{
echo "### t27c unit tests"
echo ""
echo "\`cargo test -p t27c --release\` -- **${totals}** across ${bins} test binaries."
echo ""
echo "Not covered: 10 tests in \`bootstrap/src/math_compare.rs\` (no \`mod\` declaration"
echo "anywhere, so rustc never compiles the file) and 8 in \`bootstrap/src/proxy.rs\`"
echo "(behind \`feature = \"server\"\`, which no workflow enables)."
} >> "$GITHUB_STEP_SUMMARY"
exit $rc

- name: Explain a unit-test failure
if: failure() && steps.unit_tests.outcome == 'failure'
run: |
echo "::error title=t27c unit tests failed::A Rust test in the t27c crate failed."
echo ""
echo "This is NOT the corpus ratchet and has nothing to do with"
echo "docs/reports/suite_expectations.json. Do not bless an expectation for it."
echo ""
echo "Reproduce locally with:"
echo " cargo test -p t27c --release"
echo ""
echo "The failing test names are in the step log above and in the"
echo "corpus-ratchet-log artifact (unit_tests.log)."

- name: Run the corpus ratchet
id: ratchet
run: |
Expand All @@ -71,8 +131,13 @@ jobs:
sed -n '/--- Ratchet (W628) ---/,$p' ratchet.log
exit $rc

# Scoped to the ratchet step. A bare `failure()` here fires for ANY earlier
# failure, so once a unit-test step exists above it, a failing Rust test
# would print "add an entry to suite_expectations.json" -- sending the
# reader to bless a corpus expectation for a defect that is not in the
# corpus. The guard keeps each failure explained by its own step.
- name: Explain a failure
if: failure()
if: failure() && steps.ratchet.outcome == 'failure'
run: |
echo "::error title=Corpus ratchet failed::See the Ratchet section above."
echo ""
Expand All @@ -98,4 +163,5 @@ jobs:
path: |
ratchet.log
suite_summary.json
unit_tests.log
retention-days: 14
77 changes: 77 additions & 0 deletions docs/NOW.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,80 @@
# NOW -- 1221 tests that had never run in CI now run in CI (2026-08-20)

Last updated: 2026-08-20

## ci(corpus-ratchet): `cargo test -p t27c` runs on master for the first time

Eight workflows on `origin/master` run `cargo build --release -p t27c`. `cargo build`
does not compile `#[cfg(test)]` modules. So every test in the compiler crate --
**1221 of them** -- was unbuilt and unrun on the branch that gates merges.

This adds one step to `.github/workflows/corpus-ratchet.yml`, which already builds
the crate one step above. Measured before landing: **1221 passed, 0 failed,
0 ignored**, across 22 test binaries, in `--debug` and `--release` alike. Because
nothing fails, it lands as a **plain gate, not a ratchet** -- a ratchet over an
all-green baseline is strictly worse, adding a file to maintain while ceasing to
detect the first regression it exists for.

It also scopes the pre-existing `Explain a failure` step to
`steps.ratchet.outcome == 'failure'`. It was a bare `failure()`, which with a test
step above it would answer a failing Rust unit test with "add an entry to
`docs/reports/suite_expectations.json`" -- sending the reader to bless a corpus
expectation for a defect that is not in the corpus.

### What #2288 got wrong, and the one thing it got right

#2288 said "36 unit tests in `bootstrap/src/suite.rs`"; there are **14**. It said the
tests had "never been type-checked, let alone executed"; on `feat/wave-547/host-heapsort`
they had -- `.github/workflows/formal-yosys.yml:899` has run `cargo test -p t27c --release`
since `0d8635842` (2026-08-12).

The real defect is that **`formal-yosys.yml` has never landed on `origin/master`**:

```
$ git cat-file -e origin/master:.github/workflows/formal-yosys.yml
fatal: path '.github/workflows/formal-yosys.yml' exists on disk, but not in 'origin/master'
```

A gate that exists only on a feature branch gates nothing.

### Honesty limits (BINDING)

- **1221 tests, 0 failures, and none of them had ever executed in CI on master
before today.** That is the whole measurement. The count is `cargo test -p t27c`'s
own total across 22 binaries (895 from `unittests src/main.rs`, 326 from the 21
files in `bootstrap/tests/`); there are no doc-tests, as t27c is a bin-only crate.
- **This gate does NOT run every test in `bootstrap/src`.** 913 `#[test]` attributes
live there; **895** compile into this run. The missing 18 are named in the workflow
comment and printed in the step summary so a green tick is not misread:
- `bootstrap/src/math_compare.rs` -- **10 tests, compiled by nothing**. No
`mod math_compare;` exists in the crate, so rustc never opens the file; it is
one of 8 orphan `.rs` files in `bootstrap/src` and the only one with tests.
Built standalone it is **9 pass / 1 fail**: `test_hybrid_v2_plateau` asserts
`|n152 - n20| < 1e-9` while the file's own `GOLDEN_V2` records values 2.100e-9
apart. Independent recomputation reproduces the goldens exactly, so **the test
is wrong, not the implementation**. It was NOT adjusted and NOT wired in --
wiring it in today would land this gate red. Tracked in #2290.
- `bootstrap/src/proxy.rs:326` -- 8 tests behind
`#[cfg(all(test, feature = "server"))]`. No workflow passes `--features server`,
so they are stripped before type-checking.
- **The failure path was verified by a shell-level proxy, not by a real failing
cargo test.** A shape-identical harness confirms the step exits non-zero when the
test command fails (exit 101), exits 0 when it passes, and does not false-fail when
`grep -c` matches nothing. No test was edited to produce a real red run, per the
rule that a wrong test is reported rather than adjusted.
- **Cost measured on one machine (darwin/arm64), not on the CI runner.** Release:
1m43s cold compile, 1.75s warm, +351 MiB in `target/`. The runner's cold number
will differ; the 45-minute ceiling has ~44 minutes of headroom against a ~314s job.
- **`bootstrap/Cargo.toml`'s `[profile.release]` is silently ignored** -- the workspace
root has no `[profile]` section, so cargo discards `lto`/`codegen-units`/`strip` on
every `--release` build in 8+ workflows. Real, unfixed, and deliberately out of
scope: it changes the binary every workflow produces. Recorded in #2289.
- **No claim is made that CI is now well-tested.** One crate's unit tests now run.
The 348-of-1114 specs that generate with no backend, and everything else the corpus
ratchet already tracks, are untouched by this.

---

# NOW -- an absent or corrupt gate input now fails instead of counting as zero (2026-08-20)

Last updated: 2026-08-20
Expand Down
Loading