Hard mutation gate: 75% total MSI and 80% covered MSI with mutarust

[jonbaldie]

Problem Statement

The messrust test suite passes, and the strict self-analysis gate passes. This
tells nobody if the tests detect a change of behaviour. A test can run a code
path and assert almost nothing. Today the repository has no measurement of test
strength, and no gate on it.

The sibling mutarust repository shows the failure mode to avoid. Its own
mutation workflow mutates four files, runs only the --lib suite, and uses
--git-diff-lines with --ignore-msi-with-no-mutations on pull requests. The
printed score is high, but it does not speak for the crate.

The maintainer wants a hard quality gate for messrust production code:

• 75 percent total MSI (mutation score indicator).
• 80 percent covered-code MSI.
• Full production scope, on every push to main and every pull request.
• No escape hatch. No baseline, no blacklist, no diff-only scope, no disabled
  mutators, no continue-on-error.

Solution

Measure the current mutation score of src with the released mutarust
command. Add tests through the existing library seam until the total score is 75
percent or more and the covered-code score is 80 percent or more. Then add a CI
job that runs the same command with --min-msi 75 --min-covered-msi 80. A score
below a threshold returns exit value 4 and fails the build.

Add one fast policy test to cargo test. The test reads the workflow file, the
mutarust.yml policy file, and the production sources. It fails if the
thresholds change, if a forbidden escape-hatch option appears, or if a
mutation-disable annotation appears in production source. The gate then protects
itself.

User Stories

1. As a crate maintainer, I want a mutation score gate on production code, so
   that a weak test cannot pass review.
2. As a crate maintainer, I want the gate set at 75 percent total MSI, so that
   the crate meets the "stable library" level in the mutarust CI guide.
3. As a crate maintainer, I want the gate set at 80 percent covered-code MSI, so
   that covered code is tested strongly, not only executed.
4. As a crate maintainer, I want both scores checked in one run, so that a
   contributor cannot satisfy one score and ignore the other.
5. As a crate maintainer, I want the gate to run on every pull request, so that
   a defect cannot enter main.
6. As a crate maintainer, I want the gate to run on every push to main, so
   that a merge does not lower the score without a signal.
7. As a crate maintainer, I want the gate to mutate all production source in
   src, so that no module is outside the measurement.
8. As a crate maintainer, I want no baseline file in the repository, so that a
   known escaped mutant cannot stay accepted.
9. As a crate maintainer, I want no blacklist file in the repository, so that an
   accepted checksum cannot hide a mutation.
10. As a crate maintainer, I want no --git-diff-lines scope in the gate, so
    that the score always speaks for the full crate.
11. As a crate maintainer, I want no --ignore-msi-with-no-mutations option, so
    that an empty mutant set cannot pass the gate.
12. As a crate maintainer, I want no disabled mutators, so that a difficult
    mutator cannot be removed to raise the score.
13. As a crate maintainer, I want no exclude_dirs and no ignore_source_lines
    policy values, so that source cannot leave the scope quietly.
14. As a crate maintainer, I want no mutator-disable annotations in src, so
    that a source comment cannot remove a mutation.
15. As a crate maintainer, I want no continue-on-error and no || true in the
    workflow, so that a failed gate cannot report success.
16. As a crate maintainer, I want the mutation job to be a required status
    check, so that a pull request cannot merge with a failed gate.
17. As a contributor, I want one documented local command, so that I can measure
    the score before I open a pull request.
18. As a contributor, I want a committed mutarust.yml policy file, so that my
    local run uses the same thresholds as CI.
19. As a contributor, I want escaped mutants shown as GitHub annotations, so
    that I can see the exact line to test.
20. As a contributor, I want the failure output to name the mutator and the
    source line, so that I can write a test without a second run.
21. As a contributor, I want the new tests to use the existing run seam, so
    that I do not learn a new test style.
22. As a contributor, I want a fast cargo test suite, so that the mutation run
    stays in CI and does not slow local work.
23. As an AFK agent, I want a binary success checkpoint, so that I know when the
    task is complete.
24. As an AFK agent, I want the forbidden option list written down, so that I do
    not add an escape hatch to make the gate pass.
25. As an AFK agent, I want the instruction to delete unreachable production
    code instead of adding a unit test for it, so that I do not add a private
    seam.
26. As a reviewer, I want a policy test in cargo test, so that a weakened gate
    fails fast and visibly in the diff.
27. As a reviewer, I want the thresholds written in two places that must agree,
    so that a silent change is not possible.
28. As a reviewer, I want a README section about mutation testing, so that a new
    contributor finds the rules.
29. As a release manager, I want the mutation score measured before a release,
    so that a published version has strong tests.
30. As a CI operator, I want the mutation job in its own workflow file, so that
    the fast test job stays fast.
31. As a CI operator, I want a job timeout, so that a stuck run does not hold a
    runner for six hours.
32. As a CI operator, I want the mutarust version pinned, so that a new
    release of the tool does not change the gate result without a commit.
33. As a CI operator, I want actions pinned by commit identifier, so that the
    workflow matches the security level of the sibling repository.
34. As a maintainer of the test suite, I want the packaging smoke test excluded
    from the per-mutant command, so that the run does not do a cargo install
    for each mutant.
35. As a maintainer of the test suite, I want a written statement that this
    exclusion lowers the score and never raises it, so that it is not an escape
    hatch.
36. As a maintainer, I want the mutation job to fail if the tool cannot collect
    coverage, so that the covered-code gate cannot pass without coverage data.
37. As a maintainer, I want the score printed in the job log, so that I can see
    the margin above the threshold.

Implementation Decisions

Tool and installation

• Use the released mutarust command from crates.io. Do not add it as a
  dependency of messrust.
• Install it in CI with cargo install mutarust --locked --version <pinned>.
  The current release is 0.1.1. A version change is a commit.
• Install cargo-llvm-cov with --locked, and add the llvm-tools-preview
  toolchain component. --coverage needs both.

Scope

• The gate mutates all production source of the crate. Before the workflow is
  written, run mutarust --list-files and confirm the output holds every file
  in src: analyze.rs, discover.rs, lib.rs, main.rs, metrics.rs,
  report.rs, ruleset.rs, suppressions.rs.
• Use the workspace target . if that output is complete. If the workspace
  target selects only the declared lib.rs and main.rs paths, use the
  recursive target src... instead. Record the chosen target in the workflow.
• Do not name individual files in the command. A file list becomes stale when a
  module is added.

Gate command

• One run does all the work: --coverage, --min-msi 75,
  --min-covered-msi 80, --logger-github.
• --coverage is mandatory. A positive covered-score gate without --coverage
  returns exit value 4, so the option cannot be dropped quietly.
• Exit value 4 fails the job. The workflow does not trap it.
• Set --exec-timeout to a value that fits the full suite, and --workers to
  the runner CPU count. These options change speed, not the score.

Per-mutant test command

• Each mutant runs the full cargo test suite, less the packaging smoke test,
  with --test-flags "-- --skip pack_install_smoke".
• Reason: tests/pack_install_smoke.rs runs cargo package and
  cargo install. A per-mutant packaging run is recursive and very slow.
• This exclusion removes killing power. It can only lower the measured score.
  It is therefore not an escape hatch, and the two thresholds still apply.
• Do not use --test-flags "--lib". The crate has almost no library unit tests,
  so that option would make the gate meaningless.
• Do not add a target list such as --test cli --test naming. A new test file
  would then leave the mutation run without a signal.

Forbidden in the gate

The workflow, the policy file, and the repository must not hold any of these:

• --baseline, --fail-on-escaped, mutarust-baseline.json
• --blacklist
• --ignore-msi-with-no-mutations
• --git-diff-lines, --git-diff-base
• --match, --run-mutant-id, --dry-run, --no-exec, --exec
• disable_mutators, enable_mutators with values
• exclude_dirs, ignore_source_lines with values
• skip_without_test: true, skip_with_cfg: true
• continue-on-error, || true, if: conditions on the mutation step
• a paths: filter or a branch filter that lets a pull request skip the job
• workflow_dispatch as the only trigger
• // mutator-disable-func, // mutator-disable-next-line, or
  // mutator-disable-regexp in any file under src

Policy file

• Commit mutarust.yml at the repository root with min_msi: 75,
  min_covered_msi: 80, all Boolean fields false, and all lists empty.
• The CI command repeats the same two numbers as command options. A command
  option takes priority over the file, so the two values must agree. The policy
  test asserts that they agree.

Workflow

• Add .github/workflows/mutation.yml. Do not put the job in ci.yml, because
  the fast test job must stay fast.
• Triggers: push to main and pull_request. No paths filter.
• Set timeout-minutes to a value near 120.
• Pin actions/checkout and dtolnay/rust-toolchain by full commit
  identifier, as the sibling repository does.
• The maintainer makes the job a required status check in branch protection.
  This is a human step outside the code.

Score work

• Raise the score with new tests through messrust::run. Do not add a public
  function only to make a test possible.
• If a mutant cannot be killed through the command seam, the mutated code is not
  reachable from the command. Delete that code. Do not add a #[cfg(test)] unit
  test to protect it.
• Do not change rule behaviour to make mutants easier to kill.

Documentation

• Add a "Mutation testing" section to README.md with the local command, the
  two thresholds, and the rule that the thresholds do not move down.
• Add the mutation workflow badge to the README title line.

Testing Decisions

What makes a good test here

A good test drives the command from the outside and asserts what a user sees.
The test builds an argument vector, calls messrust::run, and asserts the exit
code, the standard output text, and the standard error text. It does not call a
private function, and it does not assert an internal data shape. A mutation that
changes the report a user reads must fail such a test.

Seams

• Primary seam, already present: messrust::run(&[String], &mut dyn Write, &mut dyn Write) -> i32. All new tests use it. Prior art: tests/cli.rs,
  tests/codesize.rs, tests/naming.rs, tests/design.rs,
  tests/cleancode.rs, tests/controversial.rs, tests/unusedcode.rs.
• Process seam, already present: tests/pack_install_smoke.rs spawns the
  installed binary. Do not add tests here. It is slow.
• New policy seam: the repository files themselves. One new test file, for
  example tests/mutation_gate.rs, reads the workflow file, mutarust.yml, and
  the files in src as text. Prior art: tests/self_analysis.rs, which runs
  the crate policy against the crate itself in cargo test.

No other seam is added.

Modules under test

• Every module in src is under mutation. analyze.rs is 3269 lines and holds
  most of the rule logic, so it needs most of the new tests.
• The report formats in report.rs (text, json, sarif, github) need assertions
  on the exact output text. A format mutation escapes when a test asserts only
  the exit code.
• ruleset.rs, suppressions.rs, discover.rs, and metrics.rs need tests
  for their error paths, because an error message mutation escapes when no test
  reads standard error.

Policy test assertions (binary)

1. mutarust.yml holds min_msi: 75 and min_covered_msi: 80.
2. The workflow command holds --min-msi 75 and --min-covered-msi 80.
3. The workflow command holds --coverage.
4. The workflow holds a pull_request trigger and no paths filter.
5. No forbidden token from the list above is in the workflow or the policy file.
6. No mutator-disable comment is in any file under src.

Success checkpoints (binary)

1. mutarust --list-files <target> lists all eight files in src.
2. The full local gate command exits 0, prints a total score of 75 percent or
   more, and prints a covered-code score of 80 percent or more.
3. cargo test passes and includes the new policy test.
4. cargo clippy --all-targets adds no new warning.
5. The mutation job passes on a pull request in GitHub Actions.
6. A temporary threshold raise makes the job fail with exit value 4. Revert this
   change after the check.

Out of Scope

• A higher threshold than 75 and 80. A later issue can raise them.
• A line-coverage percentage gate. Coverage is collected for the covered-code
  score only.
• Mutation of test code. mutarust does not mutate #[cfg(test)] items, and
  the tests directory is not production source.
• Any change to mutarust itself, or to its own mutation workflow.
• Any change to rule behaviour, to the ruleset catalogue, or to the report
  formats, except a change that a mutant proves to be dead code.
• Any change to the existing strict self-analysis gate.
• --per-test mode as a requirement. Use it only if the job time needs it.
• Publication of the crate to crates.io.

Further Notes

• Expect the test work to be large. analyze.rs holds most of the rule logic,
  and the current tests assert findings more than they assert exact output.
• Sequence the work as: measure, then write tests, then add the gate. Do not
  commit a baseline file as an interim step. A baseline is an escape hatch.
• If the job time is too long, the correct actions are more workers, a longer
  timeout, or --per-test. A smaller scope and a lower threshold are not
  correct actions.
• The toolchain must be Rust stable 1.85 or newer. AGENTS.md says Cargo.lock
  is not in git; the file is tracked, and the CI jobs use --locked.
• mutarust exit values: 0 pass, 1 command error, 3 invalid annotation, 4 gate
  failure.

[jonbaldie]

Amendment: the gate is diff-aware on pull requests

The maintainer changed the gate design. #48 now specifies two modes:

• Pull request — mutate only the changed production source lines. Options
  --git-diff-lines, --git-diff-base, and --ignore-msi-with-no-mutations
  are permitted here, and only here.
• Push to main — mutate all production source. No scope-narrowing option
  is permitted.

Both modes keep --coverage, --min-msi 75, and --min-covered-msi 80. A
lower threshold in the diff-aware mode is not permitted.

This amends two parts of the body above:

• User story 10 ("no --git-diff-lines scope") and user story 11 ("no
  --ignore-msi-with-no-mutations") now apply to the push path only.
• The forbidden list entries for --git-diff-lines, --git-diff-base, and
  --ignore-msi-with-no-mutations now apply to the push path only. Every other
  forbidden entry applies to both paths.

Accepted residual risk. A pull request can pass the diff-aware gate and
still lower the whole-crate score. The job on main reports that fall after
the merge. It does not prevent the merge. The policy test in #48 keeps the two
modes at the same thresholds, so the diff-aware mode cannot become a soft gate.

[jonbaldie]

Handoff (ship-spec session stopped)

Parent goal

Ship child issues under #33 serially to main until the hard 75/80 mutation gate is real.

Child progress (repo state at stop)

Closed (merged earlier, not this session): #34 harness, #35 discover, #36 metrics, #38 ruleset, #39 report, #44 cleancode.

Open, ready-for-agent (dependency order still applies):
#37 suppressions (in progress), #40 CLI entry, #41 codesize, #42 naming, #43 unusedcode, #45 design, #46 controversial, #47 analyze, then gate issues #48 / remaining #33 workflow work.

This session focused on #37 only

See the handoff comment on #37 for full detail. Summary:

• Branch issue-37-suppressions has committed 27 tests plus an uncommitted expansion to 95 tests.
• Full-file mutarust (321 mutants) OOMs / freezes 16 GB hosts even at --workers 1.
• Switched to --match <function> batches; partial scores only (from_source 70%, apply_enables 87.5%, apply_disables / add_directive 100%). Four known escapes are documented on Mutation score 75/80 for suppression directives #37.
• No PR. Scores not yet at 75/80. Parent Hard mutation gate: 75% total MSI and 80% covered MSI with mutarust #33 still blocked on children.

Coordination notes for the next agent

1. Finish Mutation score 75/80 for suppression directives #37 first (handoff on that issue). One child at a time per ship-spec.
2. Memory rule on this hardware: never run unbounded full-crate or large-file mutarust without a free-RAM watchdog; prefer function batches and CARGO_BUILD_JOBS=1.
3. Stale Claude worktrees exist for Mutation score 75/80 for suppression directives #37/Mutation score 75/80 for the command entry and exit codes #40/Mutation score 75/80 for the naming rules #42/Mutation score 75/80 for the design rules #45/Mutation score 75/80 for the controversial rules #46 — verify before reuse; prefer a fresh branch from latest origin/main.
4. Do not open the CI gate workflow (Hard CI mutation gate at 75/80: diff-aware on pull requests, full scope on main #48 / final Hard mutation gate: 75% total MSI and 80% covered MSI with mutarust #33 gate pieces) until module scores are actually in range; a failing required check would block the repo.

[jonbaldie]

Process note: function-scoped mutarust batches only

On 16 GB hosts, a full-file mutarust --coverage run with the full suite (even at --workers 1 and CARGO_BUILD_JOBS=1) can push multi‑tens of GB of memory pressure and freeze the machine. Root-module targets such as src/lib.rs are worst case: every mutant rebuilds the whole crate, then runs every integration test.

Until a module’s scores look safe, agents must measure only in small function-scoped batches:

1. Prefer --match <function_name> on one function at a time (or a tiny related set).
2. Prefer a narrow test filter while iterating, e.g.
   --test-flags "--test <relevant_test_file> --lib"
   not the full suite.
3. Keep --workers 1 and CARGO_BUILD_JOBS=1 (or lower).
4. On macOS, set TMPDIR to a real path under $HOME before any --coverage run (see README). Default /var/folders breaks LCOV path restore.
5. Do not start a full-file acceptance gate (--coverage, full suite less pack_install_smoke, scoped only by file) until batch scores are clearly near or above 75 / 80.
6. Kill runaway mutarust / rustc / cargo processes and clear mutarust-* temp dirs if memory spikes.

Final acceptance for each child still needs the issue’s documented gate command when ready — but only after batches show the gate is likely to pass without thrashing the host.

[jonbaldie]

Progress halt

Stopped mid-#40 after host memory pressure. No merge this session.

• Mutation score 75/80 for the command entry and exit codes #40 WIP on issue-40-cli-entry (pushed): stronger tests/cli.rs + memory-gated scripts/mutarust-fn-batch.sh. Partial function batches only (handle_info_flags 100% MSI). Acceptance gate not run.
• Open children remain: Mutation score 75/80 for the command entry and exit codes #40–43, Mutation score 75/80 for the design rules #45–48.
• Continue function-scoped batches only; do not start full-file covered gates on 16 GB hosts until batches look safe.