ci: nightly workspace mutants, drop the PR shards

.agents/skills/ship-pr/SKILL.md

@@ -74,12 +74,10 @@ Coverage and native `windows` / `macos` stay GitHub Actions;
 
 ## Push and poll
 
-Required jobs: `qc`, `test`, `windows`, `macos`, `coverage`, and
-`mutants (1/4)` through `mutants (4/4)` (`--shard` stays 0–3). `qc` is
+Required jobs: `qc`, `test`, `windows`, `macos`, and `coverage`. `qc` is
 fmt, ast-grep, deny, the script self-tests, clippy, then
-nixos-module-eval on one runner. A mutants shard killed at
-30 minutes with no `MISSED` warns and passes. `MISSED` before that kill fails the shard. A
-finished non-zero `cargo mutants` exit fails. Structural scan is
+nixos-module-eval on one runner. Mutants run nightly with workspace
+tests (`mutants.yml`); they are not a required check. Structural scan is
 `./scripts/ast-grep.sh`. `windows` / `macos` are native store plus `--smoke`,
 not operator zips.
 

.github/workflows/ci.yml

@@ -26,13 +26,7 @@ env:
 # `qc` is one Ubuntu runner: fmt, ast-grep, deny, scripts, clippy, then
 # nixos-module-eval. test, windows, and macos each have their own runner.
 # coverage waits for qc and test so a red test does not start it.
-# Mutants checks are `mutants (1/4)` through `mutants (4/4)` (`--shard`
-# stays 0–3). They wait for the same three. Under 400 changed lines,
-# `mutants (1/4)` runs once (`--shard 0/1`); the other three exit 0.
-# A 30m kill with no MISSED in the log passes and posts a warning.
-# MISSED before that kill fails the shard. A finished non-zero
-# cargo-mutants exit fails.
-# Weekly sweep stays mutants.yml.
+# Mutants are not a PR check. Nightly workspace runs live in mutants.yml.
 # nixos-module-runtime is a labeled job (nixos-module-runtime.yml), not this file.
 # Operator binaries are GitHub Releases (release.yml), not this workflow.
 #
@@ -96,6 +90,8 @@ jobs:
         run: ./scripts/ci-os-smoke.test.sh
       - name: pr-checks-watch self-test
         run: ./scripts/pr-checks-watch.test.sh
+      - name: mutants queue self-test
+        run: ./scripts/mutants-queue.test.sh
       - name: overlay-functional runner self-test
         run: ./scripts/overlay-functional/run.sh.test.sh
       - uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
@@ -174,154 +170,6 @@ jobs:
       - name: store + node smoke
         run: ./scripts/ci-os-smoke.sh
 
-  # PR in-diff. Required checks `mutants (1/4)` through `mutants (4/4)`
-  # (`--shard` 0–3). Under 400 changed lines, mutants (1/4) runs the
-  # whole in-diff set (`--shard 0/1`) and the other three exit 0.
-  # A 30m kill with no MISSED passes and posts a warning. MISSED already
-  # in that log fails the shard. A finished non-zero exit fails.
-  # Docs-only diffs exit 0. Weekly sweep stays mutants.yml. Owner: TESTING.md.
-  mutants:
-    name: mutants (${{ matrix.n }}/4)
-    needs: [qc, test]
-    if: >-
-      github.event_name == 'pull_request'
-      && needs.qc.result == 'success'
-      && needs.test.result == 'success'
-    runs-on: ubuntu-latest
-    timeout-minutes: 40
-    strategy:
-      fail-fast: false
-      # Expressions have no + operator; a `+` in ${{ }} invalidates this file.
-      # n is the check number. shard is the cargo --shard argument.
-      matrix:
-        include:
-          - shard: 0
-            n: 1
-          - shard: 1
-            n: 2
-          - shard: 2
-            n: 3
-          - shard: 3
-            n: 4
-    env:
-      # Workflow env disables incremental for rust-cache and denies warnings.
-      # cargo-mutants rebuilds one mutant at a time; incremental is the point.
-      # Weekly mutants.yml does not pass -D warnings. A warning-only build
-      # failure would look caught here and missed on Sunday.
-      CARGO_INCREMENTAL: "1"
-      CARGO_TERM_COLOR: never
-      RUSTFLAGS: ""
-    steps:
-      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
-        with:
-          fetch-depth: 0
-      - name: rust or workflow diff
-        id: diff
-        env:
-          BASE_REF: ${{ github.base_ref }}
-        run: |
-          set -euo pipefail
-          git diff "origin/${BASE_REF}".. --unified=0 -- \
-            crates Cargo.toml Cargo.lock \
-            .github/workflows/ci.yml .github/workflows/mutants.yml \
-            > git.diff
-          if [[ ! -s git.diff ]]; then
-            echo "rust=false" >> "$GITHUB_OUTPUT"
-            echo "small=false" >> "$GITHUB_OUTPUT"
-            echo "no rust or workflow diff; shard passes"
-            exit 0
-          fi
-          echo "rust=true" >> "$GITHUB_OUTPUT"
-          lines="$(grep -E '^[+-]' git.diff | grep -cvE '^[+-]{3}' || true)"
-          lines="${lines:-0}"
-          if [[ "${lines}" -lt 400 ]]; then
-            echo "small=true" >> "$GITHUB_OUTPUT"
-            echo "small diff (${lines} changed lines); mutants (1/4) runs all of it (--shard 0/1)"
-          else
-            echo "small=false" >> "$GITHUB_OUTPUT"
-            echo "large diff (${lines} changed lines); four shards"
-          fi
-      - uses: dtolnay/rust-toolchain@a5f673d0ba8626c3977bb416a1612774bc82181b # rustc 1.95.0
-        if: steps.diff.outputs.rust == 'true' && (steps.diff.outputs.small != 'true' || matrix.shard == 0)
-      - uses: taiki-e/install-action@9114bf4d891761788c546334fd37538eae1bf8b3 # v2.87.16
-        if: steps.diff.outputs.rust == 'true' && (steps.diff.outputs.small != 'true' || matrix.shard == 0)
-        with:
-          tool: cargo-mutants@27.1.0
-      - name: in-diff shard
-        id: run
-        if: steps.diff.outputs.rust == 'true' && (steps.diff.outputs.small != 'true' || matrix.shard == 0)
-        run: |
-          set -euo pipefail
-          if [[ "${{ steps.diff.outputs.small }}" == "true" ]]; then
-            shard_arg="0/1"
-          else
-            shard_arg="${{ matrix.shard }}/4"
-          fi
-          set +e
-          timeout --signal=TERM --kill-after=60s 30m \
-            cargo mutants --workspace --in-diff git.diff \
-              --shard "${shard_arg}" --sharding slice -j 2 \
-            > mutants-shard.log 2>&1
-          ec=$?
-          set -e
-          missed="$(grep -c '^MISSED' mutants-shard.log || true)"
-          echo "missed=${missed}" >> "$GITHUB_OUTPUT"
-          cat mutants-shard.log
-          if [[ "$ec" -eq 124 ]]; then
-            if [[ "$missed" -gt 0 ]]; then
-              echo "::warning::mutants shard hit 30m with ${missed} MISSED already in the log"
-              echo "result=timeout-missed" >> "$GITHUB_OUTPUT"
-              exit 1
-            fi
-            echo "::warning::mutants shard hit the 30m limit with no MISSED in the partial log; counting as success"
-            echo "result=timeout" >> "$GITHUB_OUTPUT"
-            exit 0
-          fi
-          echo "result=finished" >> "$GITHUB_OUTPUT"
-          exit "$ec"
-      - name: summary
-        if: always()
-        run: |
-          {
-            echo "### mutants ${{ matrix.n }}/4"
-            echo "Required. Outcome: ${{ steps.run.outcome }}. Result: ${{ steps.run.outputs.result }}. Missed: ${{ steps.run.outputs.missed }}."
-            echo "A 30m kill with no MISSED warns and passes. MISSED before that kill fails. A finished non-zero exit fails."
-          } >> "$GITHUB_STEP_SUMMARY"
-      - name: fail unless success or clean timeout
-        if: always() && steps.diff.outputs.rust == 'true'
-        run: |
-          set -euo pipefail
-          result="${{ steps.run.outputs.result }}"
-          outcome="${{ steps.run.outcome }}"
-          if [[ "${{ steps.diff.outputs.small }}" == "true" && "${{ matrix.shard }}" != "0" ]]; then
-            echo "small diff; mutants (${{ matrix.n }}/4) does not run cargo mutants"
-            exit 0
-          fi
-          missed=0
-          if [[ -f mutants-shard.log ]]; then
-            missed="$(grep -c '^MISSED' mutants-shard.log || true)"
-          fi
-          if [[ "$result" == "timeout-missed" || ( "$missed" -gt 0 && ( "$result" == "timeout" || "$outcome" == "cancelled" ) ) ]]; then
-            echo "::warning::mutants shard hit the time limit with ${missed} MISSED already in the log"
-            exit 1
-          fi
-          if [[ "$result" == "timeout" || "$outcome" == "cancelled" ]]; then
-            echo "::warning::mutants shard hit the time limit with no MISSED in the partial log; counting as success"
-            exit 0
-          fi
-          if [[ "$outcome" == "success" && "$result" == "finished" ]]; then
-            exit 0
-          fi
-          echo "mutants shard failed (outcome=${outcome} result=${result})"
-          exit 1
-      - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
-        if: always()
-        with:
-          name: mutants-pr-${{ matrix.n }}
-          path: mutants.out
-          if-no-files-found: ignore
-          retention-days: 5
-
   # Line-coverage 92% floor + CRAP --fail-above 30 (allowlist in .cargo-crap.toml).
   # Slow; waits for qc and test so a red check does not start
   # this job. Required. Job+step timeouts so a hung llvm-cov test binary

.github/workflows/mutants.yml

@@ -1,38 +1,74 @@
-# Weekly cargo-mutants. Not a required check.
-# PR in-diff shards are ci.yml job `mutants` (required). A 30m kill with
-# no MISSED warns and passes. MISSED before that kill fails the shard.
-# Must pass --workspace: default-members is rbitcoin-node only.
-# Owner: TESTING.md (Mutation testing). Snapshot lists: docs/mutants/.
+# Nightly workspace mutants. Not a required check.
+# New lines since the cursor run first, then a rotating backlog.
+# 90 minute budget, one workspace suite at a time. MISSED is an artifact,
+# not a failure. Owner: TESTING.md (Mutation testing).
 
 name: mutants
 on:
   schedule:
-    - cron: "0 3 * * 0"
+    - cron: "0 7 * * *"
   workflow_dispatch:
 
 permissions:
   contents: read
+  actions: read
 
 jobs:
-  mutants-weekly:
+  mutants-nightly:
     runs-on: ubuntu-latest
-    timeout-minutes: 360
-    strategy:
-      fail-fast: false
-      matrix:
-        shard: [0, 1, 2, 3, 4, 5, 6, 7]
+    timeout-minutes: 110
+    env:
+      CARGO_INCREMENTAL: "1"
+      CARGO_TERM_COLOR: never
+      RUSTFLAGS: ""
     steps:
       - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
+          fetch-depth: 0
           persist-credentials: false
-          submodules: false
       - uses: dtolnay/rust-toolchain@a5f673d0ba8626c3977bb416a1612774bc82181b # rustc 1.95.0
       - uses: taiki-e/install-action@9114bf4d891761788c546334fd37538eae1bf8b3 # v2.87.16
         with:
           tool: cargo-mutants@27.1.0
-      - run: cargo mutants --workspace --shard ${{ matrix.shard }}/8 -j 2
+      - name: previous cursor
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: |
+          set -euo pipefail
+          mkdir -p mutants-nightly
+          run_id="$(gh run list --repo "$GITHUB_REPOSITORY" --workflow mutants.yml --limit 20 \
+            --json databaseId,status,conclusion \
+            --jq '[.[] | select(.status=="completed")][0].databaseId')"
+          if [[ -n "${run_id}" && "${run_id}" != "null" ]]; then
+            gh run download "$run_id" --repo "$GITHUB_REPOSITORY" \
+              --name mutants-cursor --dir mutants-nightly || true
+          fi
+      - name: nightly queue
+        run: ./scripts/mutants-nightly.sh
+      - name: summary
+        if: always()
+        run: |
+          {
+            echo "### mutants nightly"
+            echo "Not required. Workspace tests. New code first, then the backlog."
+            if [[ -f mutants-nightly/missed.txt ]]; then
+              echo
+              echo '```'
+              cat mutants-nightly/missed.txt
+              echo '```'
+            fi
+          } >> "$GITHUB_STEP_SUMMARY"
+      - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        if: always()
+        with:
+          name: mutants-cursor
+          path: mutants-nightly/cursor.json
+          if-no-files-found: error
+          retention-days: 14
       - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
         if: always()
         with:
-          name: mutants-${{ matrix.shard }}
-          path: mutants.out
+          name: mutants-nightly
+          path: mutants-nightly/
+          if-no-files-found: ignore
+          retention-days: 14

AGENTS.md

@@ -126,9 +126,8 @@ Playbooks:
   [`.agents/skills/core-functional/SKILL.md`](.agents/skills/core-functional/SKILL.md).
 - Unreleased notes are a new file under `changelog.d/`. Do not edit
   `CHANGELOG.md` on a feature branch. Owner: [`docs/releases.md`](docs/releases.md).
-- Required CI jobs are `qc`, `test`, `windows`, `macos`, `coverage`, and
-  `mutants (1/4)` through `mutants (4/4)`. `qc` is fmt, ast-grep, deny,
-  the script self-tests, clippy, then nixos-module-eval. `--shard` is
-  0–3. A 30-minute kill with no `MISSED` warns and passes. `MISSED`
-  before that kill fails the shard. A finished non-zero `cargo mutants`
-  exit fails. Owner: [`TESTING.md`](TESTING.md).
+- Required CI jobs are `qc`, `test`, `windows`, `macos`, and `coverage`.
+  `qc` is fmt, ast-grep, deny, the script self-tests, clippy, then
+  nixos-module-eval. Mutants are a nightly workspace oracle
+  ([`mutants.yml`](.github/workflows/mutants.yml)), not a PR check.
+  Owner: [`TESTING.md`](TESTING.md).

CONTRIBUTING.md

@@ -244,8 +244,7 @@ Do not edit [`CHANGELOG.md`](./CHANGELOG.md) on a feature branch. The release
 cut folds those files in. Owner: [`docs/releases.md`](./docs/releases.md).
 
 Required **jobs** match [`.github/workflows/ci.yml`](./.github/workflows/ci.yml)
-(`qc`, `test`, `windows`, `macos`, `coverage`, `mutants (1/4)` …
-`mutants (4/4)`). `qc` is fmt, ast-grep, deny, the script self-tests,
+(`qc`, `test`, `windows`, `macos`, `coverage`). `qc` is fmt, ast-grep, deny, the script self-tests,
 clippy, then nixos-module-eval, in that order on one runner.
 Label **`nixos-module-runtime`** to run the NixOS module qemu test (not
 eval). That job is not required.

TESTING.md

@@ -254,7 +254,7 @@ matches scalar in default tests). Owner: [`docs/quality.md`](./docs/quality.md).
 | **cargo-crap** | After LCOV, `./scripts/coverage.sh` calls `./scripts/coverage-crap.sh` (skip if `cargo-crap` missing). `--fail-above --threshold 30`; `.cargo-crap.toml` allowlists today's production CRAP>30 functions (remove a name when it scores ≤30). Dry-run: `CRAP_DRY_RUN=1 ./scripts/coverage-crap.sh`. Self-test: `./scripts/coverage-crap.test.sh` | Rides required `coverage`. No `--fail-regression` (llvm-cov coverage % jitters per function) |
 | **coverage ignore / badge** | `./scripts/coverage.test.sh` (filename ignore, Tier A IBD not skipped, 92% floor, Shields JSON). Publish dry-run: `BADGE_DRY_RUN=1 ./scripts/publish-coverage-badge.sh` | `test` job self-test; `coverage` job writes `coverage/badge.json` and, on green `master`, pushes `badges/coverage.json` |
 | **Miri** | `./scripts/miri.sh` → `cargo +nightly miri test -p rbitcoin-primitives`. Dry-run: `MIRI_DRY_RUN=1 ./scripts/miri.sh`. Self-test: `./scripts/miri.test.sh` | Nightly `miri.yml` (not required). Never `--workspace` |
-| **cargo-mutants** | Must pass `--workspace` (`Cargo.toml` `default-members` is `rbitcoin-node` only; without `--workspace` the list is ~7 mutants). PR: `git diff origin/<base>.. --unified=0 -- crates Cargo.toml Cargo.lock .github/workflows/ci.yml .github/workflows/mutants.yml > git.diff` then `cargo mutants --workspace --in-diff git.diff --shard k/4 --sharding slice -j 2` (k is 0–3). Under 400 changed lines the PR job uses `--shard 0/1` once, on check `mutants (1/4)`. Weekly: `cargo mutants --workspace --shard N/8 -j 2`. Snapshot of missed/timeouts: [`docs/mutants/`](docs/mutants/). | PR: `ci.yml` `mutants (1/4)`–`mutants (4/4)` (`--shard` 0–3), required, needs `qc` and `test`. A finished non-zero exit fails. A 30-minute kill with no `MISSED` in the log warns and passes. `MISSED` already in that log fails the shard. No rust or workflow diff: every shard exits 0. Weekly `mutants.yml` Sunday 03:00 UTC, 8 shards, 360min cap (not required). `workflow_dispatch`. |
+| **cargo-mutants** | Nightly, not a PR check. `./scripts/mutants-nightly.sh` lists `--workspace` mutants, runs new diff lines first, then walks a cursor through the rest. Each mutant uses `--test-workspace=true` so a higher journey counts, `--baseline=skip`, `-j 1`, 20 minute mutant timeout, 90 minute budget. Cursor artifact `mutants-cursor`. `MISSED` is uploaded and does not fail the run. Snapshot lists in [`docs/mutants/`](docs/mutants/) are host history, not the gate. | `mutants.yml` daily `0 7 * * *` and `workflow_dispatch`. Job timeout 110 minutes. |
 
 Artifact silos above are unchanged: ast-grep / Miri dry-run / crap dry-run do
 not write `target/`. `mutants.out/` is gitignored.
@@ -530,6 +530,14 @@ python3 scripts/core-functional/check_inventory.py
 
 ## What a mutant kill looks like
 
-The PR gate is the four `mutants` shards in [`ci.yml`](.github/workflows/ci.yml), described in the table above. Do not run a different `cargo mutants` command and treat that as the gate.
-
-A new production behavior needs a test that fails when that behavior is removed or inverted. Assert the observable result (the error variant, the bytes, the height), not the test name. Concurrency, ordering, and cancellation are not what `cargo-mutants` models; those need a direct test. A diff that does not touch `crates/`, the Cargo manifests, or the mutants workflows does not run mutants.
+Mutants are a nightly oracle, not a pull-request check. The command is
+`./scripts/mutants-nightly.sh` ([`mutants.yml`](.github/workflows/mutants.yml)).
+Do not treat a package-only `cargo mutants` run as the gate: a journey
+outside the mutated crate must be able to catch the mutant.
+
+A new production behavior still needs a test that fails when that behavior
+is removed or inverted. Put that assert on a catalog journey, or on a
+store-less unit when the journey cannot see it. `MISSED` in the nightly
+artifact means no workspace test cared. Extend the journey. A same-crate
+twin that only existed to satisfy a package-local mutant is a deletion
+candidate once a workspace run shows the journey catching it.

changelog.d/mutants-nightly.md

@@ -0,0 +1,5 @@
+Changed
+
+- Mutants are a nightly workspace run, not a pull-request check. New
+  code is first in the queue, then a cursor walks older mutants. A
+  `MISSED` line is an artifact. It does not fail the night or the PR.

docs/releases.md

@@ -138,7 +138,8 @@ show as `core-functional / core-functional` and so on.
 
 | Check | Who |
 |-------|-----|
-| `qc` `test` `windows` `macos` `coverage` `mutants (1/4)`–`(4/4)` | Every PR (`ci.yml`). `qc` runs fmt, ast-grep, deny, script self-tests, clippy, then nixos-module-eval. A 30-minute mutants kill with no `MISSED` warns and passes. `MISSED` before that kill fails. A finished non-zero exit fails. |
+| `qc` `test` `windows` `macos` `coverage` | Every PR (`ci.yml`). `qc` runs fmt, ast-grep, deny, script self-tests, clippy, then nixos-module-eval. |
+| `mutants` | Nightly `0 7 * * *` and `workflow_dispatch` (`mutants.yml`). Workspace tests, new code first, then a backlog cursor. 90 minute budget. Not required. `MISSED` is an artifact. |
 | `core-functional` | Nightly, `workflow_dispatch`, label **`core-functional`**; via `release-gate.yml` on label **`release`** **or** ship version |
 | `overlay-functional` | Nightly (`42 6`), `workflow_dispatch`, label **`overlay-functional`**; via `release-gate.yml` on label **`release`** **or** ship version |
 | `warnet-example` | Label **`warnet`**, `workflow_dispatch`; via `release-gate.yml` on label **`release`** **or** ship version. Two-tank Docker lab. |