Skip to content

ci: fix mutants job cycle and harden mutants-baseline-diff - #726

Closed
Hero-Gamer wants to merge 14 commits into
reardencode:masterfrom
Hero-Gamer:lint/mutants-baseline-diff
Closed

Hero-Gamer wants to merge 14 commits into
reardencode:masterfrom
Hero-Gamer:lint/mutants-baseline-diff

Conversation

@Hero-Gamer

@Hero-Gamer Hero-Gamer commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Update spec 2026-09-24 3.35PM ET

Aligned with #739 — Advisory Only ✅

  • mutants-baseline-diff = reports, never blocks → exits 0 on new survivors
  • Real errors (missing data/files) still fail → no silent misbehavior
  • Mutant scanning → nightly mutants.yml; PR defense → ast-grep (lint: weak asserts outside test files, and a waiver date token #687)
  • CI cycle fixed; baseline auto-initializes; no 25k-line file committed

Follow up from #688 item 1.3

What

  • Restore clean ci.yml from upstream/master to fix 9x Expected — Waiting for status to be reported.
  • Brings back required checks: qc, test, coverage, macos, windows, mutants (1/4), mutants (2/4), mutants (3/4), mutants (4/4).
  • Ensures trigger is:
    on:
      push:
        branches: [master, main, "v*.*.x"]
      pull_request:
    so fork PRs report status instead of staying in Waiting.
  • mutants job remains:
    • 4-way sharded matrix 0/4..3/4, name mutants (n/4)
    • needs: [qc, test] (overlaps coverage, does not wait for macos/windows)
    • small diffs (<400 lines): mutants (1/4) runs --shard 0/1, others exit 0
    • docs-only diffs exit 0
    • uploads mutants.out artifact
  • If branch contains mutants-baseline-diff job + scripts/mutants-diff.py + docs/mutants/survivors-baseline.list, this PR keeps them:
    • mutants-baseline-diff needs [mutants], if: github.event_name == 'pull_request'
    • permissions: contents: read, issues: write, pull-requests: write
    • uses actions/download-artifact@v4 with pattern: mutants-out-*
    • scripts/mutants-diff.py handles first-run empty baseline, ignores # comments, deduplicates shards, skips target/

Why

Previous 13 commits deleted qc and introduced needs: [mutants] self-cycle / invalid YAML, so no jobs started on lint/* branches. Every push showed Failure lint/mutants-baseline-diff with no logs and Expected — Waiting for fmt, clippy, test, etc.

Correct DAG is: qc -> test -> mutants -> mutants-baseline-diff -> coverage

Verification

grep -n "^\s*[a-z0-9_-]*:\s*$" .github/workflows/ci.yml | head -n 20
python3 -c "import yaml,pathlib; yaml.safe_load(pathlib.Path('.github/workflows/ci.yml').read_text()); print('YAML OK')"
cargo fmt --all -- --check
git diff upstream/master --stat
# should NOT contain docs/mutants/all-mutants.list (25233 lines)

Actions now: 11 successful, 3 in progress (coverage, mutants 4/4, codeql), 4 skipped (core-functional, nixos-module-runtime, overlay-functional, warnet-example - expected for ci.yml-only diff).

Notes

  • No production code change. No coverage impact.
  • Docs-only diffs exit 0, weekly sweep stays mutants.yml.
  • Owner: TESTING.md / docs/quality.md

52a260e master: 25233 total mutants. 1.3 gate:
- docs/mutants/survivors-baseline.list starts empty, weekly will fill
- scripts/mutants-diff.py merges 4 shards mutants.out and diffs vs baseline
- before reardencode#679 in-diff was continue-on-error no comment, after 1.3 it
  posts comment and fails on new survivors. Per reardencode#688, docs+YAML only.
…s, normalization

- download-artifact path: mutants-shards + merge-multiple: false
  prevents last-shard-wins false negative (L2)
- needs: [fmt, clippy, test, mutants] ensures shards finish before diff (L3)
- permissions: issues+PR write for comment bot (L5)
- script: is_file + target filter + resolve dedup + normalize ./crates
    + fail if 0 artifacts found
- addresses reardencode#724 scratch reuse lesson and reardencode#688 waiver spec
- single job, needs [fmt, clippy, test, mutants]
- path: mutants-shards, merge-multiple: false -> shards:4 not 1
- no ${body} backtick template, uses msg concat
- fixes line 421 Invalid workflow file
Comment thread .github/workflows/ci.yml Fixed
Comment thread .github/workflows/ci.yml Fixed
Comment thread .github/workflows/ci.yml Fixed
- Only the 'new survivors vs baseline' path → exit 0
- Actual error conditions (missing files/data) still fail as before
- ::error:: → ::notice::; clearly states non-blocking
- Aligns with nightly mutants model, no PR gating
@Hero-Gamer

Copy link
Copy Markdown
Contributor Author

Updated — aligned fully with #739:

  • mutants-diff.py: new survivors → exit 0 / advisory only — never blocks merge
  • Real errors (missing data/artifacts) still fail → no silent misbehavior
  • CI DAG fixed, baseline initializes automatically, shard merging intact
  • This infrastructure is exactly what nightly scanning needs: artifact collection → baseline comparison → transparent reporting

@rearden-grok

rearden-grok Bot commented Sep 25, 2026

Copy link
Copy Markdown
Contributor

Thanks for putting this together and for working through the CI edge cases. We are going to use the overnight mutants runs to surface survivors and drive improvements over time, rather than maintain a checked-in survivor baseline or an advisory diff job, so I am closing this PR for now.

@rearden-grok rearden-grok Bot closed this Sep 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants