Repository navigation
Gremlins mutation testing tracker #128
Description
Gremlins mutation testing tracker
Auto-updated weekly by cplieger/ci/.github/workflows/weekly-gremlins.yaml.
Last update: 2026-10-26 16:08
This week: 93.3% efficacy (±0.1% across 3 runs), 99.2% mutant coverage. 18 confirmed live mutants.
Trend: → +0.1% from 12-week mean (93.2%).
⚠️ Verdicts disagree across attempts. 1 mutant came back LIVED in some of the 3 runs and KILLED in others, so one verdict per disagreement is false and the mean is partly noise. Either a flaky test, or cross-worker interference in the runner — gremlins workers share one/tmp, so a test bound to a fixed absolute path can delete a marker another worker is polling for (the weekly runner probes for that and drops to--workers 1when it reproduces). Equivalent mutants cannot all be killed, so a 100% attempt next to a survivor in another attempt is a measurement failure, not a perfect score.
Rolling 12-week history
| Run (UTC) | Mean efficacy | Stddev | Mutant coverage | Live mutants | Δ efficacy | Mode |
|---|---|---|---|---|---|---|
| 2026-10-26 16:08 | 93.3% | ±0.1% | 99.2% | 18 | -0.1% | integration |
| 2026-10-19 14:49 | 93.4% | ±0.0% | 99.2% | 17 | +0.4% | integration |
| 2026-10-12 13:25 | 93.0% | ±0.0% | 98.8% | 17 | +0.3% | integration (switched from per-package; score jump expected) |
| 2026-10-05 15:01 | 92.7% | ±0.2% | 98.8% | 18 | +0.0% | |
| 2026-10-04 20:18 | 92.7% | ±0.2% | 98.8% | 18 | -3.2% | |
| 2026-10-03 03:16 | 95.9% | ±0.0% | 98.8% | 10 | +0.0% | |
| 2026-09-21 08:55 | 95.9% | ±0.0% | 98.8% | 10 | +0.1% | |
| 2026-09-14 08:58 | 95.8% | ±0.2% | 98.8% | 11 | -0.1% | |
| 2026-09-07 08:51 | 95.9% | ±0.0% | 98.8% | 10 | +0.1% | |
| 2026-08-31 07:56 | 95.8% | ±0.2% | 98.8% | 11 | +0.2% | |
| 2026-08-24 05:00 | 95.6% | ±0.2% | 98.8% | 11 | +0.3% | |
| 2026-08-21 15:35 | 95.3% | ±0.0% | 98.7% | 11 | +3.2% |
Current live mutants (18, this week)
Rare flake — LIVED in 1 of 3 runs (test usually catches it) — 1
httpx.go
- L957 — CONDITIONALS_BOUNDARY
Solid gap — LIVED in all 3 runs (known coverage hole) — 17
httpx.go
- L420 — CONDITIONALS_BOUNDARY
- L429 — CONDITIONALS_BOUNDARY
- L447 — CONDITIONALS_BOUNDARY
- L451 — CONDITIONALS_BOUNDARY
- L544 — CONDITIONALS_BOUNDARY
- L556 — CONDITIONALS_BOUNDARY
- L560 — CONDITIONALS_BOUNDARY
loop.go
- L235 — CONDITIONALS_BOUNDARY
- L296 — CONDITIONALS_BOUNDARY
- L364 — CONDITIONALS_BOUNDARY
- L499 — INVERT_NEGATIVES
- L499 — ARITHMETIC_BASE
- L517 — CONDITIONALS_BOUNDARY
- L543 — CONDITIONALS_BOUNDARY
roundtripper.go
- L159 — CONDITIONALS_BOUNDARY
- L324 — CONDITIONALS_BOUNDARY
- L324 — CONDITIONALS_BOUNDARY
How to read
- Mean efficacy: % of KILLED over KILLED + LIVED mutants, averaged across the N runs; TIMED OUT mutants count in neither
- Stddev: variance across runs — high stddev (>3%) signals flaky tests
- Mutant coverage: % of mutants reached by the test suite (test depth)
- Mode:
per-packageruns each mutant against its own package's tests;
integrationruns the whole module's suite with coverage credited across
packages, and is chosen each run when its measured cost fits a quarter of the
job's time cap. A row marked as switched starts a new baseline, so its score
jump is expected.
The "Current live mutants" section is bucketed by how many of the N runs the
mutant LIVED in (N = attempts that week, normally 3):
- Rare flake (1/N): a mutant your tests usually KILL but occasionally let
through. Most actionable — almost always means a flaky test that's not
reliable. Open the section to see the file:line; expect to fix the test, not
the production code. - Weakly flaky (k/N for 1<k<N): tests catch this mutant inconsistently.
Less ideal than rare-flake but still worth investigating. - Solid gap (N/N): every run lets this mutant through. Either there's no
test exercising the path, or the test asserts the wrong thing. Action: add a
test or strengthen an existing assertion.
The mutation-regression label is added when this week's mean efficacy drops
5% below the rolling 12-week mean of the weeks run in the same mode.
A
suite: either the attempts disagreed about which mutants survive (one verdict per
disagreement is false), or the score jumped to a flawless 100% from a week with
live mutants. Equivalent mutants are a permanent noise floor, so a reproducible
100% is not something a test suite can reach.
Free-form notes
Add anything below — won't be touched by the auto-updater.
Activity