Skip to content

test(analog): sar_adc mutation census, 76 -> 85 killed, 14 -> 5 survived, rest accepted (Closes #12096) - #12108

Merged
github-actions[bot] merged 1 commit into
masterfrom
lane/analog-adc-mutants-12096
Oct 24, 2026
Merged

github-actions[bot] merged 1 commit into
masterfrom
lane/analog-adc-mutants-12096

Conversation

@gHashTag

Copy link
Copy Markdown
Collaborator

Closes #12096. Part of epic #10486 (weak spots of the analog v0.1 release).

What changed

The mutation census of specs/analog/sar_adc.t27 (98 mutants) left 14 survivors and 8 hung mutants on master. This change appends three tests after the last original test (so no line number above moved), writes the survivors that no test can or must kill into docs/analog/mutation-accepted-sar_adc.txt, and records the census in docs/analog/mutation-sar_adc-2026-10-25.md.

Run Mutants Killed Survived Hung Unviable
Before (master, as recorded in #12096; not re-run here) 98 76 14 8 0
After (this change; a census and the gate, both read here) 98 85 5 8 0

The 5 survivors are equal-value ties (> to >= where the tie writes the value already held) and two clamp edges where the fall-through cast gives the same integer; each has its arithmetic in the accepted file. The 8 hung mutants are dropped loop steps. The gate with --accepted and --timeout 30 ran on the final tree (04:02-04:06 +07): exit 0, "13 not killed (5 survived, 8 hung), 13 accepted by the file, 0 accepted line(s) now killed. Gate passed." tri coverage --strict: gaps 0 coverage 130/130.

The tests

  • the ideal quantizer clamps below the first code and above the last (7 asserts);
  • fit_cp returns the first point of most matches and least margin, equal to a hand scan of the 41 Cp points, on every one of the 41 offsets of the fit grid (164 asserts);
  • a model one step past the end of a fit grid is not found, and one in the middle of an interval is found at its first point; the 31 inputs are built by bisection on the model itself (290 asserts).

tri test specs/analog/sar_adc.t27: 18 tests pass, 0 vacuous. Seal re-saved with the master-built t27c (tri seals changed; built_by ends in +19a89f1e3).

Not claimed

🤖 Generated with Claude Code

… -> 5 survived, rest accepted (Closes #12096)

Three tests appended after the last original test of specs/analog/sar_adc.t27 (15 -> 18 tests, 0 vacuous):
the ideal quantizer clamps at both ends (7 asserts), fit_cp equals a hand scan of the 41 Cp points on every one
of the 41 offsets of the fit grid (164), and models one step past the end of a fit grid, or in the middle of
an interval, are built by bisection on the model itself and found at the right point (290).

The 5 survivors left are equal-value ties and clamp edges, the 8 hung mutants are dropped loop steps; all 13
are in docs/analog/mutation-accepted-sar_adc.txt, and the gate with --accepted ends green.
Seal re-saved with the master-built t27c. The census and the arithmetic of each accepted line are in
docs/analog/mutation-sar_adc-2026-10-25.md.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-10-24 21:12:29 UTC

Summary

Status Count
Total Open PRs 47
PRs with Failing Checks 42
PRs with All Checks Green 5
READY 4
FAILING 42
PENDING 0
NO CHECKS YET 0

These columns do not partition: 4 + 42 + 0 + 0 = 46, and there are 47 open PRs. A PR is being counted twice or not at all.

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=55c0f1b1b32d != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

@tri-claw

tri-claw Bot commented Oct 24, 2026

Copy link
Copy Markdown

Summary by t27-bees

Checked by the t27 compiler

t27c parse, parse-complete and typecheck on each changed .t27 file at 5fbeed3. Nothing was generated, built or run. A pass means the file is well formed, not that it is right.

  • specs/analog/sar_adc.t27: parses and typechecks.

Read by the model

This PR adds three tests to specs/analog/sar_adc.t27 to kill 9 of 14 mutation survivors (76→85 killed), documents the 5 remaining equivalent survivors and 8 hung mutants in docs/analog/mutation-accepted-sar_adc.txt, and records the census in docs/analog/mutation-sar_adc-2026-10-25.md. Tests are appended to preserve line numbers for the accepted file. The seal is regenerated with the new spec hash and test count (15→18).

Worth a look

  • specs/analog/sar_adc.t27:860-900 (second test): The test reimplements fit_cp's selection logic (most hits → least viol → first index) to compute the expected pick, then asserts fit_cp returns that same pick. If both the implementation and this test share a logical error (e.g., tie-breaking when viol[p] == viol[pick]), the test passes despite a bug. The comment cites a "hand scan" but the test computes the expectation algorithmically.

  • specs/analog/sar_adc.t27:870-880 (second test): pick is initialized to FIT_NA (41) as a sentinel. The loop only updates pick when hits[p] == most. Since most is the maximum of hits, at least one index matches, but if the logic changed so most could exceed all hits[p], pick would stay 41 and viol[pick] would read out of bounds (arrays are [41]u32, valid indices 0–40).

  • specs/analog/sar_adc.t27:930-940 (third test): The call fit_grid(2, &vp, &vn) passes a magic number 2 with no comment explaining its meaning (e.g., pass count, tolerance index, mode). If the parameter's semantics change, the test's intent becomes opaque and it may silently verify the wrong behavior.

  • specs/analog/sar_adc.t27:950-970 (third test): The bisection uses 44 iterations on a ±80 mV bracket to find code edges, claiming "the model's gain is 2% off the ideal one." 44 iterations yields ~9e-12 V precision—far beyond what the 2 uV placement offset requires. If the gain error exceeds ~2% or the 80 mV bracket is too narrow, the initial assert(convert_sar(...) < c) / > c could fail, making the test flaky or incorrect.

  • docs/analog/mutation-accepted-sar_adc.txt: The file accepts 5 equivalent and 8 hung mutants by line number (e.g., 239, 240, 370, 387, 388, 214, 225, 349, 372, 389, 427, 429, 449). Any edit to the spec above the original end (line 833) shifts these lines, causing the mutation gate to fail. The PR appends tests to avoid this, but future changes above line 833 require regenerating the accepted file.


Read: 4 files, +252 -9, the whole diff. Head 5fbeed3. Model: openai-compatible/nvidia/nemotron-3-ultra-550b-a55b. Commands: @t27-bees review | summary | help | pause | resume. Free for every repository.

@github-actions
github-actions Bot merged commit 2c3b48e into master Oct 24, 2026
37 checks passed
gHashTag added a commit that referenced this pull request Oct 24, 2026
…n" -- the scan is written in the test (Refs #12117) (#12118)

The t27-bees review of PR #12108 pointed at the phrase: the fit_cp test does not compare against a scan
done by hand, it re-scores all 41 Cp points with match_count and miss_margins inside the test. Say that, and
say what it does and does not check (the choice rule, not those two functions). No number changes.

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
@gHashTag
gHashTag deleted the lane/analog-adc-mutants-12096 branch October 26, 2026 03:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

analog: pin the 14 surviving mutants of sar_adc.t27 (Refs #10486)

1 participant