test: close affected-tests blind spots for seven unmapped scripts across five plugins

[kyle-sexton]

No linked issue

Summary

A repo-wide /improvement:find scan surfaced seven executable scripts across five plugins that mapped to zero test suites under scripts/affected-tests.sh. The repo contract (AGENTS.md) treats a zero-suite mapping as an error, and this is the same gap class that let the kindle-dedrm firewall truthiness defect (#3396) ship undetected: scripts whose only "coverage" was accidental substring selection of unrelated suites. Writing the tests also forced out two latent defects in the instruction-placement eval harness, fixed here.

Fix

One sibling test suite per unmapped script, plus the two defect fixes:

• wizard 0.2.4: new template.test.sh (144 assertions) covering the wizard library's fail-closed prompt gates, _drain_tty paste bypass, write_env quoting/atomicity/permissions, open_url https-only dispatch, gh secret/variable helpers (secret never on argv), stage framing, and the EXIT trap.
• songwriting 1.4.15: new datamuse.test.sh (93 assertions), fully offline via a curl PATH-shim stub that fails hard if it does not shadow real curl. Covers the mode-to-relation table, TSV contract, family merge/dedupe/re-sort, LIMIT handling, and transport failures.
• prototype 0.9.6: sibling suites for the canonical detect-ecosystems.sh and both skill wrappers (91 assertions): marker detection, glob-order pinning, near-miss plus non-marker manifests (so widening the marker list fails), symlink/anchoring/CLAUDE_PROJECT_DIR semantics, and wrapper delegation contracts. The scripts themselves are unchanged.
• kindle-dedrm 0.7.9: new sync-prep.test.sh (73 assertions) proving the script is print-only: PATH shims shadow rm/pwsh/netsh with recording no-ops; "printed the command, ran nothing, deleted nothing" is asserted over a call log plus tree snapshot; the dry-run/live guard is tested in both directions (the firewall-bug class); the printed firewall rule name is checked against firewall.ps1's actual $RuleName.
• instruction-placement 0.11.11: new adherence-experiment.test.sh (49 cases via a stub CLI) plus two fixes the tests forced out: the underscore scoring criterion was a constant (it grepped the whole edited file, which always contains the seeded _unitPrice, so the published underscore/both columns measured nothing; scoring now scans only the InvoiceTotal body with an under-crediting fallback), and the --filler flag was undocumented. adherence-results.md carries a correction note; the experiment's conclusion is unchanged (it rests on the correctly-scored sealed column).

Verification

• Every previously-unmapped script now selects its sibling suite under scripts/affected-tests.sh --explain (each was UNMAPPED, exit 1, before), and --run over the full diff passes (7 suites).
• shellcheck (repo rcfile), shfmt -d, and scripts/check-shell-portability.sh clean on all new suites; changelog parity (--check, --check-bump, --check-preserved) green; fixture-git-isolation, silent-skips, discriminating-skips, orphaned-fixtures, and purged-em-dashes gates green.
• Each plugin's work was reviewed by an independent fresh-context verifier: diff review, gate re-runs, adversarial assertion audit, and independent mutation testing with byte-identity restoration proofs (implementer + verifier mutation batteries: wizard 33+9, songwriting 23+10, prototype 19+10, kindle-dedrm 12+9). All verifier findings, including one MEDIUM (the family-dedupe assertion could not distinguish unique from unique_by(.word)) and several LOW precision gaps, were closed with mutation-kill confirmation.

Related

• fix(kindle-dedrm): compare firewall rule state against 'True', not truthiness #3396 (the shipped defect exemplifying this coverage-gap class)

🤖 Generated with Claude Code

https://claude.ai/code/session_01XGLX1xYgy27JiRqLjoiH8T

---

Generated by Claude Code

[claude]

Claude encountered an error after 29s —— View job

---

Claude is working on this

• Gather context (PR diff, files changed)
• Run /review:security-review per custom instructions
• Post findings

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 76beda05b6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[claude]

Claude encountered an error after 0s —— View job

---

I'll analyze this and get back to you.

[claude]

Claude finished @kyle-sexton's task in 1m 30s —— View job

---

Claude is working on this

• Gather context (PR diff, files changed)
• Run /review:security-review per custom instructions
• Post findings

---

Security review

No security issues found. Skip gate applies: this change has no security-relevant surface.

What changed (verified via git diff origin/main...HEAD): 19 files, entirely test infrastructure — five new/expanded *.test.sh suites (wizard, songwriting, prototype ×3, kindle-dedrm), one modified eval harness (plugins/instruction-placement/evals/adherence-experiment.sh), plugin.json version bumps, and CHANGELOG/results-doc updates. No .github/workflows/** files are touched.

Specifically checked:

• sync-prep.test.sh (kindle-dedrm): uses mktemp -d sandboxes and PATH-shimmed rm/pwsh/powershell.exe/netsh/icacls recorders that perform nothing — asserts the script under test never runs a real destructive/elevated command. No real firewall or filesystem state outside the sandbox is reachable.
• datamuse.test.sh (songwriting): curl is shadowed via a PATH stub with an explicit self-check (command -v curl must resolve to the stub or the suite fails), so no live network call to api.datamuse.com is possible.
• template.test.sh (wizard): the new assertions actively verify good security properties in the library under test — secrets are sent over stdin and never appear on argv or in printed output (set_secret/ask_secret), the env file is tightened to 0600, and open_url refuses non-https URLs. No new exposure introduced.
• detect-ecosystems.test.sh (prototype, ×3): pure marker-detection/glob-order fixtures; no script changes, no network or credential surface.
• adherence-experiment.sh scoring fix (instruction-placement): tightens two regexes used to grade locally-generated eval transcripts (anchoring the class-name match, requiring a field-declaration terminator for the underscore check). This is a measurement-correctness fix in an internal eval harness, not a trust-boundary or injection-relevant change — no untrusted external input reaches these regexes.

No secrets, injection surfaces, authz logic, or GitHub Actions permissions/triggers are touched by this diff.

(Note: the two Codex review comments on this PR concern eval-scoring precision, not security — out of scope for this lane; /review:code-review is the right lane for those.)

[github-actions]

Last security-reviewed head: f2116e79374e25522f2460909b253c9210a1c6a3. On the next push, the relevance gate compares only the commits since this SHA; delete this comment to force a full re-review.

[claude]

Claude finished @kyle-sexton's task in 7m 43s —— View job

---

Claude finished the code review

• Gather context (PR diff, files changed)
• Run /review:code-review per custom instructions
• Post findings

Note: the review:code-review skill failed to execute in this environment (returned an infrastructure error on every invocation attempt), so this review was conducted manually following the same lane's scope: high-signal correctness/maintainability findings only, security out of scope (separate lane).

Summary

Reviewed the full diff (git diff origin/main...HEAD, 19 files, +2818/-11) via a dedicated static-review pass plus direct verification of the two logic changes in adherence-experiment.sh. No new correctness or maintainability findings.

On the two open Codex review comments

Codex's two P2 comments (unanchored InvoiceTotal class match, and the underscore check not requiring a real field declarator) describe defects that are already fixed in this exact commit (76beda0, unchanged by the later merge commit). I verified both directly against the current code:

• Class-name anchoring (adherence-experiment.sh#L254): the pattern is InvoiceTotal([^A-Za-z0-9_]|$). Tested against sealed class InvoiceTotals { ... } — does not match, correctly rejected.
• Field-declarator constraint (adherence-experiment.sh#L303): the pattern requires the match to end in ; or =[^>]. Tested against private decimal _total { get; set; }, private decimal _GetTotal() => total;, and private decimal _Total => x; — none match (auto-property, expression-bodied method, and expression-bodied property all correctly rejected); private readonly decimal _unitPrice; still matches as expected.

Both appear to be exactly the two defects the PR body says were fixed ("the underscore scoring criterion was a constant... scoring now scans only the InvoiceTotal body"). No action needed — flagging here so the thread doesn't read as unresolved.

Scope covered

• adherence-experiment.sh's score_trial() awk body-extraction logic (decoy-class, primary-constructor, unbalanced-brace edge cases) — sound.
• sync-prep.test.sh's PATH-shim stubs for rm/pwsh/netsh — command-scoped PATH= prefix, can't leak into the suite's own cleanup trap; sync-prep.sh itself never invokes those binaries, so the "touched nothing" assertions are real regression guards, not tautologies.
• datamuse.test.sh's curl stub — self-verifies with command -v curl before any assertion runs, so it can't silently fall through to the network.
• The three detect-ecosystems.test.sh copies (top-level + two skill wrappers) — near-identical by design, each wrapper asserts against its sibling and byte-diffs to catch drift; not flagging the duplication since it's a documented, low-drift-risk pattern.
• wizard/skills/generate/template.test.sh (764 lines, read in full) — the fd-3 seam rewrite, fail-closed guards, write_env quoting/atomicity, open_url https-only dispatch, gh secret/variable helpers (stdin-only, never argv), and the EXIT trap under set -e all check out; the shipped-file contract assertions (single exec 3</dev/tty, single STAGES marker, TOTAL_STAGES consistency) correctly pin the seam so it can't silently drift onto executing a different code path.

[github-actions]

Claude has reviewed this PR 1 time. The lane skips further automatic reviews after 5; deleting this comment resets the count.