test(autonomy): grade the security-binding golden suite — 109 fixtures, zero orphans

[kyle-sexton]

Summary

Fix-direction (a) from the issue, per the #634 graded-fixture idiom: every fixture under plugins/autonomy/skills/setup/evals/fixtures/security-binding/ is now graded.

• One table-driven runner (check-security-binding.fixtures.test.mjs) + one co-located expectations manifest: each entry pins the checker invocation (--probe-evidence-root at the fixtures dir; per-fixture --egress-hosts/--evidence where needed) and the expected outcome — exit code plus defect-naming stderr substrings. 109 fixtures: 14 pass-expected (12 valid bindings + 2 evidence-input pairings), 95 reject-expected. Zero quarantined — no name-vs-behavior mismatches surfaced.
• Self-policing both directions: a new top-level fixture without a manifest (or quarantine) entry fails; a manifest ref whose file vanished fails; the 67 probe-transcripts/ suite inputs are enumerated and reconciled against disk both ways.
• Baseline drained: all 178 security-binding lines leave scripts/orphaned-fixtures-baseline.txt; the orphaned-fixture gate (feat(ci): orphaned-fixture + CHANGELOG-parity gates (#663) #681) passes with the set consumed, exactly as its stale-guard demands.
• Thin .test.sh wrapper joins plugins/** CI test discovery. Suite: 394/394 checks pass. validate-plugin-contracts.mjs, orphaned-fixtures --check, changelog-parity --check/--check-bump all green. Autonomy plugin bumped 0.7.3 with CHANGELOG entry.

Note for reviewers: the convention-level decision about the golden-fixture idiom repo-wide stays with #664 (needs-human); this PR instantiates the already-precedented #634 shape for the one suite #662 names, which the issue's own fix-direction (a) authorizes.

Related

• feat(ci): orphaned-fixture + CHANGELOG-parity gates (#663) #681 (the orphaned-fixture gate whose baseline this drains)
• conventions: standardize the golden-fixture idiom on the graded shape (#634 pattern) — fixtures carry asserted expected outcomes #664 (repo-wide golden-fixture convention decision — untouched)

Closes #662

🤖 Generated with Claude Code

[claude]

Claude finished @kyle-sexton's task in 3m 14s —— View job

---

PR Review

• Gather PR context
• Review changed files
• Analyze test manifest structure and correctness
• Check orphaned-fixture gate alignment
• Check CHANGELOG and version bump
• Post final review

---

Overall this is well-executed. The runner logic is correct, the self-policing is bidirectional, and the orphaned-fixture baseline drain is clean. A couple of minor observations worth noting.

---

Correctness

Runner control flow is correct. The three-phase structure in check-security-binding.fixtures.test.mjs is sound:

1. Self-policing: top-level fixtures vs. manifest (lines 44–55) — each fixture must be in fixtures or quarantined.
2. Probe transcript reconciliation (lines 61–74) — bidirectional: on-disk vs. listed.
3. Reverse drift (lines 79–84) — manifest entries whose referenced files have been deleted.
4. Grading loop (lines 87–113) — exit code assertion first, then substring assertions against the correct stream (stdout when exit 0, stderr when exit 1).

The early-continue pattern in the grading loop is the right call: a launch failure or exit code mismatch skips substring assertions without double-counting.

Evidence-pairing logic is correct. evidence-contrary.json and evidence-stale.json appear in fixtures keyed by their own filename, satisfying the self-policing check; the grading entry then redirects entry.binding → "promoted-c2.json" and passes --evidence evidence-contrary.json (or evidence-stale.json). The reverse-drift check verifies that both the redirected binding file and the evidence file are present on disk — this is correct since both are top-level fixtures.

Stream routing is safe. The checker's contract (findings → stderr, verdicts/evaluation lines → stdout) is documented in the checker itself and the test comment at line 104. The entry.exit === 0 ? result.stdout : result.stderr selector correctly follows that contract.

spawnSync invocation is safe. process.execPath avoids shell lookup; no user-supplied strings are concatenated into a shell command; { encoding: "utf8" } is correct.

---

Minor observations (non-blocking)

No maxBuffer on spawnSync. The default is 1 MB. For 109 fixtures each producing a few lines of output this is not a real risk, but the next maintainer adding a verbose checker might not notice the implicit limit. Passing maxBuffer: 10 * 1024 * 1024 would make the ceiling explicit without any functional change.

Fix this →

No timeout on spawnSync. If check-security-binding.mjs ever hangs (e.g., awaiting I/O it never gets), the test suite hangs indefinitely with no diagnostic. Adding timeout: 30_000 would bound the blast radius and emit a clear ETIMEDOUT error via result.error.

Fix this →

---

Shell wrapper

check-security-binding.fixtures.test.sh matches the repo's *.test.sh convention exactly — set -uo pipefail, SCRIPT_DIR via BASH_SOURCE[0], SKIP idiom for missing runtime, exec hand-off. Nothing to flag.

---

Manifest

The _comment sentinel-key approach is fine — the runner only reads fixtures, quarantined, and probe_transcripts by name, so the comment key is silently ignored. 109 entries (14 exit-0, 95 exit-1, 0 quarantined) against 109 on-disk JSON files — counts match. 67 probe transcripts match what's on disk.

---

Orphaned-fixture gate

All 178 security-binding lines removed from scripts/orphaned-fixtures-baseline.txt. The remaining entries (otlp-demo/, knowledge/.../variation-matrix-backlog.json) are unrelated to this PR. The stale-guard requirement (every baseline line must shadow an actual orphan) is correctly satisfied by the deletion since the fixtures are now graded.

---

Version bump and CHANGELOG

0.7.2 → 0.7.3 is the right semver tier for test-only additions (no public contract change). CHANGELOG entry under [0.7.3] is accurate and cites the issue number.

---

Verdict: approve pending the two optional hardening items above. The self-policing guarantees are sound — this is exactly the kind of graded-fixture harness that prevents future regressions silently.

[kyle-sexton]

Both hardening items applied: spawnSync now passes maxBuffer 10 MiB + timeout 30s (suite re-run green, 394/394).

[claude]

Claude finished @kyle-sexton's task in 32s —— View job

---

PR Review — Follow-up

• Gather PR context
• Verify hardening fixes applied correctly
• Final assessment

---

Both hardening items from the initial review are correctly applied in check-security-binding.fixtures.test.mjs lines 93–97:

const result = spawnSync(process.execPath, args, {
  encoding: "utf8",
  maxBuffer: 10 * 1024 * 1024,
  timeout: 30_000,
});

• maxBuffer: 10 * 1024 * 1024 — explicit 10 MiB ceiling, preventing silent truncation if a future verbose checker run produces large output.
• timeout: 30_000 — 30-second bound; a hung checker now surfaces ETIMEDOUT via result.error rather than hanging the suite indefinitely. The existing result.error branch at line 98 already handles this path correctly.

Both values are well-chosen: 10 MiB is generous for fixture-scale output, and 30s is a clear upper bound for any synchronous checker invocation without being unreasonably tight.

This PR is ready to merge. The self-policing guarantees, bidirectional reconciliation, stream routing, and orphaned-fixture baseline drain are all sound — and now hardened against the two edge cases flagged in the first pass.