test(architecture): tighten eval graders the #7437 ready review found lenient #7438

Refs: #7437, #6551

The graders below were frozen for the measured runs in #7437, so the PR leaves them unchanged. The ready-time code review then found ways each one passes when it should fail. Every input quoted below was run through plugins/evals/skills/validate/scripts/validate-cases.py's js_regex and passes the current grader.

Effect on #7437's verdicts: the paired runs used only md, mf and my. Every run of those scored 1.00 and wrote the expected files, so finding 7 (md) and finding 8 (mf) are the only ones that touch a measured case. The other cases ship unmeasured against these graders.

Findings

Shell-write and interpreter bypasses

1. Fixed in c7acfae. mc-two-deployables-stops no-components-shell-write misses a quoted target. [^"|;&]* cannot cross the \" that compact JSON writes for a quoted path. These all pass the grader:

   • printf x > "docs/architecture/components.md"
   • cp /tmp/v.md "docs/architecture/components.md"
   • perl -pi -e s/a/b/ docs/architecture/components.md
   • python3 -c "open('docs/architecture/components.md','w').write('x')"

   Fix: use \S* after the operator, add perl\s+-\S*i, python3?\s+-c and node\s+-e, and add a quoted-path must-fail sample.

2. Fixed in c7acfae. ml-check-writes-nothing no-shell-write has the same quoting gap, plus three more misses: >|, perl in-place edits, and interpreter writes. These all pass:

   • printf x > "docs/architecture/landscape.md"
   • echo x >| docs/architecture/landscape.md
   • perl -pi -e s/a/b/ docs/architecture/landscape.md
   • python3 -c "open('docs/architecture/landscape.md','w').write('x')"

   Fix: the same as finding 1.

3. Fixed in c7acfae. rd-no-convention-writes-nothing no-shell-write misses interpreter writes and extensions outside its list. These pass:

   • python3 -c "open('docs/adr/0001-postgres.md','w').write('x')"
   • printf x > docs/adr/0001-postgres.markdown

   Fix: add the interpreter verbs and markdown|yaml|yml.

Negation and wording gaps

4. Fixed in c7acfae. mp-unresolved-not-matched reply-names-unresolved: alternative 1 lacks the n't lookbehind, and neither alternative sees a negator further back. These pass:

   • "Nothing is unresolved: ../Shared/Shared.csproj was matched to legacy/Shared/Shared.csproj."
   • "The reference isn't unresolved; ../Shared/Shared.csproj points at legacy/Shared."

   Fix: add a not_contains grader on legacy/Shared/Shared\.csproj in last_message.

5. no-findings-deep-* states-no-candidates: up to three arbitrary words may sit between "no" and "candidates". These pass:

   • "No surprise there are candidates here. Candidate 1: merge rates and rounding."
   • "There are no fewer than two candidates: the parser and the formatter."
   • "No strong candidates, but two weak candidates: Candidate 1 (Speculative) merge slug.js."

   Fix: allow only adjective fillers, or add a not_contains grader on Candidate \d. This also corrects the feat(architecture): concise skill text measured by evals, plus outcome cases and probes #7437 changelog line, which overclaims here.

6. deep-module-not-over-flagged says-already-deep: the guard sees only a negator immediately before "already", and the just|only|merely|simply move alternative has no guard at all. These pass:

   • "The pricing engine is not yet already deep; split it."
   • "It is hardly already deep; split it."
   • "Splitting TaxTable out would not just move code, it deepens the engine. Candidate: split it."

7. md-prisma-sql-mismatch reply-reports-mismatch:

   • It passes "Expected one mismatch, but Prisma and SQL agree, so none was recorded."
   • It is also too narrow: it fails "Mismatch count: 1".

8. mf-interface-hop-unresolved reply-reports-di-hop: it passes "Expected one interface hop, but the trace bound IOrderRepository to OrderRepository."

9. rd-no-convention-writes-nothing offers-numbered-shape: a template's Date: YYYY-MM-DD line satisfies it even when no numbering scheme is offered. It passes "Option A: docs/decisions/ with sections Title, Date: YYYY-MM-DD, Context, Decision, Consequences."

   Fix: anchor the token to a filename, YYYY-MM-DD-[\w-]+\.md.

Missing-file passes

10. not_contains file graders pass when the target file is missing: gateway-not-container, no-bound-implementation, not-matched-by-name, no-derived-people, and the no-secret-in-* file graders.

    Fix: pair each with a file_exists grader, or fold the negative into the positive pattern with a lookahead, as record-unchanged does.

Done when

• Each grader above rejects its quoted inputs, with each input added as a must-fail sample.
• /evals:validate reports no FAIL.
• md and mf are rescored or re-run under the fixed graders, on both models and both arms.

🤖 Generated with Claude Code