Skip to content

fix(testing): make the date guard discriminating, and correct two records it exposed - #3167

Merged
kyle-sexton merged 1 commit into
mainfrom
claude/close-verifier-nits
Aug 23, 2026
Merged

fix(testing): make the date guard discriminating, and correct two records it exposed#3167
kyle-sexton merged 1 commit into
mainfrom
claude/close-verifier-nits

Conversation

@kyle-sexton

Copy link
Copy Markdown
Contributor

No linked issue.

Summary

Three items surfaced by adversarial verification passes on #3097 and #3094, each re-confirmed against origin/main before this branch was cut. One is a test that cannot fail, inside the skill built to find tests that cannot fail. The other two are records that misstate something, in documents whose job is to be accurate.

Nothing behavioral changes. All three plugins take patch bumps.

Fix

1 — a non-discriminating guard in testing:audit's own suite. cant-fail-scan.test.sh asserted:

assert_contains "date frontmatter is present" "$out" "date: 20"

That needle is a truncated prefix of a structured value, so it accepts anything beginning date: 20 — including the malformed date: 2026-08-21T13-36-00Z that #3097 just removed from ai-slop's emitter, where an identically-shaped guard had been holding the bug in place while passing green. testing:audit exists to detect exactly this, which is why it is worth fixing on principle as well as on merit.

Replaced with an anchored shape assertion, via a new assert_matches <name> <haystack> <ERE> helper added beside the file's three existing asserts so shape checks have somewhere to live other than a prefix match:

assert_matches "date frontmatter is ISO-8601 extended UTC (YYYY-MM-DDThh:mm:ssZ)" "$out" \
  '^date: [0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$'

Proven to discriminate, not asserted. Both assertion bodies run against three inputs, the last produced by actually invoking cant-fail-scan.sh --findings:

Input Old assertion New assertion
date: 2026-08-21T13-36-00Z (hyphenated time) PASS FAIL
date: 2026-08-21 (no time, no Z) PASS FAIL
date: 2026-08-23T04:37:40Z (real emitter output) PASS PASS

Sibling assertions were audited and deliberately left alone. date: 20 was the only needle in the file that is a truncated prefix of a structured value; every other one pins a whole-token literal a malformed value would not contain (type: review-findings, the full table header, | IMPORTANT | high |, left\|\|right, the refusal strings). No test-suite rewrite.

2 — a justification that does not hold, in overengineering's findings-artifact contract. Its frontmatter date field was justified as "Colon-free UTC, Windows-safe, lexically sortable". Windows-safety is a filename property — a colon is illegal in a Windows path component and irrelevant to a field inside a file.

The format is not changed and was never wrong. ISO-basic YYYYMMDDTHHMMSSZ is valid ISO-8601 and lexically sortable; this artifact has its own consumer (/overengineering:realign) and is explicitly not a detector-findings producer, so nothing obliges it to match ai-slop's extended form. The format specification at all three sites is byte-identical to main; only the rationale cell changed.

The claim is dropped rather than re-sited, because there is no filename to attach it to: this contract fixes "one stable filename per home, rewritten in place" and rejects a timestamped filename by name. The cell now records why colon-freedom does not apply here, so the claim does not quietly return.

3 — the one derived plugin with no reverse pointer to the upstream record. Measured: plugins/ai-slop/CHANGELOG.md cited docs/upstream/cursor-pstack.md zero times, against at least one citation in each of the eleven other derived plugins.

The natural home — ai-slop's #3031 entry — is published history this repo does not edit, so the pointer went to the plugin README instead, onto the sentence that already named Cursor's unslop skill and the overlap map. Flagged as a new placement: no plugin README cites docs/upstream/ today; all eleven existing citations are in changelogs. Defensible, but new, and easy to relocate if you would rather it went elsewhere.

Verification

Gate Result
Discrimination proof (item 1) old assertion accepts both malformed inputs; new one rejects both, accepts real emitter output
cant-fail-scan.test.sh All 69 checks passed
ai-slop detect.test.sh (regression) All 92 cases passed
check-changelog-parity.sh --check / --check-bump / --check-order / --check-preserved PASS — 83 changelogs; 3 changed, 36 headings preserved
check-changed-skills.sh origin/main PASS — 0 errors, 0 warnings
check-stale-base-overlap.sh --check origin/main up to date
shellcheck clean
markdownlint-cli2 on 5 changed files 0 issues
Published changelog entries all three files +N -0 — additive only

Versions verified against main before bumping rather than assumed: testing 0.7.3 → 0.7.4, ai-slop 0.3.3 → 0.3.4, overengineering 0.2.0 → 0.2.1. All patch — every change is test-only or documentation-only.

Related


Generated by Claude Code

…e, a missing reverse pointer

Three items found by adversarial verification of #3097 and #3094, each
re-confirmed against origin/main before this branch.

testing 0.7.3 -> 0.7.4 — cant-fail-scan.test.sh asserted `date: 20` for the
`--findings` frontmatter date. That is a truncated prefix of a structured
value: it passes for the emitter's real `2026-08-23T04:37:40Z` and equally for
`2026-08-21T13-36-00Z`, a hyphenated time that is ISO-8601 in neither profile —
the same guard shape that pinned ai-slop's emitter bug instead of catching it,
sitting inside the skill whose purpose is finding tests that cannot fail. Now
anchored to the full extended form with an explicit Z, via a new `assert_matches`
helper so shape assertions have somewhere to go other than a prefix check.
Confirmed discriminating: FAILS on both malformed shapes, PASSES on real emitter
output. The file's other assertions were re-read; each pins an exact literal a
malformed value would not contain, so nothing else was rewritten.

overengineering 0.2.0 -> 0.2.1 — findings-artifact.md justified its `date:`
field as "Colon-free UTC, Windows-safe, lexically sortable". Windows-safety is a
filename property and is meaningless for a field inside a file; this contract
fixes one stable filename per home and rejects a timestamped one, so there is no
filename to re-site the claim onto either. The rationale now says what is true
(compact, unambiguous about its zone, lexically sortable). The FORMAT is
unchanged — ISO-basic YYYYMMDDTHHMMSSZ is valid ISO-8601 and this artifact is
not a detector-findings adopter.

ai-slop 0.3.3 -> 0.3.4 — docs/upstream/cursor-pstack.md names ai-slop's catalog
and rewrite guide as where Cursor's `unslop` skill landed, but nothing under
plugins/ai-slop/ pointed back; it was the only derived plugin with no citation
of that file. The README's existing sentence naming the upstream skill now
carries the link. Added there rather than to the published entry that recorded
the port, which is history and is not edited.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AqwX2njhiLMWGrMg3QNt4g
@claude

claude Bot commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 44s —— View job


Security review

  • Gather context (PR diff, files changed)
  • Run /review:security-review per custom instructions
  • Post findings / summary

No security issues found.

This PR is test-only and documentation-only:

  • plugins/testing/skills/audit/scripts/cant-fail-scan.test.sh — adds an assert_matches shell helper that runs grep -qE against a fixed, in-file regex literal (never external/user input), and tightens a test assertion from a truncated-prefix substring check to an anchored ISO-8601 shape check. No injection surface, no untrusted data flow.
  • plugins/ai-slop/README.md, plugins/overengineering/context/findings-artifact.md — prose-only corrections (a rationale claim and a cross-reference link).
  • plugins/*/CHANGELOG.md, plugins/*/.claude-plugin/plugin.json — changelog entries and patch version bumps.

No GitHub Actions workflows, permissions, dependency pins, auth/authz logic, or data-handling code are touched by this PR. Skip-gate condition 2 (no security-relevant surface after reading the diff) applies — nothing further to report.

@github-actions

Copy link
Copy Markdown
Contributor

Last security-reviewed head: 527561e27e78cb39d41f42d74b1a6102f33193f9. On the next push, the relevance gate compares only the commits since this SHA; delete this comment to force a full re-review.

@claude

claude Bot commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 2m 17s —— View job


Claude's review

  • Gather context (PR diff, CLAUDE.md, code-review skill)
  • Review changed files for correctness/maintainability
  • Post inline findings (if any)
  • Final summary

No blocking findings. This is a clean, well-scoped fix.

What I checked:

  • The new assert_matches helper (cant-fail-scan.test.sh:43-53) — traced the ERE (^date: [0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$) against the emitter's actual timestamp construction at cant-fail-scan.sh:347 (date -u +%Y-%m-%dT%H:%M:%SZ). The two match by construction, and the anchors correctly reject both malformed shapes described in the PR body. grep -qE matches line-by-line against the multi-line $out, so anchoring to a single line inside a larger string is correct here, not a bug.
  • Confirmed no other occurrence of the old date: 20 prefix pattern survives anywhere in the testing plugin (grepped the whole plugin tree) — the fix is complete, not partially applied.
  • The dropped "Windows-safe" rationale in overengineering/context/findings-artifact.md — grepped every other "Windows-safe"/"colon-free" usage across the marketplace; every other instance legitimately describes a filename timestamp (e.g. session-flow, bug-report, mutation-testing, ai-slop's own emitter). This was the only misapplied instance (a frontmatter field, not a filename), so the fix is correctly scoped and nothing analogous was missed elsewhere.
  • The new README citation of docs/upstream/cursor-pstack.md in plugins/ai-slop/README.md — verified the relative path (../../docs/upstream/cursor-pstack.md from plugins/ai-slop/README.md) resolves correctly, and cross-checked the claim against the actual unslop row in that ledger — the row genuinely does record what was taken/deduplicated/rejected and does set the next-diff baseline, so the sentence is accurate.
  • Version bumps, changelog heading format, and diff scope (each changed file matches what the PR body claims) all check out.

Not independently re-run: the shell test suites, shellcheck, and markdownlint invocations the PR body reports — this lane's tool grant doesn't extend to running arbitrary scripts, only gh pr diff/view/comment/review plus read-only file tools. Those results are author-claimed, not independently verified by this review; the static trace above (regex vs. actual emitter output) is what I could confirm without executing anything.

@github-actions

Copy link
Copy Markdown
Contributor

Claude has reviewed this PR 1 time. The lane skips further automatic reviews after 5; deleting this comment resets the count.

@kyle-sexton
kyle-sexton merged commit ce0ed05 into main Aug 23, 2026
50 checks passed
@kyle-sexton
kyle-sexton deleted the claude/close-verifier-nits branch August 23, 2026 05:07
kyle-sexton pushed a commit that referenced this pull request Aug 23, 2026
Second overengineering version collision in a row: main took 0.2.1 for the
findings-artifact date-rationale fix (#3167) after taking 0.2.0 for the delta
lane (#3146). Main's entries stay where they are and this branch's
documentation-only entry moves to 0.2.2.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RiAekqvuMHJs3CjusBbNWH
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants