Skip to content

pr-test-analyzer: add 'weakened-to-pass / gate-gaming' critical-gap class #5

Closed

Description

@wasikarn

Parent

#1 — PRD: Loop Engineering adoption

What to build

In the pr-test-analyzer agent's "Identify Critical Gaps" list, add a named "Weakened-to-pass / gate-gaming" gap class with the explicit taxonomy (lifted from the corpus):

Weakened-to-pass tests — the suite is made green without fixing the code: deleting tests, loosening assertions, wrapping in try/catch to silence errors, marking tests skipped/xfail.

Include two operational additions:

  • Inspect diffs for touched test files even when the suite is green (the cheat hides in test edits, not source).
  • Gate-rot variant — a test that still passes but no longer catches the failure mode it was written for.

The bullet(s) stay under the agent's existing 1-10 behavioral-criticality + file:line paper-trail ritual — they do NOT bypass that format (this gap class is typically 7-9 criticality). External support: "current approaches place excessive faith in AI-generated tests, currently inadequate for behavior verification."

Acceptance criteria

  • A named weakened-to-pass / gate-gaming gap class is added to "Identify Critical Gaps," matching sibling noun-phrase style.
  • The taxonomy (delete / loosen / try-catch-silence / skip-mark) is present, plus the "inspect touched test files even when green" instruction and the gate-rot variant.
  • The bullet remains subject to the agent's existing 1-10 rating + file:line paper-trail ritual (no instruction to omit them).
  • Green bar: bash skills/harness-audit/scripts/audit.sh clean vs baseline AND claude plugin validate --strict exits 0.

Blocked by

None — can start immediately.

Activity

  1. added
    documentationImprovements or additions to documentation
    ready-for-agentPRD/issue ready for an agent to implement
    on Jun 11, 2026
  2. wasikarn commented on Jun 11, 2026

    @wasikarn
    OwnerAuthor

    Implemented in c58e02d (squash-merged via PR #11). Green bar + fresh-context adversarial verify passed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationready-for-agentPRD/issue ready for an agent to implement

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions