Skip to content

PRD: Loop Engineering adoption — 7 stop-signal & anti-cheat harness doc-edits #1

Closed

Description

@wasikarn

Generated by to-prd from a grill-with-docs session. Source plan: ~/.claude/plans/tranquil-skipping-whisper.md (v2, post-grill). Raw grill output: session task w7l64jg1h. Pipeline: this PRD → to-issues splits it into 7 per-item tickets. Green bar: harness-audit + claude plugin validate --strict (NOT tdd — see Testing Decisions).

Problem Statement

I mined an 8-article "Loop Engineering" corpus to decide what to adopt into this harness. The honest finding is that the harness already is loop engineering — roughly 12 of the corpus's 32 concepts are already embodied (the recursive-improve cycle, orchestrate, article-mine, the verification journal, doctrine-bootstrap, 27 maker≠checker agents, METHODOLOGY Rule 4). The corpus is mostly external validation of the existing design, not new capability.

But a six-lens grill of the adoption plan surfaced a genuine, narrow gap that the corpus is unanimous about: most of loop engineering is making loops know when to stop. Several of the harness's own loops can churn or advance without an explicit stop-signal — the recursive-improve candidate executor, the fix-bug hypothesis cycle, the address-review per-cluster fix loop, the ship-merge gate — and two agent-surface gaps let a regression-disguised-as-fix (tests weakened until green) pass review. None of these is dangerous today; each is a missing escalation that, under load, turns into wasted iteration or a silent quality leak.

The constraint that makes this hard: this is a config repo that deliberately rejects any autonomous, unattended, or multi-iteration self-repair loop (the load-bearing autonomy invariant). So the fix cannot be "add an auto-retry" or "add a verifier agent that loops." It has to be doc-only, human-gated, and add no new always-on cost.

Solution

Seven surgical documentation edits to existing components. Each one either (a) adds a missing stop-signal so a loop escalates to the human instead of spinning, (b) wires an acceptance contract the workflows already produce-and-consume but never invoke, or (c) sharpens an anti-cheat check / names one principle the harness already follows. Every edit terminates at an existing human approval gate (AskUserQuestion). None introduces a new agent, a new hook, new persistent state, or any unattended iteration. The retry policy stays a single source of truth in doctrine (METHODOLOGY Rule 13); the command edits reference it rather than re-encode the number.

The seven items, by component:

  1. recursive-improve cycle — when a candidate's executor hits the same failure, it does not re-attempt; it records "not done" with the verbatim signal and surfaces at the report step (a plain restatement of Rule 13 at the candidate level — no failure counter, because the cycle runs each candidate once).
  2. fix-bug hypothesis cycle — a no-progress halt: if instrumentation returns the same missing-evidence result twice for the same failure signal, stop re-ranking hypotheses and ask the user (re-investigate / escalate). Reuses the existing fallback count; this is a stagnation guard, not a retry cap.
  3. address-review fix cycle — cap fix attempts per cluster (one retry, per Rule 13); on failure, stop the cluster, re-classify the thread as clarify/wontfix with a note, and surface it.
  4. feature-dev + fix-bug acceptance wiring — for non-trivial scope, invoke the acceptance-contract skill to lock a machine-checkable contract before architecture/implementation. Firm in feature-dev (real design phase); optional in fix-bug (surgical fixes already have the regression gate). The skill self-gates triviality; the merge review already consumes the contract — only the producer call was missing.
  5. ship-merge gate — a conditional checklist mirroring the release flow's "zero Critical" wording: if a review ran on this PR, confirm zero unresolved Critical and surface any open acceptance gap; if no review ran, note it and pass (never fabricate). Prose checklist at the existing gate, not a programmatic ledger read.
  6. pr-test-analyzer agent — add a "weakened-to-pass tests" gap class (assertion deleted / loosened / skip-xfail / try-catch-wrapped to greenwash the suite). It is rated and paper-trailed exactly like every other gap the agent reports.
  7. METHODOLOGY Rule 4 — one sentence: when a loop verifies, the stop-signal must reduce to an objective check (test result, exit code, fresh-context adversarial pass), not the implementer agreeing with itself; a verifying agent must get fresh context, not the implementer's transcript. Scoped to the verification gate so it does not contradict the human approval gates the harness is built on.

User Stories

  1. As the harness owner, I want each of my self-improvement loops to stop and ask me when it stops making progress, so that I never discover an agent burned iterations re-running the same failing fix.
  2. As the harness owner, I want the recursive-improve candidate executor to escalate on a repeated failure instead of inventing recovery, so that the autonomy invariant holds at the candidate level.
  3. As someone debugging with fix-bug, I want the hypothesis loop to stop after two no-progress instrumentation rounds, so that I'm pulled in to re-frame the problem instead of watching it churn through low-value hypotheses.
  4. As someone addressing review feedback, I want a cluster that fails its tests twice to halt and be re-classified, so that one stubborn thread doesn't block the whole address-review pass silently.
  5. As the harness owner, I want non-trivial features to lock a machine-checkable acceptance contract before design starts, so that the merge review has a concrete target to verify against instead of "looks done."
  6. As the harness owner, I want the acceptance-contract producer actually invoked by the workflows, so that the consumer (the merge review's acceptance-gap check) stops being dead wiring.
  7. As the harness owner, I want surgical bug fixes to skip acceptance ceremony, so that the fix-bug fast path stays fast (its regression test is already the locked criterion).
  8. As the harness owner, I want ship-merge to confirm zero unresolved Critical findings when a review exists, so that I don't merge over an open Critical by reflex.
  9. As the harness owner, I want ship-merge to be honest when no review ran (note and pass, never fabricate a clean bill), so that the gate never lies.
  10. As a reviewer (human or agent), I want "tests weakened until the suite goes green" named as a critical gap class, so that a regression disguised as a fix is caught before merge.
  11. As the harness owner, I want one doctrine sentence stating that a verification stop-signal must be objective and a verifying agent must get fresh context, so that "loop until verified" can't degrade into the implementer signing off on its own work.
  12. As the harness owner, I want that doctrine sentence scoped to verification gates, so that it doesn't read as forbidding the human approval gates the whole harness depends on.
  13. As the harness owner, I want the retry cap to live once in Rule 13 and be referenced by the commands, so that tuning the cap is a one-line change, not four.
  14. As a future maintainer, I want every edit to terminate at an existing human gate, so that I can trust the harness never grew an autonomous loop while I wasn't looking.
  15. As the harness owner, I want no new always-on doctrine cost beyond a single sentence, so that the recent token-tightening of the doctrine files isn't quietly reversed.
  16. As a future maintainer, I want each item verified by harness-audit + plugin validate rather than by a string-presence "test," so that the green bar is honest and not theater.
  17. As the harness owner, I want the items that didn't earn their place (cosmetic edits, a doc row for a non-existent check, vocabulary that collides with existing terms) left out, so that the change stays minimal.

Implementation Decisions

  • Seven components touched, all existing: the recursive-improve skill, the fix-bug command, the address-review command, the feature-dev command, the ship-merge command, the pr-test-analyzer agent, and the METHODOLOGY doctrine file. No new files, no new agents, no new hooks.
  • Single source of truth for the retry cap. The "one retry, then escalate" policy stays in METHODOLOGY Rule 13. The three loop edits (items 1–3) reference Rule 13 and add only their site-specific halt mechanic; they do not restate the number. This was a grill finding — the draft had four near-identical copies.
  • Item 2 is a stagnation guard, not a retry cap. Its metric is "same missing-evidence result twice," which is a different concept from "retry count." It must not borrow the words "cap"/"retry," to keep it distinct from Rule 13.
  • Item 1 carries no failure counter. The recursive-improve cycle executes each candidate once; a "count to 2" would presuppose a retry budget that doesn't exist and would imply autonomous re-attempts. The edit is a plain Rule-13 restatement at the candidate level, routed to the existing "not done" sink.
  • Acceptance-contract wiring (item 4) uses the slash invocation form to match the one existing reference in the merge-review skill (the skill is model-invocable, so the slash form is valid). It does not re-encode the triviality threshold — that is deferred to the skill's own when-to-use (per the doctrine rule against duplicating a decision). Firm in feature-dev, optional in fix-bug.
  • Acceptance contract locks before architecture, not "end of clarifying questions." The lock point is the transition into the design phase, framed "lock the contract before architecture/implementation," and cross-references (not duplicates) the existing informal acceptance-criteria language in the discovery phase.
  • Anti-cheat bullet (item 6) conforms to the agent's existing paper-trail contract. It is rated 1–10 and cites file:line like every other gap — a grill correction of a draft that had it wrong (the draft told the implementer to avoid the rating format, which inverted the agent's actual contract).
  • Doctrine addition (item 7) is one sentence appended to Rule 4, scoped to the verification gate. It absorbs the only genuinely-new idea from a larger Theme-4 draft (the verifier-objectivity / fresh-context constraint) and drops the rest. A separating blank line must be added before the next rule's header (currently missing).
  • Gate vocabulary is split, not merged. "Gate" continues to mean a human-confirmation checkpoint (the DB-write gate, AskUserQuestion approval). "Stop-signal" means the objective machine-checkable condition that lets a loop terminate. Item 7 talks about the stop-signal feeding a verification gate; it does not redefine "gate."
  • ship-merge check (item 5) is a prose checklist at the existing gate, not a programmatic read — the review ledger is ephemeral and has no resolved/unresolved field. It mirrors the release flow's existing "zero Critical" wording.

Testing Decisions

  • What makes a good test here: these are markdown edits to doctrine/agents/skills/commands; there is no executable behavior to exercise through a public interface, so traditional behavioral TDD does not apply. A grep -q "<the string I just wrote>" "test" is a tautology that the harness's own audit explicitly classifies as test theater — we will not write those.
  • The green bar is: harness-audit (the audit script) exits clean (findings ≤ a pre-edit baseline), claude plugin validate --strict exits 0 (the existing CI conformance bar), and a human re-reads each edited section. For the one doctrine edit, a before/after character-count check guards against a token-budget regression.
  • Per-item acceptance is a manual content check: the named clause is present at the named anchor and no sibling contract was broken (e.g. the anti-cheat bullet must remain under the agent's rating ritual). This is honestly marked manual — the acceptance-contract skill explicitly permits manual criteria when a check can't be mechanized.
  • Modules covered by the mechanical bar: all seven, via harness-audit (frontmatter/symlink/loadability) and plugin validate (manifest/frontmatter). Note: neither asserts prose content — they are the no-regression sentinel, not the primary acceptance.
  • Prior art: the existing hook test harness (hooks/tests/) and the audit script's own check suite are the model for objective harness self-verification.

Out of Scope

  • Cut by the grill (Rule 2 — simplicity): the code-reviewer "positive sentence" edit (its headline action removed a clause that doesn't exist anywhere; the residue duplicated existing framing); the DBGATE "objective gate" edit (it inverted DBGATE's own definition of "gate"); the "loop author vs prompter" doctrine vocabulary (collides with the existing "author" meaning, names /loop which isn't a repo component, and adds always-on cost for no behavior); the harness-audit "stale-component" doc row (a row for a detector that isn't built would make the audit's "what it checks" table lie about the tool).
  • Rejected corpus concepts: autonomous self-rewriting meta-loop; autonomous goal-runner; Stop/PostToolUse self-verification hook (this is a config repo with no app test suite — that belongs in consumer projects); permission re-audit / sample-trace always-on hooks; a fixer-mode agent; numeric review scores / merge thresholds.
  • The ~12 already-embodied concepts are external validation of the existing design, not work — building components for them would be inventing work.
  • tdd as the verification method for this PRD (see Testing Decisions).

Further Notes

  • Provenance: items and anchors were produced by a multi-agent mine of the Loop Engineering corpus, then stress-tested by six grilling lenses (ubiquitous-language, invariant-adversary, code-truth/anchor, PRD-readiness, simplicity/removal, workflow-fit). The grill reads against the real files and corrected several draft anchors; the surviving anchors are verified.
  • Cleanup ticket METHODOLOGY Rule 4: objective verification stop-signal + fresh-context verifier #8 (separate, does not block this PRD): the autonomy invariant — the load-bearing constraint behind this whole PRD — is currently cited to a HARNESS.md file that git history confirms never existed, yet four hooks, one hook-test, and two skill citations reference it as a real doctrine file ("added 2026-06-10"). Recommended fix: home the invariant canonically in CONTEXT.md's invariants section, repoint the two skill citations, and remove the phantom from the hook gate-lists + hook-test in one cleanup commit. This PRD's item 1 already references Rule 13, never the phantom citation, so it is unaffected.
  • Known doc drift (separate): CONTEXT.md lists "no plugin validation CI" as a non-goal, but a plugin-validate CI workflow now exists — a one-line correction, tracked separately.
  • Execution: per the chosen hybrid pipeline, this PRD is split by to-issues into seven per-item tickets (P1 first: items 1, 2, 4, 6), each carrying its verified file:line anchor and its manual acceptance string. Implementation is intended to flow through the recursive-improve ritual (human-gated, one bounded cycle) so each edit gets Observe → ask → verify → drift-guard for free.

Activity

  1. added
    documentationImprovements or additions to documentation
    ready-for-agentPRD/issue ready for an agent to implement
    on Jun 11, 2026
  2. wasikarn commented on Jun 11, 2026

    @wasikarn
    OwnerAuthor

    Epic complete. All 7 core doc-edits (#2–#8) + 2 candidates (#9, #10) implemented, verified (green bar + fresh-context adversarial pass per ticket), and merged to develop across PR #11 and PR #12. The HARNESS.md phantom repoint and CONTEXT.md:52 cleanup landed alongside. Nothing outstanding from this PRD.

  3. 1 remaining item

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationready-for-agentPRD/issue ready for an agent to implement

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions