Skip to content

feat(debugging): add minimize, pristine-copy diff and HITL template note to /debugging:debug #8825

Description

@kyle-sexton

Refs: #7581
Refs: #7555

Agent Brief

Type: Feature
Summary: Add three steps to /debugging:debug, all as new text. Phase 2 gains a minimize step
that shrinks the reproduction to the smallest case that still fails. Phase 5 gains a pristine-copy
diff: when the red was forced by editing code or a fixture, prove the edit landed before trusting
the red. The human-driven loop gains a note that every answer the script collects is pasted back
into the session, so it asks only for observations and leaves signing in to an instruct step.

Problem: Phase 5 tells the agent to write its test against "the minimized repro", but no phase
produces one. Without a minimize step the agent carries every input, caller and config value from
the first reproduction into Phase 3, so more causes stay in play and the regression test is larger
than the bug needs. A red forced by editing code or a fixture can come from an edit that never took
effect (wrong file, wrong copy, a build that did not pick it up), and nothing asks the agent to
check. The human-driven script prints every answer into a block the user pastes into the session,
and nothing warns the author against asking for a password or token.

Current behavior:

  • Phase 2 runs the loop and checks three things: the failure repeats, it is the reported failure,
    and the symptom is on record (plugins/debugging/skills/debug/SKILL.md:151-160). The gate reads
    "Phase 3 starts only once the loop has reproduced the bug" (SKILL.md:162). No step shrinks the
    reproduction.
  • Phase 5 step 1 says "write a test that fails on the minimized repro" (SKILL.md:236). Step 5
    re-runs the Phase 1 loop "on the original, full-size scenario" (SKILL.md:245-246).
  • Phase 5 step 2 checks that the test fails "for the intended reason" and names a typo, a bad import
    or an unrelated defect as false reds (SKILL.md:239-242). Nothing covers a red produced by a
    deliberate edit to code or a fixture.
  • Intermittent bugs are tracked as a failure rate raised until about every other run fails
    (SKILL.md:124-135).
  • Comparing against a known-good build already exists as loop kinds: an automated bisect and a
    side-by-side run of two builds (SKILL.md:96-97).
  • The human-driven loop: copy the bundled template, replace its prompts, ask the user to run it, and
    ask the user to paste the === results === block back (SKILL.md:100-105). The template prints
    one NAME=value line per answer (plugins/debugging/skills/debug/scripts/hitl-loop.template.sh:13-14,
    :53-57), and the user "pastes that block back into the session" (hitl-loop.template.sh:14). Its
    example asks for observations only (hitl-loop.template.sh:39-45). Neither file warns against
    collecting a password, token or sign-in detail through ask.
  • The secrets table covers loop scripts, command lines and captures, not answers typed into the
    human-driven script (SKILL.md:54-67).
  • The bundled checklist's Phase 2 line is "Reproduce. Confirm the bug manifests deterministically OR
    characterize non-determinism" (plugins/debugging/skills/debug/templates/checklist.md:8). The
    README's Phase 2 line is "Run the loop; confirm it shows the failure the user described"
    (plugins/debugging/README.md:25).
  • skills/debug/evals/evals.json has 14 cases (ids 1-14). None covers minimizing, a forced red or
    the human-driven script.

Desired behavior:

  • Minimize (Phase 2). A ### Minimize subsection after the three checks and before the gate.
    Once the loop is red, cut inputs, callers, config, data and steps one at a time and re-run the loop
    after each cut. Keep a cut only when the loop stays red. Stop when every remaining element is
    needed: removing any one of them turns the loop green. Say why: fewer parts to suspect in Phase 3,
    and the minimized case becomes the Phase 5 test. For an intermittent bug, an element counts as
    needed when removing it drops the measured failure rate, not when one run happens to pass.
  • Phase 2 gate becomes "Phase 3 starts only once the loop has reproduced the bug and the
    reproduction is minimized." The Phase 2 heading stays ## Phase 2: Reproduce.
  • Pristine-copy diff (Phase 5 step 2). One added sentence: when the red came from editing code or
    a fixture on purpose, diff the edited file against a pristine copy (the committed version, or a copy
    saved before the edit) and confirm the edit is present before trusting the red.
  • HITL note (Phase 1, the human-driven loop paragraph). Add: every answer the script collects
    with ask is printed in the results block and pasted into the session, so ask only for what the
    user observes (yes or no, a number, copied error text). Never ask for a password, token or other
    secret. Signing in stays an instruct step the user does on their own. Point to the secrets table
    in the same file. The template script is not edited.
  • templates/checklist.md:8 and README.md:25 each gain the minimize step, in the same short form
    the line already uses.
  • skills/debug/evals/evals.json gains three cases (ids 15, 16, 17), prompt-only, files: []:
    1. Minimize: the prompt gives a loop that is red on a request with many fields, headers and config
      values. Expectation: before ranking causes, the output plans or makes cuts one element at a
      time with a loop run after each, keeps only elements whose removal turns the loop green, and
      does not move to hypotheses with the full request.
    2. Forced red: the user says they changed a fixture value to force the red. Expectation: the output
      diffs the edited fixture against the committed or saved copy and confirms the edit landed before
      trusting the red.
    3. Human-driven loop: the bug shows only after signing in to a staging site that sends a one-time
      code to the user's phone, so no automated loop can reach it. Expectation: the drafted steps put
      signing in as an instruct step, no ask collects a password, code, token or other sign-in
      detail, and the output hands the script to the user to run.
  • A changelog fragment .changes/debugging/<slug>.md with bump: minor and an ### Added entry
    describing the three steps (debugging is listed at scripts/fragment-plugins.txt:27). Write it by
    hand in the shape of an existing debugging fragment, named <branch-slug>-<8 hex>.md (this run
    cannot run scripts/new-changelog-fragment.sh). Do not edit plugin.json or CHANGELOG.md.

Known calls (2026-10-21):

  • The HITL note goes in SKILL.md, not in hitl-loop.template.sh. The script is mode 100755, and
    this lane cannot edit an executable.
  • "Pristine-copy diff" means proving that a forced edit landed. Comparing against a known-good build
    is already covered by the bisect and side-by-side loop kinds (SKILL.md:96-97) and is not changed.
  • The template keeps its rule that the user runs the script, because read needs a TTY the Bash
    tool lacks (SKILL.md:100-105; hitl-loop.template.sh:5-7). The upstream wording, in which the
    agent runs it, was declined.
  • The change only adds text. No existing sentence is removed. The Phase 2 gate sentence is extended,
    not rewritten. So no before-and-after eval run is owed.
  • No volatile upstream specific is added. The HITL note describes our own template's output, not a
    Claude Code behavior.
  • SKILL.md, the checklist, the README and the fragment name no upstream source, per the repo's
    rule against provenance notes.
  • Spelling is American English (_typos.toml sets locale = "en-us"), so the skill text says
    "minimize", matching "the minimized repro" at SKILL.md:236.
  • No shell script is added or edited. No path in the diff has a .claude segment. No eval case
    needs a scaffold script, so none is added under plugins/debugging/evals/.

Key interfaces:

  • plugins/debugging/skills/debug/SKILL.md (Phase 1 human-driven paragraph, Phase 2, Phase 5 step 2).
  • plugins/debugging/skills/debug/templates/checklist.md.
  • plugins/debugging/README.md.
  • plugins/debugging/skills/debug/evals/evals.json.
  • .changes/debugging/<slug>.md.

Done when:

  • plugins/debugging/skills/debug/SKILL.md has a ### Minimize heading inside Phase 2, before the
    **Gate: Phase 3 starts only line, and that gate line names minimizing (read from the diff).
  • Phase 5 step 2 in SKILL.md names a diff against a pristine copy for a red forced by editing
    code or a fixture (read from the diff).
  • The human-driven loop paragraph in SKILL.md says that ask answers are pasted back into the
    session and that signing in is an instruct step (read from the diff).
  • Every sentence of SKILL.md, checklist.md and README.md at the base is still present
    after the change, apart from the Phase 2 sentences it extends (the gate, the checklist line and
    the README line). Rewrapped lines are allowed
    (read from the diff).
  • plugins/debugging/skills/debug/scripts/hitl-loop.template.sh is not in the diff, no path in the
    diff has a .claude segment, and no added file is executable (read from the diff).
  • evals.json has cases with ids 15, 16 and 17 covering minimize, the forced red and the
    human-driven loop (read from the diff).
  • plugins/skill-quality/scripts/check-evals-quality.sh plugins/debugging/skills/debug/evals/evals.json
    exits 0.
  • scripts/check-changelog-fragments.sh --check passes.
  • CI's check-skills passes on the PR (skill gate, eval-set schema and quality lint). This is a PR
    check; this run does not run it.
  • CI's lint-repo passes on the PR (markdown). This is a PR check; this run does not run it.

Out of scope:

  • Editing hitl-loop.template.sh or adding a PowerShell port of it.
  • Renaming Phase 2 or changing the skill's description.
  • Any change to /debugging:analyze-profile (the compare, GC-root and capture-mode slices).
  • New plugin-root eval cases with scaffolds, and running claude plugin eval.
  • Changing the bisect or side-by-side loop kinds.

Origin:

  • Minimize: cut one element at a time, re-run after each cut, done when every remaining element is
    needed, and the minimized case becomes the regression test. Matt's: diagnosing-bugs/SKILL.md:78-86
    at 49dd158d, as summarized in .work/poteto-review/topic10/sections/B-quality.md:75, :91. Lauren:
    no counterpart (B-quality.md:228).
  • Pristine-copy diff: "If you forced the red by mutating code or a fixture, diff against a pristine
    copy to prove the mutation landed before you trust it." Matt's: diagnosing-bugs/SKILL.md:125
    (his test(markdown-format): port differential C1 fd1-leak detector from desktop-notification #1209), as summarized in B-quality.md:75, :92. Lauren: no counterpart.
  • HITL note: the upstream template says each answer is printed back, so it should "capture
    observations, and leave signing in to the user as a step". Matt's: hitl-loop.template.sh:15-16,
    as summarized in B-quality.md:75, :93. Lauren: no counterpart. This run must not fetch those
    sources (github.com is not a lane host). The quotes here are the record.
  • The owner's, as recorded in .work/poteto-review/shared-understanding.md:165: accepting the
    three additions (Q524, the Topic 10 accept list); the stop after two failed fixes on one assumption
    stays in /debugging:debug (Q16); live mechanism proof stays in /debugging:debug (D55). This
    slice changes neither of the last two.
  • Placing the HITL note in SKILL.md instead of the script, the failure-rate rule for minimizing an
    intermittent bug, and the three eval cases: ours.

Do not run with: the other #7581 debugging slices (compare and GC-root path in
/debugging:analyze-profile; dropping its capture mode). They edit the same plugin.

Size Small, Complexity Low, Uncertainty Known.


Slice of #7581 (Phase 13), drafted by the poteto-review lane from the thread's recorded decisions.

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.status: readyTriaged, unblocked, and fully specified; eligible to pick up.

    Fields

    No fields configured for Feature.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions