From c71e466a67d495e563e4adee975bed9abae39853 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Tue, 21 Jul 2026 05:04:17 -0400 Subject: [PATCH 1/2] feat(planning): add incumbent mode to devils-advocate MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Extends /planning:devils-advocate with an `incumbent` mode that turns the same adversarial discipline on the status quo β€” stress-testing an incumbent tool or approach already in place against alternatives, as pre-implementation decision support (not a post-hoc runtime audit). Adds an Alternatives Sweep that runs in place of Rounds 1-4 in this mode: explore the incumbent first-hand via a fresh sub-agent (digest-bias guard β€” carries what to investigate, never the parent's conclusions), name the problem it solves, walk the native > official > vetted-third-party ladder with coupling priced, and reach a KEEP / MIGRATE / RESEARCH verdict. Inherits the skill's evidence mandate (no training-data-only findings) and routes load-bearing evaluations to /discovery:research, pointing to /re-anchor:pick-for-the-problem for the full selection discipline. Research depth is a per-invocation deep/shallow token, not a userConfig knob. Reconciles the new always-dispatch rule with the skill's existing Fresh-context requirement into one policy. Adds 4 eval cases; planning 0.23.1 -> 0.24.0. πŸ€– Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Fable 5 --- plugins/planning/.claude-plugin/plugin.json | 5 +- plugins/planning/CHANGELOG.md | 20 ++++++++ .../planning/skills/devils-advocate/SKILL.md | 48 ++++++++++++++++--- .../skills/devils-advocate/evals/evals.json | 48 +++++++++++++++++++ 4 files changed, 114 insertions(+), 7 deletions(-) diff --git a/plugins/planning/.claude-plugin/plugin.json b/plugins/planning/.claude-plugin/plugin.json index 5cf041e43e..4081089d0e 100644 --- a/plugins/planning/.claude-plugin/plugin.json +++ b/plugins/planning/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json", "name": "planning", - "version": "0.23.1", + "version": "0.24.0", "userConfig": { "use_ask_user_question": { "type": "boolean", @@ -29,6 +29,9 @@ "devils-advocate", "plan", "stress-test", + "incumbent", + "alternatives", + "status-quo", "implementation-plan", "draft-goal-condition", "goal", diff --git a/plugins/planning/CHANGELOG.md b/plugins/planning/CHANGELOG.md index 70b324e812..e5cb835376 100644 --- a/plugins/planning/CHANGELOG.md +++ b/plugins/planning/CHANGELOG.md @@ -3,6 +3,26 @@ All notable changes to the `planning` plugin are documented here. Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); this plugin uses semantic versioning. +## [0.24.0] + +### Added + +- **`devils-advocate` gains an `incumbent` mode β€” adversarial review of the status + quo.** Alongside stress-testing a plan you hand it, the skill can now turn the same + discipline on an **incumbent** tool, library, or approach already in place: + `/planning:devils-advocate incumbent `. A new **Alternatives Sweep** replaces + the assumption-driven rounds β€” it explores the incumbent first-hand (a fresh + sub-agent runs `/discovery:explore`, never trusting a parent digest), names the + problem the incumbent actually solves, surveys alternatives on the + native > official > vetted-third-party ladder with coupling priced, and reaches a + **KEEP / MIGRATE / RESEARCH** verdict. It inherits the skill's evidence mandate (no + training-data-only findings) and routes load-bearing evaluations to + `/discovery:research` (`/re-anchor:pick-for-the-problem` supplies the full selection + discipline when installed). Research depth is a per-invocation `deep` / `shallow` + token, defaulting to the existing risk-scaled behavior. Scope is pre-implementation + decision support β€” keep-or-replace before a plan commits β€” not a post-hoc audit of a + running system. Additive; plan-review mode is unchanged. + ## [0.23.1] ### Changed diff --git a/plugins/planning/skills/devils-advocate/SKILL.md b/plugins/planning/skills/devils-advocate/SKILL.md index e0258e59db..f46ae2a80f 100644 --- a/plugins/planning/skills/devils-advocate/SKILL.md +++ b/plugins/planning/skills/devils-advocate/SKILL.md @@ -1,7 +1,7 @@ --- name: devils-advocate -description: "Stress-test plans and proposals via systematic adversarial review β€” assumption extraction, evidence check, failure scenarios, operational gotchas β€” before implementation begins. Use for 'devil's advocate', 'stress test', 'poke holes', 'what could go wrong', new dependencies, infrastructure/CI/build changes, or any architecture decision with cross-module blast radius; not for code correctness bugs or pre-PR verification." -argument-hint: "[plan text or file path] β€” works from conversation context if no argument given" +description: "Stress-test plans and proposals via systematic adversarial review β€” assumption extraction, evidence check, failure scenarios, operational gotchas β€” before implementation begins. Use for 'devil's advocate', 'stress test', 'poke holes', 'what could go wrong', new dependencies, infrastructure/CI/build changes, or any architecture decision with cross-module blast radius. An `incumbent` mode turns the same adversarial lens on the status quo β€” 'is there a better way now', 'should we still use X', 'reconsider the current approach', 'is the incumbent still the right choice' β€” surveying alternatives before a plan commits to keeping an existing tool or approach. Not for code correctness bugs or pre-PR verification." +argument-hint: "[incumbent [target]] or [plan text or file path] β€” an optional leading deep/shallow sets research depth; works from conversation context if no argument given" user-invocable: true disable-model-invocation: false --- @@ -21,9 +21,16 @@ Plans fail for predictable reasons: unchecked assumptions, undiscovered bugs in Not a rubber stamp. Find real issues that would cause rework, not generic warnings. Every finding must be backed by evidence β€” a specific bug number, doc reference, code path, or logical argument. "This might break" without evidence is not a finding. +The same discipline runs against the status quo. An incumbent tool, library, or approach already in place is a decision too, and "we already use it" is evidence of what is, never proof it still fits. The `incumbent` mode stress-tests that choice β€” naming the problem the incumbent actually solves, surveying alternatives, and asking whether a better fit exists now β€” before a plan commits to keeping or replacing it. + ## Fresh-context requirement -If the plan or proposal under review was produced in THIS context/session, dispatch this stress-test to a fresh-context sub-agent rather than running it inline β€” the producing context shares the assumptions that created the plan's blind spots and drifts toward approving its own work. When you were invoked on an artifact this context did not author (a file, a plan from another session, a diff), you are already the fresh pair of eyes β€” proceed directly. +This stress-test runs from a fresh pair of eyes, and dispatches to a fresh-context sub-agent in two cases: + +- **Plan-review mode** β€” if the plan under review was produced in THIS context/session, the producing context shares the assumptions that created the plan's blind spots and drifts toward approving its own work; dispatch the stress-test to a fresh-context sub-agent. When you were invoked on an artifact this context did not author (a file, a plan from another session, a diff), you are already the fresh pair of eyes β€” proceed directly. +- **`incumbent` mode** β€” always dispatch. The incumbent lives in the current codebase, so any read of it you already hold is a digest; a first-hand exploration is what forms an independent view. The sub-agent runs `/discovery:explore` (if installed, else explores directly) on the incumbent itself (Alternatives Sweep, Step 1). + +In both cases the dispatch prompt carries only WHAT to investigate β€” the plan artifact, or the incumbent's identity and where it lives β€” never your conclusions about it: "here is the target; go look yourself," not "here is what I found; confirm it." A sub-agent handed the parent's verdict inherits the parent's blind spot. ## When to Use @@ -39,15 +46,24 @@ If the plan or proposal under review was produced in THIS context/session, dispa - `/planning:devils-advocate` β€” review the plan currently being discussed in conversation - `/planning:devils-advocate ` β€” review a plan from a specific file - `/planning:devils-advocate ` β€” review the provided text directly +- `/planning:devils-advocate incumbent ` β€” stress-test the incumbent tool/approach against alternatives (target empty β‡’ take it from conversation context) ## Input Resolution -1. If `$ARGUMENTS` contains a file path (ends in `.md`, `.txt`, or `.json`), read that file -2. If `$ARGUMENTS` contains inline text, use that as the plan -3. If `$ARGUMENTS` is empty, work from the current conversation context β€” identify the most recent plan, proposal, or design being discussed +Parse `$ARGUMENTS` in this order: + +1. **Depth token (optional).** If the first token is `deep` or `shallow`, consume it as the research-depth override (see "Research depth" below) and continue with the rest. +2. **Mode.** If the next token is `incumbent`, enter **`incumbent` mode** (incumbent-target); the remainder identifies the incumbent β€” a tool, library, approach, or module β€” or is empty to take the incumbent from the current conversation. The keyword selects the mode only as this leading token; a plan that merely contains the word elsewhere is not a mode switch. +3. **Plan-review mode (default).** Otherwise: if the remainder is a file path (ends in `.md`, `.txt`, or `.json`), read that file; if it is inline text, use it as the plan; if empty, work from the current conversation context β€” the most recent plan, proposal, or design being discussed. + +### Research depth + +Both modes default to **risk-scaled** research (Round 2's high/medium/low scale). A leading `deep` token forces the heaviest tier β€” route load-bearing evaluations to `/discovery:research-deep` if installed; `shallow` restricts to codebase read/grep with no external research. Depth is a per-invocation choice, not a stored setting. ## Analysis Process +**Mode branch.** In plan-review mode, run Rounds 1–4 below. In `incumbent` mode, run the **Alternatives Sweep** instead (it reuses Round 2's evidence discipline and Round 3's mitigation / residual-risk format); Rounds 1–4 do not apply. + Run the rounds below (up to 4). Stop early if a round produces no new critical or high findings β€” except Round 4, which runs whenever its multi-layer / multi-context trigger matches, regardless of how quiet Rounds 1-3 were. ### Round 1: Assumption Identification @@ -127,6 +143,23 @@ Findings use the same severity / failure-scenario / mitigation / residual-risk f **When to run Round 4:** plans involving multi-layer composition (config layering, plugin extension points, hook chains, override mechanisms), or any plan whose blast radius spans multiple contexts (local + CI + cloud). Skip Round 4 for single-context single-mechanism plans where Round 3 already covers the failure surface. +### Alternatives Sweep (`incumbent` mode) + +Runs in place of Rounds 1–4 when `incumbent` mode is selected. It inherits the evidence mandate β€” every finding is backed by a specific bug number, doc reference, code path, or concrete logical argument, never training-data recall. + +1. **Explore the incumbent first-hand.** Dispatch the fresh sub-agent (see Fresh-context requirement) to run `/discovery:explore` on the incumbent β€” what it is, where it is used, what it is coupled to, and any recorded reason it was chosen. The sub-agent forms its own read; it receives the incumbent's identity, never a parent conclusion about it. +2. **Name the actual problem.** State what the incumbent solves β€” the real requirements, present and plausible-future β€” before any alternative is on the table. Do not let the incumbent's shape define the problem. +3. **Survey the field.** Judge candidate alternatives against those requirements, walking the preference ladder β€” **native** (what the platform / language / framework already provides) > **official / authoritative** > **vetted third-party** (well-maintained, known, safe, secure) β€” where an earlier rung wins when it covers the requirements. Price each dependency's coupling: abandonment, a pricing pivot, a license change, security posture, exit cost. The full selection discipline lives in `/re-anchor:pick-for-the-problem` (apply it if installed); this baseline is enough to run the sweep without it. +4. **"Is there a better way now?" β€” evidence, not memory.** The research-heavy step; scale to risk (Round 2) or the depth token. Route load-bearing evaluations to `/discovery:research` (or `/discovery:research-deep`) if installed β€” a tool's maintenance, security, licensing, and native-alternative landscape drift constantly since the training cutoff. Look especially for what changed since the incumbent was chosen: a new native capability, a shifted dependency, a since-published better-fit option. +5. **Verdict per candidate.** One of: + - **KEEP** β€” re-derived from the problem and still the best fit; the duty is to re-derive, not to switch for switching's sake. An incumbent that audits clean is a clean finding β€” say so. + - **MIGRATE** β€” a better-fit alternative exists; state the coupling price and the migration cost, not just the upside. + - **RESEARCH** β€” the evaluation is load-bearing and unverified; route it (step 4), never a verdict from recall. + + Findings use the same severity / failure-scenario / mitigation / residual-risk format as Round 3. + +**Scope guard.** This is pre-implementation decision support β€” should the plan adopt or keep X versus an alternative β€” not a post-hoc audit of a running system's health or correctness. + ## Output Format ### Risk Summary @@ -158,6 +191,8 @@ If critical or high findings exist, present specific plan modifications: - What to add (new steps, new checks, new graceful degradation) - What to remove (mechanisms that don't work) +In `incumbent` mode, this is the KEEP / MIGRATE / RESEARCH verdict with its coupling price and, for a MIGRATE, the migration cost β€” not just the upside. + ### Suggested Next Steps Based on findings, suggest relevant follow-up actions: @@ -172,6 +207,7 @@ Based on findings, suggest relevant follow-up actions: - **Does not replace code review** β€” it reviews plans, not code (use your code-review tooling for code) - **Does not do exhaustive security analysis** β€” it finds design-level risks, not vulnerability scanning (use dedicated security tools for that) - **Does not generate generic warnings** β€” every finding must have specific evidence. "This might break" without a bug number, doc reference, or logical argument is not acceptable +- **Does not audit a running system's health** β€” `incumbent` mode is a pre-implementation keep-or-replace decision against alternatives, not a runtime performance / correctness audit of production ## Workflow position diff --git a/plugins/planning/skills/devils-advocate/evals/evals.json b/plugins/planning/skills/devils-advocate/evals/evals.json index 3efeea0358..4e9bd2b783 100644 --- a/plugins/planning/skills/devils-advocate/evals/evals.json +++ b/plugins/planning/skills/devils-advocate/evals/evals.json @@ -60,6 +60,54 @@ "Output suggests a targeted follow-up (research the flag, or file a deferred research/monitoring item) instead of declaring the plan verified", "Output does not fabricate a definitive claim about the undocumented behavior it did not verify" ] + }, + { + "id": 6, + "name": "incumbent-mode-triggers-alternatives-sweep", + "prompt": "/planning:devils-advocate incumbent our logging β€” we built a custom JSON log formatter a year ago and everything uses it. Before the new service plan commits to it, is there a better way now?", + "expected_output": "The skill enters incumbent-target mode and runs the Alternatives Sweep: names the problem the custom formatter solves, surveys alternatives against the native > official > vetted-third-party ladder, prices coupling, and reaches a KEEP / MIGRATE / RESEARCH verdict β€” rather than treating the message as a plan to poke holes in.", + "files": [], + "expectations": [ + "Output evaluates the incumbent (the custom log formatter) against alternatives rather than only extracting assumptions from a plan", + "Output names the actual problem the incumbent solves before proposing alternatives, and considers native / first-party options, not only third-party", + "Output reaches a keep / migrate / research verdict, and treats 'we already use it' as not by itself a reason to keep" + ] + }, + { + "id": 7, + "name": "incumbent-first-hand-not-parent-digest", + "prompt": "/planning:devils-advocate incumbent the auth middleware. I'm pretty sure it's fine and we should keep it, but check whether a better option exists before the rewrite plan assumes it.", + "expected_output": "The skill investigates the incumbent first-hand (explores what it is and how it is used) and evaluates alternatives on evidence, rather than rubber-stamping the user's stated 'it's fine, keep it' conclusion.", + "files": [], + "expectations": [ + "Output forms its own read of the incumbent (explores / inspects it) instead of accepting the user's 'it's fine' conclusion at face value", + "Any keep-or-migrate verdict is backed by evidence (code path, doc, bug number, or concrete argument), not by the user's assertion", + "Output does not simply confirm the pre-stated conclusion without independent investigation" + ] + }, + { + "id": 8, + "name": "keep-verdict-is-legitimate-not-forced-switch", + "prompt": "/planning:devils-advocate incumbent we use the platform's built-in fetch for HTTP; a plan proposes swapping to a third-party client. Is the incumbent still the right call?", + "expected_output": "When the incumbent is re-derived from the problem and still fits (a native option covering the requirements), the skill returns a clean KEEP verdict and does not manufacture a migration for its own sake; it may flag the proposed third-party swap as a downgrade on the preference ladder.", + "files": [], + "expectations": [ + "Output can conclude KEEP when the incumbent (a native capability) still fits, rather than forcing a switch", + "Output does not fabricate a better alternative or a migration recommendation without evidence it is a better fit", + "Output weighs the native incumbent above an unjustified third-party dependency and prices the coupling of adopting one" + ] + }, + { + "id": 9, + "name": "incumbent-mode-not-post-hoc-system-audit", + "prompt": "/planning:devils-advocate incumbent our database β€” go audit production for slow queries and tell me what's unhealthy right now.", + "expected_output": "The skill clarifies that incumbent-target mode is pre-implementation decision support β€” whether to keep or replace the database choice against alternatives β€” not a post-hoc health / performance audit of the running system, and reframes or declines the runtime-audit request.", + "files": [], + "expectations": [ + "Output clarifies its scope is a keep-vs-alternative decision, not a runtime health / performance audit of the live system", + "Output does not attempt to diagnose production slow queries or current system health", + "Output reframes toward the pre-implementation choice (is this database still the right fit vs alternatives) or points elsewhere for the runtime audit" + ] } ] } From 22a03e4e8a5bf1032c7b5d9579c9664abd4aae4e Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Tue, 21 Jul 2026 05:12:14 -0400 Subject: [PATCH 2/2] docs(planning): fold devils-advocate incumbent-mode review findings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Independent review of #823 (clean, no critical). Folds: - README catalog row now documents the incumbent mode / Alternatives Sweep inline, matching sibling rows that describe their sub-behaviors. - Output Format notes how the verdict (KEEP/MIGRATE/RESEARCH) and the Assumption/Failure-scenario field labels re-read in incumbent mode. - Input Resolution adds an escape hatch: an inline plan beginning with incumbent/deep/shallow is passed as a file path so the leading word is not consumed as a mode/depth token. πŸ€– Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Fable 5 --- plugins/planning/README.md | 2 +- plugins/planning/skills/devils-advocate/SKILL.md | 4 ++++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/plugins/planning/README.md b/plugins/planning/README.md index b577c9d379..f530f56bed 100644 --- a/plugins/planning/README.md +++ b/plugins/planning/README.md @@ -17,7 +17,7 @@ where artifacts land in the consuming repo. | `/planning:draft-goal-condition` | Goal authoring | Crafts a paste-ready `/goal` completion condition from a stated intent β€” reads the current official `/goal` docs live for the condition shape and character limit (nothing hardcoded), drafts a transcript-demonstrable condition, and proves it fits the limit with a deterministic character counter instead of model guesswork; a lever-fit gate routes interval-shaped or cloud/sessionless work elsewhere. Standalone. | | `/planning:design` | Design space | Explores types, contracts, module boundaries, and package topology through collaborative discussion rounds, producing capability-matrix / type-inventory / design-threads / topology artifacts; its `handoff` action delegates to `/planning:design-handoff`. | | `/planning:design-handoff` | Designβ†’plan gate | Gates a finished design for `/planning:plan` β€” a binary check that every `design-threads.md` thread is RESOLVED, directional, or TAGGED-DEFERRED β€” then packages the plan-ready summary and resume prompt, or FAILs and routes back to `/planning:design`. | -| `/planning:devils-advocate` | Adversarial review | Stress-tests plans via assumption extraction, evidence checks, failure scenarios, and operational-gotcha sweeps β€” every finding evidence-backed, never generic warnings. | +| `/planning:devils-advocate` | Adversarial review | Stress-tests plans via assumption extraction, evidence checks, failure scenarios, and operational-gotcha sweeps β€” every finding evidence-backed, never generic warnings. An `incumbent` mode turns the same lens on the status quo: an Alternatives Sweep that stress-tests keeping an incumbent tool/approach against alternatives (native > official > vetted ladder, coupling priced, KEEP / MIGRATE / RESEARCH verdict), exploring the incumbent first-hand in a fresh sub-agent. | | `/planning:plan` | Implementation plan | Produces a structured plan (goal, approach, test strategy, blast radius, parallelism analysis, tagged unilateral decisions) with a mandatory fresh-context stress-test and a user approval gate, persisted to PLAN.md. | | `/planning:setup` | Configuration | `check` inspects the topic-docs seam and standards index read-only; `apply` interviews the consumer and persists the tracked `.claude/topic-docs.yaml` concern file that governs where every pipeline skill writes its per-topic artifacts, and bootstraps the standards index (idempotent β€” re-run to reconfigure). | diff --git a/plugins/planning/skills/devils-advocate/SKILL.md b/plugins/planning/skills/devils-advocate/SKILL.md index f46ae2a80f..78267197b4 100644 --- a/plugins/planning/skills/devils-advocate/SKILL.md +++ b/plugins/planning/skills/devils-advocate/SKILL.md @@ -56,6 +56,8 @@ Parse `$ARGUMENTS` in this order: 2. **Mode.** If the next token is `incumbent`, enter **`incumbent` mode** (incumbent-target); the remainder identifies the incumbent β€” a tool, library, approach, or module β€” or is empty to take the incumbent from the current conversation. The keyword selects the mode only as this leading token; a plan that merely contains the word elsewhere is not a mode switch. 3. **Plan-review mode (default).** Otherwise: if the remainder is a file path (ends in `.md`, `.txt`, or `.json`), read that file; if it is inline text, use it as the plan; if empty, work from the current conversation context β€” the most recent plan, proposal, or design being discussed. +To review an inline plan whose text legitimately *begins* with `incumbent`, `deep`, or `shallow`, pass it as a file path so the leading word is not consumed as a mode or depth token. + ### Research depth Both modes default to **risk-scaled** research (Round 2's high/medium/low scale). A leading `deep` token forces the heaviest tier β€” route load-bearing evaluations to `/discovery:research-deep` if installed; `shallow` restricts to codebase read/grep with no external research. Depth is a per-invocation choice, not a stored setting. @@ -162,6 +164,8 @@ Runs in place of Rounds 1–4 when `incumbent` mode is selected. It inherits the ## Output Format +**In `incumbent` mode**, the per-candidate **KEEP / MIGRATE / RESEARCH verdict** is the headline. The Risk Summary and finding bullets below still apply to the risks the sweep surfaces, with two field re-readings: **Assumption** becomes the claim under test (e.g. "the incumbent still fits" or "alternative X is better-maintained"), and **Failure scenario** becomes the cost of the wrong call (keeping a worse-fit incumbent, or paying an unpriced migration). + ### Risk Summary | Severity | Count | Action |