feat(bugs): concise skill text measured by evals, plus outcome cases and probes

.changes/bugs/feat-bugs-concise-skills-293dc1f1.md

@@ -0,0 +1,12 @@
+---
+bump: minor
+---
+
+### Changed
+
+- `write` and `scan` have shorter descriptions and bodies, and `setup` has a one-line description and a shorter body. Outcome eval cases and held-out trigger probes measured each change against the old text before it shipped.
+- The `!` pre-computed commands in `write` and `scan` fall back to a notice when the consumer-gotchas script fails.
+
+### Added
+
+- Five outcome eval cases for `write` and two for `scan` under `evals/`, and frozen trigger-probe sets for `write` and `scan` under `probes/`.

plugins/bugs/evals/sc-clean-file-no-invention/case.yaml

@@ -0,0 +1,34 @@
+schema_version: "1.1"
+name: sc-clean-file-no-invention
+context:
+  scaffold_script: scaffold.sh
+graders:
+  - name: hunter-dispatched
+    type: tool_used
+    tool: Agent
+    min: 1
+  - name: states-no-verified-findings
+    type: regex
+    pattern: '^[ \t]*[*_#]*[ \t]*no verified findings\b|\b(?:zero|0|no) (?:verified|confirmed) (?:findings|bugs)\b|\bverified(?: findings)?[ \t]*:[ \t]*(?:0|zero|none)\b|\bnothing (?:was )?verified\b'
+    flags: 'mi'
+    target: last_message
+  - name: no-verified-finding-presented
+    type: regex
+    pattern: '^[ \t]*#{1,6}[ \t]*(?:Finding[ \t]*\d|\d+[.)][ \t])|\bEvidence\b(?:\*\*)?[ \t]*(?:[:|—–]|-[ \t]|\*\*)[ \t]*(?:\*\*)?[ \t]*`?(?:reproduced|verified[- ]by[- ]reading)\b'
+    flags: 'mi'
+    match: not_contains
+    target: last_message
+  - name: no-write
+    type: tool_used
+    tool: Write
+    input_match: '.'
+    min: 0
+    max: 0
+    arm: both
+  - name: no-edit
+    type: tool_used
+    tool: Edit
+    input_match: '.'
+    min: 0
+    max: 0
+    arm: both

plugins/bugs/evals/sc-clean-file-no-invention/prompt.md

@@ -0,0 +1,10 @@
+---
+description: "A dry-run targeted scan of a correct one-file fixture reports zero verified findings and presents no finding section or evidence label, after dispatching at least one hunter"
+tags: [bugs, scan, outcome, no-invention, scaffold]
+max_turns: 30
+timeout_seconds: 900
+allowed_tools: [Skill, Bash, Read, Glob, Grep, Agent, Write, Edit]
+expected_outcome: "At least one Agent dispatch (a hunter). The reply states there are no (zero) verified findings and contains no 'Finding N' section and no 'Evidence: reproduced|verified-by-reading' line. No Write or Edit call."
+---
+
+/bugs:scan src/calendar.js --dry-run

plugins/bugs/evals/sc-clean-file-no-invention/samples/hunter-dispatched.json

@@ -0,0 +1,57 @@
+{
+  "pass": [
+    {
+      "answer": [
+        {
+          "tool": "Read",
+          "input": {
+            "file_path": "src/calendar.js"
+          }
+        },
+        {
+          "tool": "Agent",
+          "input": {
+            "description": "lens-2 hunter",
+            "prompt": "...",
+            "subagent_type": "general-purpose"
+          }
+        }
+      ]
+    },
+    {
+      "answer": [
+        {
+          "tool": "Agent",
+          "input": {
+            "description": "hunter",
+            "prompt": "...",
+            "subagent_type": "general-purpose"
+          }
+        },
+        {
+          "tool": "Agent",
+          "input": {
+            "description": "gate",
+            "prompt": "...",
+            "subagent_type": "general-purpose"
+          }
+        }
+      ]
+    }
+  ],
+  "fail": [
+    {
+      "answer": [
+        {
+          "tool": "Read",
+          "input": {
+            "file_path": "src/calendar.js"
+          }
+        }
+      ]
+    },
+    {
+      "answer": []
+    }
+  ]
+}

plugins/bugs/evals/sc-clean-file-no-invention/samples/no-edit.json

@@ -0,0 +1,53 @@
+{
+  "pass": [
+    {
+      "answer": [
+        {
+          "tool": "Agent",
+          "input": {
+            "description": "hunter",
+            "prompt": "...",
+            "subagent_type": "general-purpose"
+          }
+        }
+      ]
+    },
+    {
+      "answer": []
+    }
+  ],
+  "fail": [
+    {
+      "answer": [
+        {
+          "tool": "Agent",
+          "input": {
+            "description": "hunter",
+            "prompt": "...",
+            "subagent_type": "general-purpose"
+          }
+        },
+        {
+          "tool": "Edit",
+          "input": {
+            "file_path": "src/discount.js",
+            "old_string": ">",
+            "new_string": ">="
+          }
+        }
+      ]
+    },
+    {
+      "answer": [
+        {
+          "tool": "Edit",
+          "input": {
+            "file_path": "src/discount.js",
+            "old_string": ">",
+            "new_string": ">="
+          }
+        }
+      ]
+    }
+  ]
+}

plugins/bugs/evals/sc-clean-file-no-invention/samples/no-verified-finding-presented.json

@@ -0,0 +1,37 @@
+{
+  "pass": [
+    {
+      "answer": "Targeted run, effort high, small scope (1 file), 1 lens (lens-2 boundary/edge-case); hunter and gate on the session model. Candidates: 1 raw, 1 gated. Evidence labels (reproduced or verified-by-reading) apply only to survivors.\n\n*No verified findings.*\n\n**Lane**: src/calendar.js (1 file read, 1 candidate, all refuted)\n\n## Refuted candidates\n\n| # | Candidate | Lens | Location | Refuting argument |\n|---|---|---|---|---|\n| 1 | isLeapYear on negative years | lens-2 | `src/calendar.js:5` | -4 % 4 is -0, and -0 === 0 is true. |\n\n## Candidates not gated\n\nNone.\n\n## Side observations\n\nNone.\n\n*No scan cursor: targeted run, rotation not advanced.*"
+    },
+    {
+      "answer": "Zero verified findings in src/calendar.js. The hunter returned no candidates, which is a valid outcome; nothing reached the gate.\n\n## Refuted candidates\n\nNone — every candidate this run survived the gate."
+    },
+    {
+      "answer": "*No verified findings.*\n\nNo evidence reproduced a fault.",
+      "why": "prose, no label"
+    }
+  ],
+  "fail": [
+    {
+      "answer": "*No verified findings.*\n\n## Finding 1: daysInFebruary ignores year 0"
+    },
+    {
+      "answer": "Possible issue in isLeapYear.\n\n**Evidence**: verified-by-reading"
+    },
+    {
+      "answer": "**Evidence:** `reproduced`"
+    },
+    {
+      "answer": "*No verified findings.*\n\n| Evidence | reproduced |",
+      "why": "table label"
+    },
+    {
+      "answer": "Evidence - reproduced",
+      "why": "dash label"
+    },
+    {
+      "answer": "*No verified findings.*\n\n### 1. Off-by-one in isLeapYear",
+      "why": "numbered finding heading"
+    }
+  ]
+}

plugins/bugs/evals/sc-clean-file-no-invention/samples/no-write.json

@@ -0,0 +1,51 @@
+{
+  "pass": [
+    {
+      "answer": [
+        {
+          "tool": "Agent",
+          "input": {
+            "description": "hunter",
+            "prompt": "...",
+            "subagent_type": "general-purpose"
+          }
+        }
+      ]
+    },
+    {
+      "answer": []
+    }
+  ],
+  "fail": [
+    {
+      "answer": [
+        {
+          "tool": "Agent",
+          "input": {
+            "description": "hunter",
+            "prompt": "...",
+            "subagent_type": "general-purpose"
+          }
+        },
+        {
+          "tool": "Write",
+          "input": {
+            "file_path": "/tmp/report.md",
+            "content": "x"
+          }
+        }
+      ]
+    },
+    {
+      "answer": [
+        {
+          "tool": "Write",
+          "input": {
+            "file_path": "/tmp/report.md",
+            "content": "x"
+          }
+        }
+      ]
+    }
+  ]
+}

plugins/bugs/evals/sc-clean-file-no-invention/samples/states-no-verified-findings.json

@@ -0,0 +1,21 @@
+{
+  "pass": [
+    {
+      "answer": "Targeted run, effort high, small scope (1 file), 1 lens (lens-2 boundary/edge-case); hunter and gate on the session model. Candidates: 1 raw, 1 gated. Evidence labels (reproduced or verified-by-reading) apply only to survivors.\n\n*No verified findings.*\n\n**Lane**: src/calendar.js (1 file read, 1 candidate, all refuted)\n\n## Refuted candidates\n\n| # | Candidate | Lens | Location | Refuting argument |\n|---|---|---|---|---|\n| 1 | isLeapYear on negative years | lens-2 | `src/calendar.js:5` | -4 % 4 is -0, and -0 === 0 is true. |\n\n## Candidates not gated\n\nNone.\n\n## Side observations\n\nNone.\n\n*No scan cursor: targeted run, rotation not advanced.*"
+    },
+    {
+      "answer": "Zero verified findings in src/calendar.js. The hunter returned no candidates, which is a valid outcome; nothing reached the gate.\n\n## Refuted candidates\n\nNone — every candidate this run survived the gate."
+    },
+    {
+      "answer": "Verified findings: 0"
+    }
+  ],
+  "fail": [
+    {
+      "answer": "## Finding 1: isLeapYear mishandles century years\n\n**Evidence**: verified-by-reading"
+    },
+    {
+      "answer": "Verified findings: 1"
+    }
+  ]
+}

plugins/bugs/evals/sc-clean-file-no-invention/scaffold.sh

@@ -0,0 +1,18 @@
+#!/usr/bin/env bash
+# Seed the one file the scan targets.
+set -euo pipefail
+
+mkdir -p src
+cat >src/calendar.js <<'JS'
+// Gregorian calendar helpers. `year` is an integer.
+
+// True when `year` is a Gregorian leap year.
+export function isLeapYear(year) {
+  return (year % 4 === 0 && year % 100 !== 0) || year % 400 === 0;
+}
+
+// Days in February of `year`.
+export function daysInFebruary(year) {
+  return isLeapYear(year) ? 29 : 28;
+}
+JS

plugins/bugs/evals/sc-planted-bug-dry-run/case.yaml

@@ -0,0 +1,49 @@
+schema_version: "1.1"
+name: sc-planted-bug-dry-run
+context:
+  scaffold_script: scaffold.sh
+graders:
+  - name: hunter-and-gate-dispatched
+    type: tool_used
+    tool: Agent
+    min: 2
+  - name: gate-dispatched
+    type: tool_used
+    tool: Agent
+    input_match: '[Rr][Ee][Ff][Uu][Tt]'
+    min: 1
+  - name: no-zero-findings-claim
+    type: regex
+    pattern: '^[ \t]*[*_#]*[ \t]*no verified findings\b'
+    flags: 'mi'
+    match: not_contains
+    target: last_message
+  - name: finding-names-function
+    type: regex
+    pattern: '\bfix location\b[^\n]{0,200}\bdiscountRate\b|^[ \t]*#{1,6}[ \t]*(?:Finding\b|\d+[.)])[^\n]*\bdiscountRate\b'
+    flags: 'mi'
+    target: last_message
+  - name: evidence-label
+    type: regex
+    pattern: '\bEvidence\b(?:\*\*)?[ \t]*(?:[:|—–]|-[ \t]|\*\*)[ \t]*(?:\*\*)?[ \t]*`?(?:reproduced|verified[- ]by[- ]reading)\b'
+    flags: 'i'
+    target: last_message
+  - name: path-line-locator
+    type: regex
+    pattern: '\b(?:src/)?discount\.js(?::\d+|#L\d+|,? line \d+)'
+    flags: 'i'
+    target: last_message
+  - name: no-write
+    type: tool_used
+    tool: Write
+    input_match: '.'
+    min: 0
+    max: 0
+    arm: both
+  - name: no-edit
+    type: tool_used
+    tool: Edit
+    input_match: '.'
+    min: 0
+    max: 0
+    arm: both

plugins/bugs/evals/sc-planted-bug-dry-run/prompt.md

@@ -0,0 +1,10 @@
+---
+description: "A dry-run targeted scan of a one-file fixture with a planted boundary defect (> where the comment says or more) dispatches a hunter and a separate gate and reports a verified finding on discountRate with an evidence label and a path:line, editing nothing"
+tags: [bugs, scan, outcome, scaffold]
+max_turns: 30
+timeout_seconds: 900
+allowed_tools: [Skill, Bash, Read, Glob, Grep, Agent, Write, Edit]
+expected_outcome: "At least two Agent dispatches (a hunter and a separate verification gate). The report holds a verified finding whose fix location or heading names discountRate, labeled reproduced or verified-by-reading, citing src/discount.js:<line>. No Write or Edit call."
+---
+
+/bugs:scan src/discount.js --dry-run

plugins/bugs/evals/sc-planted-bug-dry-run/samples/evidence-label.json

@@ -0,0 +1,29 @@
+{
+  "pass": [
+    {
+      "answer": "Targeted run, effort high, small scope (1 file), 2 lenses; hunters and gates on the session model. Candidates: 2 raw, 1 after triage, 1 gated.\n\n## Finding 1: `discountRate` gives no 10% discount at exactly 10000 cents\n\n**Severity**: medium\n**Suggested fix location**: `src/discount.js` `discountRate` (no patch)\n**Evidence**: verified-by-reading\n**Lens**: lens-2 boundary/edge-case\n**Scope**: in-lane\n\n### Steps to reproduce\n\n1. Call `discountRate(10000)`.\n\n### Expected behavior\n\n0.1, per the comment: 10000 cents or more get 10% off.\n\n### Actual behavior\n\n0.05.\n\n### Severity justification\n\nCustomers at exactly $100.00 are overcharged by 5%.\n\n### Evidence\n\n`src/discount.js:4`\n\n```js\n  if (totalCents > 10000) return 0.1;\n```\n\n## Refuted candidates\n\nNone — every candidate this run survived the gate.\n\n## Candidates not gated\n\nNone.\n\n## Side observations\n\nNone.\n\n*No scan cursor: targeted run, rotation not advanced.*"
+    },
+    {
+      "answer": "**Needs you**: none.\n\n## 1. discountRate excludes the $100 boundary\n\n**Suggested fix location:** src/discount.js, discountRate\n**Evidence:** `reproduced` (ran the boundary input through the function by reading; 10000 → 0.05)\nLocation: discount.js line 4"
+    },
+    {
+      "answer": "**Evidence:** verified by reading"
+    },
+    {
+      "answer": "| Evidence | reproduced |",
+      "why": "table label"
+    }
+  ],
+  "fail": [
+    {
+      "answer": "*No verified findings.*\n\n**Lane**: src/discount.js (1 file read, 1 candidate, all refuted)\n\n## Refuted candidates\n\n| # | Candidate | Lens | Location | Refuting argument |\n|---|---|---|---|---|\n| 1 | discountRate boundary at 10000 | lens-2 | `src/discount.js:4` | Could not construct a failing input. |\n\n## Candidates not gated\n\nNone.\n\n## Side observations\n\nNone."
+    },
+    {
+      "answer": "Evidence labels apply only to survivors; the gate could not reproduce anything."
+    },
+    {
+      "answer": "The hunter flagged it, but no evidence reproduced it, so the gate dropped it.",
+      "why": "prose, no label"
+    }
+  ]
+}