feat(planning): interview asks intent and quality requirements by quality bar

.changes/planning/feat-8333-interview-intent-and-quality-requirements-4c7d2a91.md

@@ -0,0 +1,7 @@
+---
+bump: minor
+---
+
+### Added
+
+- `/planning:interview`: in an engineering session the interview reads the consumer's standards before any bundled default, asks the intent question (is a second variant known, a category, or speculative; what the existing structure does; what upstream context says), and prompts quality requirements (NFRs) from a new prompt list, each scaled by the quality bar. The Brief records `Quality bar:` and `Intent:` lines under Captured assumptions, each quality requirement as a `Quality requirement:` line under Constraints, and `Overrides bundled default <principle> per <standards file>` when a consumer standard wins. Unattended, only the user's own words settle the intent question; otherwise it is recorded `blocked`, arbiter USER-RESERVED, and the Brief is still written. Upstream context is recorded as a short summary, with private titles and URLs left out of public or unknown-visibility repositories.

plugins/planning/evals/interview-consumer-standard-override/case.yaml

@@ -0,0 +1,4 @@
+schema_version: "1.1"
+name: interview-consumer-standard-override
+context:
+  scaffold_script: scaffold.sh

plugins/planning/evals/interview-consumer-standard-override/graders/emits-override-line.md

@@ -0,0 +1,5 @@
+---
+type: regex
+pattern: '(?:^|\n)[-*]?\s*`?Overrides bundled default\W{0,4}[A-Za-z][^\n]{0,160}?\W{0,4}per\W{0,4}(?:\./)?docs/standards/integrations\.md'
+target: { source: file, path: .work/address-validation/PLAN.md }
+---

plugins/planning/evals/interview-consumer-standard-override/graders/reads-standard.md

@@ -0,0 +1,5 @@
+---
+type: regex
+target: trace
+pattern: '"name":"Read","input":\{"file_path":"[^"]*docs/standards/integrations\.md"|"name":"Bash","input":\{"command":"(?=(?:[^"\\]|\\.)*\b(?:cat|sed|head|tail)\b)(?:[^"\\]|\\.)*docs/standards/integrations\.md'
+---

plugins/planning/evals/interview-consumer-standard-override/prompt.md

@@ -0,0 +1,19 @@
+---
+description: "The repository's docs/standards/integrations.md puts every external service behind an interface from its first implementation; the user says no second provider is coming: the interview reads that standard, applies it over the bundled wait-for-repetition default, and records the override line naming the standards file"
+tags: [interview, standards, override, engineering-defaults]
+runs: 3
+max_turns: 30
+timeout_seconds: 900
+allowed_tools: [Read, Glob, Grep, Skill, Bash, Write, Edit]
+expected_outcome: "The interview reads docs/standards/integrations.md, and .work/address-validation/PLAN.md carries 'Overrides bundled default <principle> per docs/standards/integrations.md', with the address-validation interface required from the first implementation despite the speculative second variant"
+---
+
+/planning:interview lock
+
+This run is unattended: nobody will answer questions, so record anything only I can decide as blocked and keep going.
+
+Lock the Brief. Use the topic slug `address-validation`.
+
+Goal: checkout catches an undeliverable delivery address before payment, using the Loqate API. This goes to production. Loqate is the only provider and nothing says another is coming.
+
+Acceptance: an address Loqate marks undeliverable keeps the Pay button disabled and shows a message. Undeliverable means Loqate returns an unverified result; partial and ambiguous matches let checkout proceed. The Pay button lives in our separate web client, which renders a `canPay` flag and a message that checkout in this repo returns. If Loqate cannot answer (an error, an outage or a timeout), checkout fails open: Pay stays enabled.

plugins/planning/evals/interview-consumer-standard-override/samples/emits-override-line.json

@@ -0,0 +1,38 @@
+{
+  "pass": [
+    {
+      "answer": "### Captured assumptions\n- Quality bar: production\n- Intent: second variant = speculative; existing structure: payments sit behind a gateway port; upstream: none\n- Overrides bundled default Reuse first per docs/standards/integrations.md\n",
+      "why": "the exact line"
+    },
+    {
+      "answer": "### Captured assumptions\n- `Overrides bundled default YAGNI cheap-now test per docs/standards/integrations.md`\n",
+      "why": "code-span formatting and another principle name"
+    },
+    {
+      "answer": "### Captured assumptions\n- Overrides bundled default \"contain now, abstract later\" (variation-points, for a speculative second variant) per docs/standards/integrations.md: every external service gets a port from its first implementation\n",
+      "why": "a long principle, from a real run"
+    },
+    {
+      "answer": "### Captured assumptions\n- Overrides bundled default \"contain now, abstract later\" (planning plugin `reference/variation-points.md`) per `docs/standards/integrations.md`: a port from the first implementation\n",
+      "why": "a long principle and a code-span path, from a real run"
+    }
+  ],
+  "fail": [
+    {
+      "answer": "",
+      "why": "no Brief written"
+    },
+    {
+      "answer": "### Constraints\n- Address validation sits behind an interface per docs/standards/integrations.md\n",
+      "why": "applies the standard without the override line"
+    },
+    {
+      "answer": "### Captured assumptions\n- Overrides bundled default Reuse first per docs/standards/README.md\n",
+      "why": "names the index, not the standards file"
+    },
+    {
+      "answer": "### Captured assumptions\n- No line here overrides bundled default Reuse first per docs/standards/integrations.md\n",
+      "why": "a negated mention, not the line"
+    }
+  ]
+}

plugins/planning/evals/interview-consumer-standard-override/samples/reads-standard.json

@@ -0,0 +1,38 @@
+{
+  "pass": [
+    {
+      "answer": "{\"type\":\"assistant\",\"message\":{\"content\":[{\"type\":\"tool_use\",\"id\":\"toolu_x\",\"name\":\"Read\",\"input\":{\"file_path\":\"/tmp/w/docs/standards/README.md\"}}]}}\n{\"type\":\"assistant\",\"message\":{\"content\":[{\"type\":\"tool_use\",\"id\":\"toolu_x\",\"name\":\"Read\",\"input\":{\"file_path\":\"/tmp/w/docs/standards/integrations.md\"}}]}}\n",
+      "why": "Read tool on the matched file"
+    },
+    {
+      "answer": "{\"type\":\"assistant\",\"message\":{\"content\":[{\"type\":\"tool_use\",\"id\":\"toolu_x\",\"name\":\"Bash\",\"input\":{\"command\":\"cat docs/standards/README.md docs/standards/integrations.md\",\"description\":\"Read standards\"}}]}}\n",
+      "why": "cat in Bash"
+    },
+    {
+      "answer": "{\"type\":\"assistant\",\"message\":{\"content\":[{\"type\":\"tool_use\",\"id\":\"toolu_x\",\"name\":\"Bash\",\"input\":{\"command\":\"for f in docs/standards/README.md docs/standards/integrations.md; do echo \\\"=== $f\\\"; cat \\\"$f\\\"; done\",\"description\":\"Read files\"}}]}}\n",
+      "why": "loop with cat after the path"
+    },
+    {
+      "answer": "{\"type\":\"assistant\",\"message\":{\"content\":[{\"type\":\"tool_use\",\"id\":\"toolu_x\",\"name\":\"Bash\",\"input\":{\"command\":\"sed -n 1,40p docs/standards/integrations.md\"}}]}}\n",
+      "why": "sed"
+    }
+  ],
+  "fail": [
+    {
+      "answer": "",
+      "why": "empty trace"
+    },
+    {
+      "answer": "{\"type\":\"assistant\",\"message\":{\"content\":[{\"type\":\"tool_use\",\"id\":\"toolu_x\",\"name\":\"Read\",\"input\":{\"file_path\":\"/tmp/w/docs/standards/README.md\"}}]}}\n{\"type\":\"user\",\"message\":{\"content\":[{\"type\":\"tool_result\",\"tool_use_id\":\"toolu_x\",\"content\":\"| integrations | ... | integrations.md |\"}]}}\n",
+      "why": "reads only the index; the matched file name appears only in a result"
+    },
+    {
+      "answer": "{\"type\":\"assistant\",\"message\":{\"content\":[{\"type\":\"tool_use\",\"id\":\"toolu_x\",\"name\":\"Bash\",\"input\":{\"command\":\"ls docs/standards/integrations.md\"}}]}}\n",
+      "why": "lists the file without reading it"
+    },
+    {
+      "answer": "{\"type\":\"assistant\",\"message\":{\"content\":[{\"type\":\"tool_use\",\"id\":\"toolu_x\",\"name\":\"Read\",\"input\":{\"file_path\":\"/tmp/w/src/integrations/ports/payment-gateway.js\"}}]}}\n",
+      "why": "another file under integrations"
+    }
+  ]
+}

plugins/planning/evals/interview-consumer-standard-override/scaffold.sh

@@ -0,0 +1,55 @@
+#!/usr/bin/env bash
+# A checkout app whose standards index routes integration work to integrations.md. That file says
+# every external service goes behind an interface from its first implementation, with no wait for a
+# second provider. The bundled reuse default waits for real repetition when a second variant is
+# speculative, so the consumer standard wins and the Brief names the default it overrode.
+set -euo pipefail
+
+git init -q -b main
+git config user.name eval
+git config user.email eval@example.invalid
+mkdir -p docs/standards src/checkout src/integrations/ports
+
+cat >docs/standards/README.md <<'MD'
+---
+standards-contract: 1.1.0
+---
+
+# Standards index
+
+| Surface | Applies when | File |
+|---|---|---|
+| javascript | **/*.js; JavaScript design or review tasks | javascript.md |
+| integrations | any external service integration, provider choice, or design of code under src/integrations/ | integrations.md |
+MD
+
+cat >docs/standards/javascript.md <<'MD'
+# JavaScript
+
+Use ES modules and test with `node:test`. Each feature area lives under `src/<area>/`.
+MD
+
+cat >docs/standards/integrations.md <<'MD'
+# Integrations
+
+Every external service sits behind an interface in `src/integrations/ports/` from its first
+implementation, with the vendor client as one adapter beside it. Do not wait for a second provider
+before adding the interface: our clients swap vendors per contract.
+MD
+
+cat >src/integrations/ports/payment-gateway.js <<'JS'
+/** @typedef {{ charge(amount: number, currency: string): Promise<{ id: string }> }} PaymentGateway */
+export {};
+JS
+
+cat >src/checkout/pay.js <<'JS'
+/** @param {import('../integrations/ports/payment-gateway.js').PaymentGateway} gateway */
+export async function pay(order, gateway) {
+  return gateway.charge(order.total, order.currency);
+}
+JS
+
+printf '{ "name": "shop", "type": "module", "scripts": { "test": "node --test" } }\n' >package.json
+
+git add .
+git commit -q -m "feat: checkout pay step behind a payment gateway port"

plugins/planning/evals/interview-intent-asked-recorded/case.yaml

@@ -0,0 +1,4 @@
+schema_version: "1.1"
+name: interview-intent-asked-recorded
+context:
+  scaffold_script: scaffold.sh

plugins/planning/evals/interview-intent-asked-recorded/graders/asks-intent.md

@@ -0,0 +1,14 @@
+---
+type: llm
+---
+
+Judge only the questions in the reply: lines that ask the user something. A statement is not a
+question.
+
+PASS only if one question asks the user whether a second variant is expected: another
+address-validation provider besides Loqate, or validation that varies per client, country or
+tenant. The question may use the words known, category or speculative, or ask whether to build it
+specific now or generalize it, and it carries a recommended answer.
+
+FAIL if no such question is asked, if the reply states an answer to it (for example "no other
+provider is coming") without asking, or if the reply asks no question at all.

plugins/planning/evals/interview-intent-asked-recorded/graders/asks-second-variant.md

@@ -0,0 +1,5 @@
+---
+type: regex
+pattern: '\b(?:is|are|will|would|do|does|should|could|might|can|has|have|any|what|which|how)\b(?:[^\n.!?]|\.(?=\S)){0,120}\b(?:second|another|other|more|additional|different|multiple)\b(?:[^\n.!?]|\.(?=\S)){0,120}\b(?:providers?|vendors?|variants?|country|countries|clients?|tenants?)\b(?:[^\n.!?]|\.(?=\S)){0,200}\?'
+flags: i
+---

plugins/planning/evals/interview-intent-asked-recorded/graders/registers-intent-row.md

@@ -0,0 +1,6 @@
+---
+type: regex
+pattern: '(?:^|\n)- Q\d+ \| open \| round \d+ \|[^|\n]*\b(?:second|another|other|variants?|providers?|generali[sz]e|specific)\b'
+flags: i
+target: { source: file, path: .work/address-validation/interview-checklist.md }
+---

plugins/planning/evals/interview-intent-asked-recorded/prompt.md

@@ -0,0 +1,11 @@
+---
+description: "Engineering task with one external provider and nothing said about others: the interview asks the intent question (is a second variant known, a category, or speculative) with a recommendation, and registers it in the ledger at ask-time"
+tags: [interview, intent, engineering-defaults]
+runs: 3
+max_turns: 30
+timeout_seconds: 900
+allowed_tools: [Read, Glob, Grep, Skill, Bash, Write, Edit]
+expected_outcome: "Round 1 asks whether a second address-validation provider or per-client variation is known, expected as a category, or speculative, with a recommendation, and the ledger .work/address-validation/interview-checklist.md carries that question as an open register row"
+---
+
+/planning:interview Add delivery-address validation to checkout using the Loqate API, so an undeliverable address is caught before payment. This goes to production. Use the topic slug `address-validation`.

plugins/planning/evals/interview-intent-asked-recorded/samples/asks-second-variant.json

@@ -0,0 +1,30 @@
+{
+  "pass": [
+    {
+      "answer": "Q2: Is a second address-validation provider known, a category you expect more of, or speculative?\nMy recommendation: **speculative** - nothing in the repo or request names another provider.",
+      "why": "asks the repeat-or-vary part in the known / category / speculative vocabulary"
+    },
+    {
+      "answer": "Q3: Will other clients need a different validation vendor, or is Loqate the only one?",
+      "why": "asks about per-client variation"
+    }
+  ],
+  "fail": [
+    {
+      "answer": "",
+      "why": "empty reply"
+    },
+    {
+      "answer": "Loqate is the only provider, so I will keep the integration specific. No other provider is planned.",
+      "why": "states the answer instead of asking it"
+    },
+    {
+      "answer": "Q1: Should the Pay button stay disabled while the lookup runs?",
+      "why": "a question, but not the intent question"
+    },
+    {
+      "answer": "No other provider is coming, so should the Pay button stay disabled?",
+      "why": "states the intent answer, then asks a different question"
+    }
+  ]
+}

plugins/planning/evals/interview-intent-asked-recorded/samples/registers-intent-row.json

@@ -0,0 +1,22 @@
+{
+  "pass": [
+    {
+      "answer": "## Open-question register\n\n- Q1 | open | round 1 | What should checkout show for an undeliverable address? |\n- Q2 | open | round 1 | Is a second address-validation provider known, a category, or speculative? |\n",
+      "why": "the intent question is registered open at ask-time"
+    }
+  ],
+  "fail": [
+    {
+      "answer": "",
+      "why": "no ledger written"
+    },
+    {
+      "answer": "## Open-question register\n\n- Q1 | open | round 1 | What should checkout show for an undeliverable address? |\n",
+      "why": "the register has no intent row"
+    },
+    {
+      "answer": "## Open-question register\n\n- Q1 | answered | round 1 | Is a second provider coming? | auto-assumed: no\n",
+      "why": "the intent row was resolved without the user's answer, not left open"
+    }
+  ]
+}

plugins/planning/evals/interview-intent-asked-recorded/scaffold.sh

@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+# A checkout app with one payment step and a standards index. Nothing in the repository or the
+# prompt says whether a second address-validation provider, or per-client variation, is coming,
+# so the interview must ask the intent question rather than assume an answer.
+set -euo pipefail
+
+git init -q -b main
+git config user.name eval
+git config user.email eval@example.invalid
+mkdir -p docs/standards src/checkout
+
+cat >docs/standards/README.md <<'MD'
+---
+standards-contract: 1.1.0
+---
+
+# Standards index
+
+| Surface | Applies when | File |
+|---|---|---|
+| javascript | **/*.js; JavaScript design or review tasks | javascript.md |
+MD
+
+cat >docs/standards/javascript.md <<'MD'
+# JavaScript
+
+Use ES modules and test with `node:test`. Each feature area lives under `src/<area>/`.
+MD
+
+cat >src/checkout/pay.js <<'JS'
+export async function pay(order, paymentClient) {
+  return paymentClient.charge(order.total, order.currency);
+}
+JS
+
+printf '{ "name": "shop", "type": "module", "scripts": { "test": "node --test" } }\n' >package.json
+
+git add .
+git commit -q -m "feat: checkout pay step"

plugins/planning/evals/interview-intent-prototype-one-line/case.yaml

@@ -0,0 +1,4 @@
+schema_version: "1.1"
+name: interview-intent-prototype-one-line
+context:
+  scaffold_script: scaffold.sh

plugins/planning/evals/interview-intent-prototype-one-line/graders/asks-a-round.md

@@ -0,0 +1,5 @@
+---
+type: regex
+pattern: '(?:^|\n)[ \t]*(?:#{1,6}[ \t]*)?(?:\*\*)?(?:Q1\b|1[.)])[^?]{0,400}\?'
+flags: i
+---

plugins/planning/evals/interview-intent-prototype-one-line/graders/no-quality-requirement-questions.md

@@ -0,0 +1,6 @@
+---
+type: regex
+pattern: '(?:^|\n)[^\n]*\b(?:latency|p9\d|response time|throughput|uptime|availability|SLA|SLO|scal(?:e|es|ing|ability)|monitor(?:ing)?|observability|alert(?:s|ing)?|encrypt\w*|maintainability)\b[^\n]*\?'
+flags: i
+match: not_contains
+---

plugins/planning/evals/interview-intent-prototype-one-line/graders/one-line-intent.md

@@ -0,0 +1,20 @@
+---
+type: llm
+---
+
+The user asked for a throwaway prototype that reads Shopify orders. Judge the reply.
+
+PASS only if all of these hold:
+
+1. The reply asks whether other stores, platforms or variants are expected (or whether to keep the
+   code specific to Shopify) as one short line, with no list of alternatives under it. A one-line
+   confirmation such as "Shopify only, kept specific?" counts; a recommendation line under it is
+   allowed.
+2. No question asks the user to set or confirm a quality requirement: performance or latency,
+   uptime or availability, security, scaling, monitoring or observability, or maintainability
+   targets.
+3. The reply names quality requirements (or non-functional requirements, or production concerns
+   such as performance, security and monitoring) as deferred, skipped or out of scope for the
+   prototype. Naming them in a statement is not a question and does not fail check 2.
+
+FAIL if any of the three does not hold.

plugins/planning/evals/interview-intent-prototype-one-line/prompt.md

@@ -0,0 +1,11 @@
+---
+description: "Stated throwaway prototype that reads one external store: the intent question is one line, and no quality requirement (performance, availability, security, scaling, monitoring) is asked; they are named as deferred"
+tags: [interview, intent, quality-bar, prototype, engineering-defaults]
+runs: 3
+max_turns: 20
+timeout_seconds: 600
+allowed_tools: [Read, Glob, Grep, Skill, Bash, Write, Edit]
+expected_outcome: "Round 1 asks one one-line question on whether other stores or platforms are expected (or confirms keeping it specific), asks no quality-requirement question, and names quality requirements as deferred for a prototype"
+---
+
+/planning:interview I want a throwaway prototype that pulls last month's Shopify orders into a CSV, so I can show finance a report idea on Friday. It will never ship and only I will run it. Use the topic slug `orders-csv`.

plugins/planning/evals/interview-intent-prototype-one-line/samples/asks-a-round.json

@@ -0,0 +1,18 @@
+{
+  "pass": [
+    {
+      "answer": "Here is what I see: an empty reports package.\n\nQ1: Which order fields should the CSV carry?\nMy recommendation: **id, date, total**.",
+      "why": "a numbered question"
+    }
+  ],
+  "fail": [
+    {
+      "answer": "",
+      "why": "empty reply"
+    },
+    {
+      "answer": "I wrote the Brief and locked it.",
+      "why": "no question asked"
+    }
+  ]
+}