diff --git a/.gitignore b/.gitignore index cf8aa137..b8244582 100644 --- a/.gitignore +++ b/.gitignore @@ -71,3 +71,6 @@ coverage/ !.cursor/skills .agents/* !.agents/skills + +# Superpowers SDD scratch +.superpowers/ diff --git a/CHANGELOG.md b/CHANGELOG.md index 8aec2b7c..56185f55 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,90 @@ All notable changes to `vinta-ai-workflows` are documented in this file. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); the project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [0.7.0] — YYYY-MM-DD + +### Added + +- **Two new opt-in foundation skills for browser QA: `dev-desk-check` and + `qa-review`.** Every QA path in this package was Playwright-shaped: + `create-qa-use-cases` generates a human checklist that nothing executes, and + `run_options.run_e2e` defaults off because the suite is slow. These two drive + a real browser through the app as a user, judge what renders, and produce + evidence someone who wasn't there can trust. + + `dev-desk-check` is the engineer path: it takes a plan from the branch diff (vs + the PR's real base, resolved per `project.code_host` — "no PR" and "the CLI + failed" are different answers, and only the first falls back to the default + branch), from ids in the project's QA use-cases doc, or from a ticket. It + preflights services → frontend → backend, enforces the project's + per-environment write policy before the browser opens, keeps an incremental + write ledger with a reversal recorded per entry, and writes a report plus a + PR `Q.A.` section. Template-rendered from + `vinta-derive-skills/resources/dev-desk-check-template.md`. It is **not** wired + into `implement-plan`'s phase gate — a browser pass per phase is too slow, + the same reasoning that keeps `run_e2e` off — and it hand-rolls no browser + tooling: with no browser skill or MCP in the harness it says so and stops. + + `qa-review` is the portable path for product managers and designers: the same + walk and the same report shape with git, diffs, dev servers, service probes, + PR edits and database access stripped out. It renders **self-contained** — no + relative links, no repo-dependent commands — so an engineer hands over the + single `SKILL.md` and it works with no checkout. Stable facts are inlined + (environment addresses, account role, design source, who to ask when + blocked); the QA use-cases doc deliberately is not, because a snapshot of a + checklist the repo keeps updating is worse than no snapshot. From + `vinta-derive-skills/resources/qa-review-template.md`. + +- **New `foundation_skills` enum entries `dev-desk-check` and `qa-review`** in + `schemas/vinta-ai-workflows-config.v1.schema.json`. They are **independent** + opt-ins — a project can take the engineer path alone, the review path alone, + or both — and both render from one shared config block. + +- **New `skills.dev-desk-check` config block**, read by both skills. Guard rails + are data, not prose: `environments.` carries `base_url`, optional + `api_url` / `api_health_path`, an opt-in-by-absence `frontend_start` (no key + means probe and stop — a second dev server on an occupied port silently + invalidates a run), a required `writes` policy (`free` / `confirm` / + `forbidden`), a `data_sensitivity` level (`none` / `client` / `phi`), and an + optional `edge_proxy` that suppresses the `curl` preflight for environments + behind a WAF that answers non-browser traffic with a 403. Alongside it: + optional `services[]` (read-only time-bounded probes, `required` verdict, + local-only `start`), `auth` (manual login only — the agent can never type a + password — with account roles and **pointers to** where credentials live, + never credentials), `feature_flags`, `design_source`, `use_cases_path`, + `report_dir` and `escalation_contact`. Absent optional blocks strip their + region from the rendered body rather than shipping it empty. + + `data_sensitivity` drives evidence handling, not just writes: `client` keeps + screenshots in the gitignored `report_dir` and off PRs and design tools; + `phi` additionally restricts the report and the write ledger to opaque ids + and removes the shareable-Artifact deliverable entirely, because publishing + to hosted storage would move records outside the managed platform's access + controls and audit logging. + +- **New `run_options.dev-desk-check` block** — `screenshots`, `scope` + (`diff` / `use-cases` / `description`), `browser`, `dry_run`. Read by both + skills; `qa-review` ignores what has no meaning without a checkout. + +- **Two new `vinta-bootstrap-ai-tools` interview questions** in the **Optional + foundation skills** group (now nine, up from seven), plus one shared block of + config follow-ups asked once when either skill is enabled: environments, + per-environment write policy and data sensitivity, edge proxies, backing + services, auth account pointers, feature-flag system, design source, + escalation contact, and per-run defaults. + +- **Sixteen new `vinta-derive-skills` placeholders** (`{{QA_*}}`) documented in + its placeholder table, plus a `qa-review`-specific **portability check** run + before the render is saved: it greps the output for relative links and + repo-dependent commands, because a config value spliced into a block can + break portability even when the template is clean. + +- **`vinta-update-project-skills` now flags `qa-review` for re-render on a + config change**, not only on a template change. Its body inlines facts its + holder cannot look up, so a changed `skills.dev-desk-check` makes the shipped + copy stale; the render carries a generation date and source commit to compare + against. + ## [0.6.1] — 2026-08-17 ### Fixed diff --git a/README.md b/README.md index f04dedb1..e6323af0 100644 --- a/README.md +++ b/README.md @@ -208,7 +208,7 @@ Standalone re-invocation works at every step — kick `create-spec` again when t Quick inventory of what `vinta-bootstrap-ai-tools` writes into your repo's `ai-tools/` layout — plus `AGENTS.md` at the repo root, the one artifact that lives outside it — and exposes via per-vendor wiring at `.claude/skills/`, `.agents/skills/`, etc. Two disclaimers up front: -> **Optional foundation skills are gated by the bootstrap interview.** `systematic-debugging`, `add-e2e-test`, `add-env-var`, `add-one-off-script`, `prepare-worktree`, `thermo-nuclear-code-quality-review`, and `handoff-to-client` ship **only** if the user says yes during `vinta-bootstrap-ai-tools` Step 0 (or later opts in via `vinta-sync-ai-tools`). They are not in every bootstrapped project. The opt-in is recorded in `.vinta-ai-workflows.yaml` under `foundation_skills.*.enabled` and is sticky across syncs. +> **Optional foundation skills are gated by the bootstrap interview.** `systematic-debugging`, `add-e2e-test`, `add-env-var`, `add-one-off-script`, `prepare-worktree`, `thermo-nuclear-code-quality-review`, `dev-desk-check`, `qa-review`, and `handoff-to-client` ship **only** if the user says yes during `vinta-bootstrap-ai-tools` Step 0 (or later opts in via `vinta-sync-ai-tools`). They are not in every bootstrapped project. The opt-in is recorded in `.vinta-ai-workflows.yaml` under `foundation_skills.*.enabled` and is sticky across syncs. > > **Stack-specific skills and sub-agents are user-supplied.** This package ships **detection signals + category lists** per stack ([resources/stacks/](skills/vinta-bootstrap-ai-tools/resources/stacks/) — Django, Medplum, Next.js App Router, Python package, TypeScript monorepo), not the bodies. When a stack is detected, the orchestrator asks the user for a path / URL to their team's existing template. If no template exists, the category is recorded as a gap in the final summary, not auto-drafted. Bodies are project- and team-specific; one shared template doesn't fit every team's conventions. @@ -249,6 +249,8 @@ Land at `ai-tools/skills//SKILL.md`. Always-on unless flagged optional. | [`prepare-worktree`](skills/vinta-derive-skills/resources/foundation-skills/prepare-worktree/SKILL.md) | **opt-in** | Provision a fully-runnable git worktree for parallel plan work — reads the plan + `.gitignore` + manifests + env / docker config and decides per ignored path whether to **symlink / copy / fork** (dev DB, env files, compose project name). Ships an OS-level write-guard (`sandbox-run.sh` + a Claude Code `PreToolUse` hook) so stray writes to the main checkout fail at the kernel layer. `implement-plan` Step 0 question (c) opts a run into it. | | [`thermo-nuclear-code-quality-review`](skills/vinta-derive-skills/resources/foundation-skills/thermo-nuclear-code-quality-review/SKILL.md) | **opt-in** | Deliberately harsh structural-maintainability audit of a diff — abstraction quality, giant files, spaghetti-condition growth — hunting "code-judo" reframes that collapse whole branches / helpers / modes / layers. Read-only; hands each fix to `fixer`. `review-phase` Layer 3 runs a condensed lens every phase and escalates to this full audit on core-architecture / ~1000-line / structurally-smelly phases. | | `handoff-to-client` | **opt-in** | (API-only repos) Generate a self-contained API-change handoff doc for the client teams consuming the repo's API — every endpoint / operation added / changed / deprecated / removed vs the default branch, with request/response shapes, auth + error changes, breaking-change flags, one example per operation, and per-platform migration notes. Template-rendered from `skills.handoff-to-client.*` config (client platforms, API style, spec path). | +| `dev-desk-check` | **opt-in** | Drive a real browser through the app as a user and report what actually renders — the manual pass that complements automated e2e. Plan source is the branch diff, ids from `QA_USE_CASES.md`, or a ticket. Preflights services / frontend / backend per environment, enforces the project's per-environment write policy and data-sensitivity rules before opening the browser, keeps an incremental write ledger with a reversal per entry, and writes an evidence-backed report plus a PR `Q.A.` section. Template-rendered from `skills.dev-desk-check.*` config. | +| `qa-review` | **opt-in** | The same browser walk and report shape for product managers and designers, with git, diffs, dev servers, service probes, PR edits and database access stripped out. Rendered **self-contained** — no relative links, no repo-dependent commands — so an engineer can hand over the single `SKILL.md` and it works with no checkout. Reads the same `skills.dev-desk-check.*` config block; opt-in independently of `dev-desk-check`. | #### `write-unit-test` in depth diff --git a/schemas/vinta-ai-workflows-config.v1.schema.json b/schemas/vinta-ai-workflows-config.v1.schema.json index 2abde485..723e0803 100644 --- a/schemas/vinta-ai-workflows-config.v1.schema.json +++ b/schemas/vinta-ai-workflows-config.v1.schema.json @@ -237,6 +237,16 @@ "description": "Write durable unit tests for a unit of behavior — mock only genuine externals, assert the full expected output (not counts/truthiness), clean up created data or run inside a rolled-back transaction, never hit an external service that can't run in dev/test, keep test logic literal, and decouple from implementation internals. Template-rendered (bucket B) from `vinta-derive-skills/resources/write-unit-test-template.md`, with best-practice packs copied verbatim into the skill's `resources/packs/` on two independent axes — one runner pack (`runners/`: pytest / vitest / jest) selected by the detected test framework, plus a stack pack per matched stack (`stacks/`: django / fastapi / flask / medplum / react / nextjs / tanstack-start / react-router / prisma / python-package / typescript-package) so framework-specific advice ships only when that framework is present. Default-on: bootstrap sets `enabled` whenever `vinta-analyze-codebase` detects a unit-test framework, `disabled` when none is found (there's nothing to render against). When `enabled`, project-level test conventions live under `skills.write-unit-test` (test style, data setup, assertion style, DB isolation, inferred extra conventions).", "type": "string", "enum": ["enabled", "disabled"] + }, + "dev-desk-check": { + "description": "Drive a real browser through the app as a user, judge what renders, and produce evidence someone who wasn't there can trust — the manual counterpart to `add-e2e-test`'s automation. Plan source is the branch diff, ids from the project's QA use-cases doc, or a ticket. Template-rendered (bucket B) from `vinta-derive-skills/resources/dev-desk-check-template.md` and opt-in (bucket C). Invoked deliberately — never wired into `implement-plan`'s phase gate, because a browser pass per phase is too slow (the same reasoning that keeps `run_options.run_e2e` off). When `enabled`, project-level configuration lives under `skills.dev-desk-check` — the same block `qa-review` renders from.", + "type": "string", + "enum": ["enabled", "disabled"] + }, + "qa-review": { + "description": "Portable QA pass for product managers and designers — the same browser walk and report shape as `dev-desk-check` with git, diffs, dev servers, service probes, PR edits and database access stripped out. Template-rendered (bucket B) from `vinta-derive-skills/resources/qa-review-template.md` and opt-in (bucket C), rendered **self-contained**: no relative links and no repo-dependent commands, so the single `SKILL.md` works when dropped at `~/.claude/skills/qa-review/SKILL.md` by someone with no checkout. Reads the same `skills.dev-desk-check` config block; this enum and `foundation_skills.dev-desk-check` are independent opt-ins, so a project can take the engineer path alone, the review path alone, or both.", + "type": "string", + "enum": ["enabled", "disabled"] } } }, @@ -341,6 +351,34 @@ "default": true } } + }, + "dev-desk-check": { + "description": "Per-run defaults for `dev-desk-check` and `qa-review`. Both read this one block; `qa-review` ignores the options that have no meaning without a checkout. The skill still asks per run — these are the suggested defaults shown.", + "type": "object", + "additionalProperties": false, + "properties": { + "screenshots": { + "description": "Capture a screenshot per distinct rendered state during the walk. Default `true` — a report with no evidence is hard for someone who wasn't there to trust. Screenshot handling is still bounded by the environment's `data_sensitivity`.", + "type": "boolean", + "default": true + }, + "scope": { + "description": "Default plan source for Phase 1. `diff` maps the branch diff vs the PR's real base to reachable surfaces; `use-cases` walks ids from the project's QA use-cases doc (the strongest source — the document already states the steps and the expected outcome); `description` works from a ticket or a plain description. `qa-review` supports `use-cases` and `description` only.", + "type": "string", + "enum": ["diff", "use-cases", "description"], + "default": "diff" + }, + "browser": { + "description": "Preferred browser-driving tool, named as the harness exposes it (e.g. `playwright-mcp`, `chrome-devtools`, the runtime's built-in browser). Free-form on purpose — this package ships no browser tooling and hardcoding a tool name goes stale. When the named tool is unavailable at runtime the skill falls back to whatever the harness does expose, and refuses to run when nothing does.", + "type": "string", + "minLength": 1 + }, + "dry_run": { + "description": "Plan and preflight only — enumerate the surfaces to walk and the writes the plan would require, print the write-authorization block, then stop before opening the browser. Default `false`.", + "type": "boolean", + "default": false + } + } } } }, @@ -522,6 +560,184 @@ "default": [] } } + }, + "dev-desk-check": { + "description": "Configuration for the `dev-desk-check` **and** `qa-review` foundation skills — both render from this one block. Required when either `foundation_skills.dev-desk-check` or `foundation_skills.qa-review` is `enabled`; ignored otherwise. Guard rails are data, not prose: a project declares its policy per environment and the rendered skill enforces exactly what it was given. **Never put a credential here** — `auth.accounts[].where` is a pointer to where a secret lives, not the secret.", + "type": "object", + "required": ["environments"], + "additionalProperties": false, + "properties": { + "use_cases_path": { + "description": "Repo-relative path to the project's QA use-cases checklist, the plan source both skills prefer. Usually the `QA_USE_CASES.md` that `create-qa-use-cases` bootstraps. The `qa-review` render deliberately does NOT inline this document's contents — a snapshot of a checklist the repo keeps updating is worse than no snapshot; it asks the reviewer to paste the use case instead.", + "type": "string", + "minLength": 1, + "default": "QA_USE_CASES.md" + }, + "report_dir": { + "description": "Where QA reports and the incremental write ledger land, repo-relative. The default sits under the gitignored `.vinta-ai-workflows/`, which is what `data_sensitivity: client` and `phi` require — screenshots from those environments must not be committed, uploaded to a PR, or attached to a design tool.", + "type": "string", + "minLength": 1, + "default": ".vinta-ai-workflows/qa" + }, + "escalation_contact": { + "description": "Who a reviewer asks when a run is blocked — a channel, a rotation, a role (e.g. `#acme-eng on Slack`, `the on-call engineer`). Inlined into the `qa-review` render because its holder has no repo and no other way to find out. Omit and the render says `the engineer who gave you this file`.", + "type": "string", + "minLength": 1 + }, + "environments": { + "description": "Declared QA environments, keyed by the name the team uses (`local`, `staging`, `production`, `demo`, …). At least one is required. Each entry states its own write policy and data sensitivity — the rendered skill enforces per environment, never globally.", + "type": "object", + "minProperties": 1, + "additionalProperties": { + "type": "object", + "required": ["base_url", "writes"], + "additionalProperties": false, + "properties": { + "base_url": { + "description": "Where the app is reached in this environment. The browser opens here.", + "type": "string", + "minLength": 1 + }, + "api_url": { + "description": "Backend base URL, when the backend is preflighted separately from the frontend. Omit for environments where there is nothing to probe independently.", + "type": "string", + "minLength": 1 + }, + "api_health_path": { + "description": "Path appended to `api_url` for the backend preflight probe. Only meaningful when `api_url` is set and `edge_proxy` is absent.", + "type": "string", + "minLength": 1, + "default": "/health/" + }, + "frontend_start": { + "description": "Command that starts the frontend for this environment. **Opt-in by absence**: no key means the rendered skill probes `base_url` and stops rather than starting anything. A second dev server on an occupied port silently invalidates a whole run. In practice this is set for `local` only.", + "type": "string", + "minLength": 1 + }, + "writes": { + "description": "What the run may mutate in this environment. `free` — writes proceed, and the ledger is still kept. `confirm` — every write is enumerated and authorised before the browser opens, in a confirmation that names this environment; the relaxed production gate and the right default for anything prod-like. `forbidden` — the run refuses to write, states what would be required, records it, and lets the operator decide. A per-run `--writes=allow` flag may only relax *within* the declared policy, never upgrade past it. Setup writes (feature flags, seeds, migrations) stay propose-only under every value.", + "type": "string", + "enum": ["free", "confirm", "forbidden"] + }, + "data_sensitivity": { + "description": "What the records in this environment are, which drives evidence handling and not just writes. `none` — screenshots and report contents unrestricted, Artifact deliverable offered. `client` — screenshots stay in the gitignored `report_dir` and are never uploaded to a PR or attached to a design tool; the report describes the shape of records rather than transcribing them; the Artifact deliverable is offered only behind an explicit confirmation. `phi` — as `client`, plus the report and the write ledger carry **opaque ids only, never field values**, and the Artifact deliverable is **not offered at all**, because publishing to hosted storage would move records outside the managed platform's access controls and audit logging.", + "type": "string", + "enum": ["none", "client", "phi"], + "default": "none" + }, + "edge_proxy": { + "description": "Name of the edge proxy / WAF fronting this environment (e.g. `cloudflare`, `akamai`). When set, the rendered skill emits **no** `curl` preflight for this environment — such proxies answer non-browser traffic with a 403, and reading that as downtime kills a healthy run on evidence that never addressed the question. The browser is the preflight there.", + "type": "string", + "minLength": 1 + } + } + } + }, + "services": { + "description": "Backing services to probe before the frontend and backend checks, because a dependency that is down produces failures indistinguishable from product bugs. Omit entirely and the rendered skill's services section is stripped, not shipped empty.", + "type": "array", + "default": [], + "items": { + "type": "object", + "required": ["name", "probe"], + "additionalProperties": false, + "properties": { + "name": { "type": "string", "minLength": 1 }, + "probe": { + "description": "Health check for this service. **Must be read-only and time-bounded** — it runs before every QA pass and must not mutate state or hang (wrap in `timeout 5 …` when the tool has no timeout flag of its own). A probe that cannot run at all (binary missing) is reported as `unknown`, never as `down`.", + "type": "string", + "minLength": 1 + }, + "required": { + "description": "`true` fails the run: stop, name the service, offer the declared `start`. `false` degrades it: continue, and record in the report that the service was down — findings from that run carry an asterisk a reader needs to see.", + "type": "boolean", + "default": true + }, + "start": { + "description": "Command that brings the service up. Offered **only** in an environment named `local`, and only with the operator's approval. Elsewhere the rendered skill proposes it and waits.", + "type": "string", + "minLength": 1 + } + } + } + }, + "auth": { + "description": "How a QA run authenticates. There is no automated option: the agent can never type a password, in any environment.", + "type": "object", + "required": ["login"], + "additionalProperties": false, + "properties": { + "login": { + "description": "Only `manual` is supported. The rendered skill asks the operator to log in before it reaches the login form, in every environment. This is an enum with one value on purpose — so a project cannot express an automated login the skills would have to refuse.", + "type": "string", + "enum": ["manual"] + }, + "accounts": { + "description": "Account roles a QA run may need, described so the rendered skill can name the right one without knowing a secret.", + "type": "array", + "default": [], + "items": { + "type": "object", + "required": ["role"], + "additionalProperties": false, + "properties": { + "role": { + "description": "What this account is, in the team's words (e.g. `firm admin with docketing enabled`, `read-only auditor`).", + "type": "string", + "minLength": 1 + }, + "where": { + "description": "**A pointer to where the credential lives, never the credential** — e.g. `e2e/.env (gitignored)`, `1Password → Acme QA vault`. Keeps this file committable and matches least-privilege.", + "type": "string", + "minLength": 1 + } + } + } + } + } + }, + "feature_flags": { + "description": "How this project gates features, so a QA run can tell 'the feature is broken' from 'the flag is off'. Omit entirely, or set `system: none`, and the rendered skill's feature-flag section is stripped rather than shipping as dead advice.", + "type": "object", + "required": ["system"], + "additionalProperties": false, + "properties": { + "system": { + "description": "The flag system in use. `env` covers plain environment-variable gating.", + "type": "string", + "enum": ["waffle", "launchdarkly", "env", "none"] + }, + "check": { + "description": "Where to find which flags apply to which QA account (e.g. `e2e/README.md lists flags per QA user`). A pointer, not a flag list — flag lists go stale.", + "type": "string", + "minLength": 1 + } + } + }, + "design_source": { + "description": "What the design pass compares against. Omit entirely, or set `type: none`, and the design pass is skipped — the rendered skill then records the skip under *Not tested* with its reason, never silently drops it.", + "type": "object", + "required": ["type"], + "additionalProperties": false, + "properties": { + "type": { + "description": "`figma` — compare against the live node through whatever Figma MCP tooling the runtime exposes. `exports` — compare against static exports the designer supplied, which the report must label as exports (they can be older than the live file). `none` — skip the design pass.", + "type": "string", + "enum": ["figma", "exports", "none"] + }, + "file_url": { + "description": "Figma file or node URL. Used when `type` is `figma`.", + "type": "string", + "minLength": 1 + }, + "export_dir": { + "description": "Repo-relative directory holding designer-supplied exports. Used when `type` is `exports`.", + "type": "string", + "minLength": 1 + } + } + } + } } } } diff --git a/skills/vinta-analyze-codebase/SKILL.md b/skills/vinta-analyze-codebase/SKILL.md index bed2d730..06933669 100644 --- a/skills/vinta-analyze-codebase/SKILL.md +++ b/skills/vinta-analyze-codebase/SKILL.md @@ -163,7 +163,7 @@ This is a load-bearing input for the rest of the bootstrap flow — surface ever For each skill, also classify: - `vinta-managed` — directory name starts with `vinta-` (installed by `vinta-ai-workflows`; can be left alone or refreshed via that CLI). -- `foundation-shape` — name matches the Vinta foundation set (`plan-feature`, `create-spec`, `create-qa-use-cases`, `implement-plan`, `implement-phase`, `review-phase`, `integrate-phase`, `amend-plan`, `add-e2e-test`, `add-env-var`, `write-unit-test`). `implement-phase` / `review-phase` / `integrate-phase` are the plan-execution sub-skills co-shipped with `implement-plan`. +- `foundation-shape` — name matches the Vinta foundation set (`plan-feature`, `create-spec`, `create-qa-use-cases`, `implement-plan`, `implement-phase`, `review-phase`, `integrate-phase`, `amend-plan`, `add-e2e-test`, `add-env-var`, `write-unit-test`, `dev-desk-check`, `qa-review`). `implement-phase` / `review-phase` / `integrate-phase` are the plan-execution sub-skills co-shipped with `implement-plan`. - `project-custom` — anything else; written by the team. **Sub-agents** — list every agent file with its name + description: diff --git a/skills/vinta-bootstrap-ai-tools/SKILL.md b/skills/vinta-bootstrap-ai-tools/SKILL.md index f7cd0e1c..df4def98 100644 --- a/skills/vinta-bootstrap-ai-tools/SKILL.md +++ b/skills/vinta-bootstrap-ai-tools/SKILL.md @@ -97,7 +97,7 @@ These bleed across sub-skills, so capture once now: ### D. Optional foundation skills -Seven skills are part of the foundation set but aren't always needed. Ask explicitly: +Nine skills are part of the foundation set but aren't always needed. Ask explicitly: 1. **`add-e2e-test`** — does the project have e2e tests (Playwright / Cypress / similar) or plan to add them? `AskUserQuestion` options: `Yes — already has them`, `Yes — planning to add`, `No — skip`. If yes, [vinta-derive-skills](../vinta-derive-skills/SKILL.md)'s **Optional — ask the user** bucket follows up to ask whether the user has a template or wants to draft from scratch. **This answer also governs `create-qa-use-cases` and the e2e content inside `plan-feature`**: on `No — skip`, set `foundation_skills.create-qa-use-cases: disabled` too — derive-skills won't ship `create-qa-use-cases` and strips `plan-feature`'s `` regions so no-e2e projects carry zero Playwright / `QA_USE_CASES.md` / `pr-screenshots/` references. On either `Yes`, set both `add-e2e-test` and `create-qa-use-cases` to `enabled`. 2. **`add-env-var`** — does the project have a non-trivial env-var propagation flow (multiple files / build configs / CI updates per new var) or a single `.env` file is enough? `AskUserQuestion` options: `Yes — non-trivial flow`, `No — single .env is enough`. Skip if `No`. @@ -136,7 +136,25 @@ Seven skills are part of the foundation set but aren't always needed. Ask explic - **API spec path.** Open prose. Repo-relative path to a machine-readable spec (OpenAPI file, GraphQL schema, proto dir) plus the command that regenerates it, if one exists (e.g. `docs/openapi.yaml (regenerate: make openapi)`). Cross-check the inventory for spec files the user forgot. Empty is fine — the rendered skill then enumerates changes from route/serializer code alone. - **Output dir.** Default `.vinta-ai-workflows/client-handoffs`. Override if the team wants handoffs committed (e.g. `docs/api-changes/`) — note that the default lives under the gitignored `.vinta-ai-workflows/`, so docs there must be shared manually. -If the user answers "No" to any of the seven, that skill won't ship. If the user answers "Yes" to `add-e2e-test` / `add-env-var` but doesn't have a template, derive-skills drafts one via interview. `systematic-debugging` is always template-rendered — no per-project drafting interview, just the MCP-server inventory above. `add-one-off-script` is copied verbatim from [vinta-derive-skills/resources/foundation-skills/add-one-off-script/](../vinta-derive-skills/resources/foundation-skills/add-one-off-script/) — its body is project-agnostic; the per-project variability lives in the `skills.add-one-off-script.*` config block above and in env vars consumed at runtime. `prepare-worktree` is also copied verbatim from [vinta-derive-skills/resources/foundation-skills/prepare-worktree/](../vinta-derive-skills/resources/foundation-skills/prepare-worktree/) — same pattern: project-agnostic body, per-project defaults under `skills.prepare-worktree.*`. `thermo-nuclear-code-quality-review` is copied verbatim from [vinta-derive-skills/resources/foundation-skills/thermo-nuclear-code-quality-review/](../vinta-derive-skills/resources/foundation-skills/thermo-nuclear-code-quality-review/) with no follow-up config — its body is fully project-agnostic. `handoff-to-client` is always template-rendered from [vinta-derive-skills/resources/handoff-to-client-template.md](../vinta-derive-skills/resources/handoff-to-client-template.md) — no per-project drafting interview, just the client-handoff config follow-up above. +8. **`dev-desk-check`** — should agents be able to QA a change by driving a real browser through the app: preflight the environment, walk the flows a user would, and report what actually renders? Ask only when the project has a user-facing frontend. `AskUserQuestion` options: `Yes — enable`, `No — skip`. Recommend `Yes` whenever the project has a frontend, and especially when `run_options.run_e2e` is off — nothing else exercises the browser then. Note the prerequisite: the skill needs a browser skill or MCP in the harness and refuses to run without one; it ships no browser tooling of its own. + + **Follow-up only when `dev-desk-check` = Yes (or `qa-review` = Yes — the two share one config block).** These land under `skills.dev-desk-check.*`. Ask them once, even if both skills are enabled. + + - **Environments.** For each environment the team QAs (`local`, `staging`, `production`, …): its base URL; optionally the API URL + health path; and for `local` only, the command that starts the frontend. **Leave the start command blank unless it should really be run** — no key means probe and stop, which is what you want when a dev server may already be up on that port. + - **Write policy per environment.** `AskUserQuestion` per environment: `free` (writes proceed; a ledger is kept either way), `confirm` (every write enumerated and authorised before the browser opens, in a confirmation that names the environment), `forbidden` (the run refuses and proposes instead). Recommend `confirm` for anything prod-like and `free` for local. Setup writes — feature flags, seeds, migrations — stay propose-only under every value; say so when the user picks `free` for production. + - **Data sensitivity per environment.** `AskUserQuestion`: `none`, `client`, `phi`. Default `none`. Explain what it drives: `client` keeps screenshots in the gitignored report dir and off PRs and design tools; `phi` adds opaque-ids-only reports and removes the shareable-Artifact deliverable entirely. Ask directly for any environment carrying production records — this is the field that keeps evidence handling correct, and guessing it wrong is the expensive kind of wrong. + - **Edge proxy per environment.** Open prose, usually empty. Name the proxy (`cloudflare`, …) when one fronts the environment; the rendered skill then emits no `curl` preflight there, because such proxies answer non-browser traffic with a 403 and reading that as downtime kills a healthy run. + - **Backing services (optional).** For each: name, a **read-only, time-bounded** probe command, whether it is required, and optionally a start command. Omit entirely and the services section is stripped from the rendered skill rather than shipped empty. + - **Auth accounts.** For each role a QA run may need: the role description, plus **a pointer to where the credential lives** (`e2e/.env (gitignored)`, `1Password → QA vault`). **Never the credential.** Login is always manual — the agent cannot type a password in any environment; say this out loud so nobody tries to supply one. + - **Feature-flag system.** `AskUserQuestion`: `waffle`, `launchdarkly`, `env`, `none`. On anything but `none`, ask where to find which flags apply to which QA account. On `none` the whole region is stripped rather than shipping as dead advice. + - **Design source.** `AskUserQuestion`: `Figma (live, via MCP)`, `Designer-supplied exports`, `None — skip the design pass`. On Figma, ask for the file / node URL; on exports, the directory. On `None` the design pass is skipped, and the rendered skill records the skip under *Not tested* rather than dropping it. + - **Escalation contact (optional).** Open prose: who a reviewer asks when blocked (`#acme-eng on Slack`, `the on-call engineer`). Mostly matters for `qa-review`, whose holder has no repo. Omit and the render says "the engineer who gave you this file". + - **Report dir + use-cases path.** Defaults `.vinta-ai-workflows/qa` and `QA_USE_CASES.md`. Keep the report default under the gitignored `.vinta-ai-workflows/` unless every environment is `data_sensitivity: none` — `client` and `phi` require it. + - **Per-run defaults.** `AskUserQuestion` for `screenshots` (default on) and default `scope` (`diff` / `use-cases` / `description`), plus open prose for the preferred `browser` tool name. Land in `run_options.dev-desk-check.*`. + +9. **`qa-review`** — do product managers or designers on this project need to QA a feature themselves, in a browser, without a code checkout? `AskUserQuestion` options: `Yes — enable`, `No — skip`. Asked **independently** of `dev-desk-check` — a project can take the engineer path alone, the review path alone, or both — but it renders from the same `skills.dev-desk-check` config block, so ask the follow-ups above exactly once. Recommend `Yes` for teams where a PM or designer signs off on a flow before release. Explain what they get: one self-contained `SKILL.md` an engineer hands over, which works when dropped into their own agent setup with no repo, no terminal, and no install. + +If the user answers "No" to any of the nine, that skill won't ship. If the user answers "Yes" to `add-e2e-test` / `add-env-var` but doesn't have a template, derive-skills drafts one via interview. `systematic-debugging` is always template-rendered — no per-project drafting interview, just the MCP-server inventory above. `add-one-off-script` is copied verbatim from [vinta-derive-skills/resources/foundation-skills/add-one-off-script/](../vinta-derive-skills/resources/foundation-skills/add-one-off-script/) — its body is project-agnostic; the per-project variability lives in the `skills.add-one-off-script.*` config block above and in env vars consumed at runtime. `prepare-worktree` is also copied verbatim from [vinta-derive-skills/resources/foundation-skills/prepare-worktree/](../vinta-derive-skills/resources/foundation-skills/prepare-worktree/) — same pattern: project-agnostic body, per-project defaults under `skills.prepare-worktree.*`. `thermo-nuclear-code-quality-review` is copied verbatim from [vinta-derive-skills/resources/foundation-skills/thermo-nuclear-code-quality-review/](../vinta-derive-skills/resources/foundation-skills/thermo-nuclear-code-quality-review/) with no follow-up config — its body is fully project-agnostic. `handoff-to-client` is always template-rendered from [vinta-derive-skills/resources/handoff-to-client-template.md](../vinta-derive-skills/resources/handoff-to-client-template.md) — no per-project drafting interview, just the client-handoff config follow-up above. `dev-desk-check` and `qa-review` are both template-rendered from [vinta-derive-skills/resources/dev-desk-check-template.md](../vinta-derive-skills/resources/dev-desk-check-template.md) and [qa-review-template.md](../vinta-derive-skills/resources/qa-review-template.md) — no per-project drafting interview, just the shared config follow-ups above. They are independent opt-ins over one config block: enabling only `qa-review` is a valid answer, and still requires the environment / write-policy / data-sensitivity questions. ### E. Existing AI artifacts (per-artifact disposition) @@ -156,7 +174,7 @@ For each artifact, read it (frontmatter + body), then ask the user via `AskUserQ - `Keep in current vendor path, don't touch` — leaves it where it is. AGENTS.md may reference it; downstream skill setup won't manage it. - `Drop` — delete (rare; usually the user wants to migrate). - Foundation-shape skills (name matches `plan-feature`, `create-spec`, `create-qa-use-cases`, `implement-plan`, `implement-phase`, `review-phase`, `integrate-phase`, `amend-plan`, `add-e2e-test`, `add-env-var`, `add-one-off-script`, `prepare-worktree`, `thermo-nuclear-code-quality-review`, `deslop-comments`, `handoff`, `handoff-to-client`, `write-unit-test`) get an extra option: `Replace with Vinta foundation version` — overwrites with the canonical foundation content, preserving the user's name. Useful when the existing version is stale. (`implement-phase` / `review-phase` / `integrate-phase` are the plan-execution sub-skills co-shipped with `implement-plan`; replace them as a unit.) + Foundation-shape skills (name matches `plan-feature`, `create-spec`, `create-qa-use-cases`, `implement-plan`, `implement-phase`, `review-phase`, `integrate-phase`, `amend-plan`, `add-e2e-test`, `add-env-var`, `add-one-off-script`, `prepare-worktree`, `thermo-nuclear-code-quality-review`, `deslop-comments`, `handoff`, `handoff-to-client`, `write-unit-test`, `dev-desk-check`, `qa-review`) get an extra option: `Replace with Vinta foundation version` — overwrites with the canonical foundation content, preserving the user's name. Useful when the existing version is stale. (`implement-phase` / `review-phase` / `integrate-phase` are the plan-execution sub-skills co-shipped with `implement-plan`; replace them as a unit.) - **Sub-agents** (each under any vendor `agents/` dir): - `Migrate to ai-tools/agents/.yaml` — converts vendor-specific format → canonical YAML; `setup-ai-tools.mjs` re-emits per-vendor copies. @@ -279,6 +297,8 @@ foundation_skills: handoff: enabled # always ships — project-agnostic session-continuation handoff docs handoff-to-client: write-unit-test: + dev-desk-check: + qa-review: foundation_agents: implementer: enabled @@ -305,6 +325,12 @@ run_options: use_worktree: amend-plan: blast_radius_signal_threshold: 2 + # Only emit this block when foundation_skills.dev-desk-check or qa-review = enabled. + dev-desk-check: + screenshots: true + scope: # qa-review supports use-cases | description only + browser: + dry_run: false skills: # Only emit this block when foundation_skills.systematic-debugging = enabled. @@ -329,6 +355,40 @@ skills: db_isolation: additional_conventions: + # Only emit this block when foundation_skills.dev-desk-check OR foundation_skills.qa-review = enabled. + # Both skills render from this one block. NEVER put a credential here — auth.accounts[].where + # is a pointer to where a secret lives, not the secret. + dev-desk-check: + use_cases_path: QA_USE_CASES.md + report_dir: .vinta-ai-workflows/qa + escalation_contact: + environments: + : # at least one required + base_url: + api_url: + api_health_path: /health/ + frontend_start: + writes: # confirm is the default for anything prod-like + data_sensitivity: + edge_proxy: + services: # omit entirely to strip the services region + - name: + probe: + required: + start: + auth: + login: manual # the only supported value + accounts: + - role: + where: + feature_flags: # omit, or system: none, to strip the region + system: + check: + design_source: # omit, or type: none, to skip the design pass + type: + file_url: + export_dir: + # Only emit this block when foundation_skills.prepare-worktree = enabled. prepare-worktree: worktree_root: -wt-`> @@ -426,6 +486,8 @@ ai-tools/ │ │ └── one_off_script_base.ts │ ├── prepare-worktree/SKILL.md ← optional — only if user opts in (verbatim copy; project-agnostic body, defaults under skills.prepare-worktree.*) │ ├── thermo-nuclear-code-quality-review/SKILL.md ← optional — only if user opts in (verbatim copy; deep structural-maintainability audit, escalation target for review-phase Layer 3) +│ ├── dev-desk-check/SKILL.md ← optional — only if user opts in (template-rendered; config under skills.dev-desk-check.*) +│ ├── qa-review/SKILL.md ← optional — only if user opts in (template-rendered, self-contained: no relative links, no repo commands — handed to a PM / designer as one file) │ ├── run-one-off-script-django/ ← optional sister skill — only when stack matches + user supplies template (authors Jupyter notebook / mgmt command runner + JupyterRuntime / DjangoMgmtRuntime adapter in the per-script folder) │ ├── run-one-off-script-medplum/ ← optional sister skill — only when stack matches + user supplies template (authors Medplum bot + MedplumBotRuntime adapter in the per-script folder) │ └── /SKILL.md ← only if user supplied templates @@ -456,7 +518,7 @@ Foundation skills break into three buckets — see [vinta-derive-skills](../vint - **Always copy verbatim**: `plan-feature`, `create-spec`, `create-qa-use-cases`, `deslop-comments`, `handoff`. Bundled with the bootstrap skill set; project-agnostic enough to ship as-is (with light path scrubs). `deslop-comments` always ships because `review-phase`'s Layer 2 comment-hygiene check + fix loop dispatch it, and `integrate-phase` / `amend-plan` run it over the PR-context file. `handoff` always ships because its session-continuation body is fully project-agnostic. - **Always generate**: the plan-execution unit — `implement-plan` (conductor) + its co-shipped sub-skills `implement-phase` / `review-phase` / `integrate-phase`, plus `amend-plan` (conductor). Bodies have too much project-specific content (test commands, branch convention, PR + co-author policy, agent dispatch) — generated from parameterized templates + shared partials using interview answers + inventory. The sub-skills are not independently opt-in; they always ship with the conductors. `write-unit-test` is also template-rendered but **conditionally generated** — it ships whenever a unit-test framework was detected (not an opt-in question, not ask-first): the rendered body bakes in the project's test command + captured `skills.write-unit-test.*` conventions, and the verbatim packs land in its `resources/packs/` — one runner pack for the detected test framework plus a stack pack per matched stack (Django / FastAPI / Flask / Medplum / React / Next.js / TanStack Start / React Router / Prisma / pure-package), so framework-specific advice only ships when that framework is present. A few stack packs also apply a one-time project-setup step at derive time (e.g. Medplum wires the two FHIR-indexing Vitest setup files — `test.globalSetup.ts` + `test.setup.ts` — into the Vitest config so `MockClient` search filtering works) — a project mutation done once, not per test. -- **Optional, ask first**: `add-e2e-test`, `add-env-var`, `systematic-debugging`, `add-one-off-script`, `prepare-worktree`, `thermo-nuclear-code-quality-review`, `handoff-to-client`. Skipped by default; orchestrator asks via `AskUserQuestion` whether the project has the relevant flow at all. `add-e2e-test` / `add-env-var`: if yes + user has a template → copy + adapt; if yes + no template → draft from scratch via interview; if no → don't ship. `systematic-debugging`: if yes → render the bundled template plus the per-tool MCP catalogue blocks for the observability tools selected in its follow-up; if no → don't ship. `add-one-off-script`: if yes → copy the bundled SKILL.md verbatim plus the language-specific `BaseOneOffScript` template (`one_off_script_base.py` / `one_off_script_base.ts`) chosen via its follow-up; if no → don't ship. `prepare-worktree`: if yes → copy the bundled SKILL.md verbatim, populate `skills.prepare-worktree.*` defaults from its follow-ups, and (when the user opted in via the worktree-default follow-up) flip `run_options.implement-plan.use_worktree` to `true` so `implement-plan`'s Step 0 question (c) defaults to yes; if no → don't ship. `thermo-nuclear-code-quality-review`: if yes → copy the bundled SKILL.md verbatim (no follow-up config); if no → don't ship. `handoff-to-client`: asked only for API-only repos; if yes → render the bundled template using `skills.handoff-to-client.*` (client platforms, API style, spec path, output dir) from its follow-ups; if no → don't ship. +- **Optional, ask first**: `add-e2e-test`, `add-env-var`, `systematic-debugging`, `add-one-off-script`, `prepare-worktree`, `thermo-nuclear-code-quality-review`, `handoff-to-client`, `dev-desk-check`, `qa-review`. Skipped by default; orchestrator asks via `AskUserQuestion` whether the project has the relevant flow at all. `add-e2e-test` / `add-env-var`: if yes + user has a template → copy + adapt; if yes + no template → draft from scratch via interview; if no → don't ship. `systematic-debugging`: if yes → render the bundled template plus the per-tool MCP catalogue blocks for the observability tools selected in its follow-up; if no → don't ship. `add-one-off-script`: if yes → copy the bundled SKILL.md verbatim plus the language-specific `BaseOneOffScript` template (`one_off_script_base.py` / `one_off_script_base.ts`) chosen via its follow-up; if no → don't ship. `prepare-worktree`: if yes → copy the bundled SKILL.md verbatim, populate `skills.prepare-worktree.*` defaults from its follow-ups, and (when the user opted in via the worktree-default follow-up) flip `run_options.implement-plan.use_worktree` to `true` so `implement-plan`'s Step 0 question (c) defaults to yes; if no → don't ship. `thermo-nuclear-code-quality-review`: if yes → copy the bundled SKILL.md verbatim (no follow-up config); if no → don't ship. `handoff-to-client`: asked only for API-only repos; if yes → render the bundled template using `skills.handoff-to-client.*` (client platforms, API style, spec path, output dir) from its follow-ups; if no → don't ship. `dev-desk-check` / `qa-review`: independent opt-ins over one shared `skills.dev-desk-check` config block (environments + per-environment write policy and data sensitivity, optional service probes, auth account pointers, feature-flag system, design source); if yes → render the matching template; if no → don't ship. `qa-review` additionally renders self-contained — no relative links, no repo-dependent commands — because it is handed to someone with no checkout. Stack-specific skills + agents land in the target only when the user provides templates for them. If they don't have templates yet, the orchestrator records the detected stacks + skill categories as a TODO list the user can address later via [vinta-derive-skills](../vinta-derive-skills/SKILL.md) / [vinta-derive-subagents](../vinta-derive-subagents/SKILL.md) standalone runs. diff --git a/skills/vinta-derive-skills/SKILL.md b/skills/vinta-derive-skills/SKILL.md index e146050d..7819bf54 100644 --- a/skills/vinta-derive-skills/SKILL.md +++ b/skills/vinta-derive-skills/SKILL.md @@ -1,6 +1,6 @@ --- name: vinta-derive-skills -description: Author project-specific skills under `ai-tools/skills/`. Always copies the project-agnostic foundation set (`plan-feature`, `create-spec`, `open-pr-from-context`) verbatim from this skill's bundled resources; ships `create-qa-use-cases` and keeps `plan-feature`'s e2e sections only when `add-e2e-test` is enabled. Generates `implement-plan` from a parameterized template using project-specific commands + branch / commit / PR conventions captured in the bootstrap interview. Asks the user whether `add-e2e-test` and `add-env-var` are needed (optional foundation skills — only ship if applicable). Stack-specific skills (Medplum, Django, etc) come from user-supplied templates pointed at by the orchestrator — see [bootstrap-ai-tools/resources/stacks//notes.md](../vinta-bootstrap-ai-tools/resources/stacks/) for the categories per stack. +description: Author project-specific skills under `ai-tools/skills/`. Always copies the project-agnostic foundation set (`plan-feature`, `create-spec`, `open-pr-from-context`) verbatim from this skill's bundled resources; ships `create-qa-use-cases` and keeps `plan-feature`'s e2e sections only when `add-e2e-test` is enabled. Generates `implement-plan` from a parameterized template using project-specific commands + branch / commit / PR conventions captured in the bootstrap interview. Asks the user whether `add-e2e-test`, `add-env-var`, `dev-desk-check` and `qa-review` are needed (optional foundation skills — only ship if applicable). Stack-specific skills (Medplum, Django, etc) come from user-supplied templates pointed at by the orchestrator — see [bootstrap-ai-tools/resources/stacks//notes.md](../vinta-bootstrap-ai-tools/resources/stacks/) for the categories per stack. --- # Derive skills @@ -26,7 +26,7 @@ Before drafting any new SKILL.md, walk through every entry in `existing_ai_artif - **Migrate to `ai-tools/skills//`** — `git mv` the existing skill folder (with all its resources) into the canonical layout. After move, scan the body for hard-coded vendor paths that no longer apply (`.cursor/skills/...` self-references, etc.) and rewrite to the new path. `setup-ai-tools.mjs` will re-link to the chosen vendors. - **Keep in current vendor path, don't touch** — leave it where it is; AGENTS.md may reference it; downstream skill setup won't manage it. **Don't ship a foundation duplicate that would shadow it.** - **Drop** — log removal; don't emit anything. -- **Replace with Vinta foundation version** (foundation-shape only — `plan-feature`, `create-spec`, `create-qa-use-cases`, `implement-plan`, `add-e2e-test`, `add-env-var`, `add-one-off-script`, `prepare-worktree`, `thermo-nuclear-code-quality-review`, `deslop-comments`, `handoff`, `handoff-to-client`, `write-unit-test`) — proceed with the bucket A / B / C flow below for that name. The user has explicitly opted into overwriting their version. +- **Replace with Vinta foundation version** (foundation-shape only — `plan-feature`, `create-spec`, `create-qa-use-cases`, `implement-plan`, `add-e2e-test`, `add-env-var`, `add-one-off-script`, `prepare-worktree`, `thermo-nuclear-code-quality-review`, `deslop-comments`, `handoff`, `handoff-to-client`, `write-unit-test`, `dev-desk-check`, `qa-review`) — proceed with the bucket A / B / C flow below for that name. The user has explicitly opted into overwriting their version. - **`vinta-managed`** (any skill whose dir starts with `vinta-`) — leave alone. These come from the `vinta-ai-workflows` CLI; `vinta-derive-skills` doesn't manage them. For each name in the foundation set: only emit it (verbatim copy / generate from template / interview-draft) when no existing skill of that name is being migrated or kept. If the user said `Migrate` or `Keep` for an existing `plan-feature/SKILL.md`, **do not overwrite**. @@ -120,6 +120,10 @@ The five are a **co-shipped unit**: `implement-plan` is always generated, so `im This is a project mutation — do it once at bootstrap. Canonical source is the team's snippet catalogue (`vintasoftware/medplum-snippet-catalog`, `TestSetup/`); the bundled copies (dated in their headers) keep bootstrap offline-safe — prefer the live catalogue if it has advanced. The rendered `medplum.md` only *mentions* the override so tests don't re-index. +- **`dev-desk-check`** — drive a real browser through the app as a user, judge what renders, and produce evidence someone who wasn't there can trust. Plan source is the branch diff (vs the PR's real base), ids from the project's QA use-cases doc, or a ticket. From [resources/dev-desk-check-template.md](resources/dev-desk-check-template.md). **Opt-in** — only generated when `foundation_skills.dev-desk-check` is `enabled`. The rendered body bakes in `skills.dev-desk-check.*`: environments with their per-environment write policy and data sensitivity, optional service probes, auth account roles (pointers, never secrets), feature-flag system, design source. Composes with, and never duplicates, its neighbours — [create-qa-use-cases](resources/foundation-skills/create-qa-use-cases/SKILL.md) writes the checklist, this walks it; `add-e2e-test` automates a use case, this reports which ids a change touches. **Not** wired into `implement-plan`'s phase gate; a browser pass per phase is too slow, the same reasoning that keeps `run_options.run_e2e` off. + +- **`qa-review`** — the same browser walk and report shape for product managers and designers, stripped of everything that assumes a repo: no git, no diff, no dev servers, no service probes, no PR edit, no database. From [resources/qa-review-template.md](resources/qa-review-template.md). **Opt-in** — only generated when `foundation_skills.qa-review` is `enabled`, independently of `dev-desk-check` (a project may take either or both). Renders from the **same** `skills.dev-desk-check` config block. Rendered **self-contained**: everything it needs is inlined, so the single file works when dropped at `~/.claude/skills/qa-review/SKILL.md` by someone with no checkout. Stable facts are inlined (environment addresses, which account role to use, the design source, who to ask when blocked); the QA use-cases doc deliberately is **not** — a snapshot of a checklist the repo keeps updating is worse than no snapshot, so the reviewer pastes the use case instead. The render is stamped with its generation date and source commit; [vinta-update-project-skills](../vinta-update-project-skills/SKILL.md) re-renders it when the facts change. + Every plan-execution shell + `systematic-debugging` consumes the placeholder set below — render each by substituting from the inventory + Step 0 interview answers. `systematic-debugging` reuses the same set plus two extra placeholders (`{{OBSERVABILITY_MCP_BLOCK}}`, `{{OBSERVABILITY_MCP_LIST}}`) rendered from the catalogue. #### Rendering the plan-execution unit @@ -193,6 +197,22 @@ The remaining per-shell placeholders: | `{{DEPENDENCY_LICENSE_LAYER1_CHECK}}` | derived from `policies.dependency_licenses` | Layer 1 review checklist item. Empty string when `enforcement: off`. When `block` / `warn`: `6. **Dependency license scan**: \`git diff package.json pyproject.toml ...\` (project-relevant manifests) — for every added dep look up its SPDX license (\`npm view license\`, PyPI metadata, repo \`LICENSE\`). A license in \`policies.dependency_licenses.forbidden_spdx\` and not in \`allowed_overrides\` is a BLOCKER (when \`block\`) or a SHOULD-FIX (when \`warn\`). A missing / \`UNKNOWN\` / undeclared license is **always a BLOCKER** regardless of enforcement mode — there is no override to silently bless undisclosed terms.` | | `{{DEPENDENCY_LICENSE_RULE_LINE}}` | derived from `policies.dependency_licenses` | Important-rules bullet. Empty string when `enforcement: off`. When `block`: `- **License check before any new dep.** Refuse \`npm add\` / \`pnpm add\` / \`pip install\` / \`poetry add\` / \`uv add\` / \`cargo add\` / \`go get\` when the package's SPDX license is in the forbidden list — see AGENTS.md **Dependency licenses**. User can grant a one-off override after acknowledging the violation; record the override in \`policies.dependency_licenses.allowed_overrides\` before re-running.` When `warn`: same line but `Proceed but record the violation in the phase report` instead of `Refuse`. | | `{{COMMIT_STRATEGY_*}}` family (`{{COMMIT_STRATEGY_STEP0_QUESTION}}`, `{{COMMIT_STRATEGY_STEP0_TRAILER}}`, `{{COMMIT_STRATEGY_CONFIRM_NOTE}}`, `{{BRANCH_NAMING_PATTERN_SUMMARY}}`, `{{BRANCH_NAMING_BLOCK}}`, `{{PER_PHASE_COMMIT_BLOCK}}`, `{{PR_OPEN_TIMING_BLOCK}}`, `{{PRS_CONTEXT_FILE_PATH}}`, `{{TRACKING_BRANCH_FIELD}}`, `{{TRACKING_PHASE_BRANCH_FIELD}}`, `{{FINAL_REPORT_BRANCH_SUMMARY}}`, `{{BRANCH_CHECKLIST_LINE}}`, `{{COMMIT_STRATEGY_CHECKLIST_BLOCK}}`, `{{BRANCH_PUSH_HEADING}}`) | Step 0 (`policies.commit_strategy`) | Render based on the project's commit strategy (`stacked-branches` / `modular-commits` / `ask`). Multi-line blocks now come from the commit-strategy partials; see the substitution table immediately after this row. | +| `{{QA_ENVIRONMENTS_TABLE}}` | `skills.dev-desk-check.environments` (dev-desk-check + qa-review) | Markdown table, one row per declared environment: `\| Name \| Base URL \| Writes \| Data sensitivity \|`. Never empty — the schema requires at least one environment. | +| `{{QA_ENVIRONMENT_NAMES}}` | `skills.dev-desk-check.environments` (dev-desk-check + qa-review) | Comma-separated environment names in declaration order, e.g. `local, staging, production`. **Frontmatter-only in both templates** — the rendered body sources its own option lists from `{{QA_ENVIRONMENTS_TABLE}}` instead, so a check that scans only the body (not the `description:` frontmatter) will find this placeholder unused there; that is expected, not a gap. | +| `{{QA_PREFLIGHT_BLOCK}}` | `environments..{api_url,api_health_path,frontend_start,edge_proxy}` (dev-desk-check only) | Per-environment preflight steps for **frontend then backend only** — services are rendered separately, above, by `{{QA_SERVICES_BLOCK}}` under its own heading; this block sits under the template's literal `### Frontend and backend` heading and must not re-emit service probes there. **`edge_proxy` decides the shape of an environment's whole entry — frontend and backend together — and it overrides every probe instruction below it.** When `edge_proxy` is set, emit no `curl` at all for that environment and render instead: ` sits behind , which answers non-browser traffic with a 403. No curl preflight — the browser is the preflight.` Probing an edge-fronted environment reports a healthy app as down, which is the one preflight failure that costs the entire run. When `edge_proxy` is **absent**, emit a frontend probe `curl -sS -o /dev/null -w '%{http_code}' --max-time 10 `, followed by a backend probe `curl -sS -o /dev/null -w '%{http_code}' --max-time 10 ` **only** when `api_url` is set. `frontend_start` governs *starting*, never *probing*: emit the start offer (local-only, asks first) when it is set, and otherwise emit `No start command is declared for — never start a server for it. A second dev server on an occupied port silently invalidates the whole run.` **That fallback must not carry a probe instruction of its own**, or it contradicts the `edge_proxy` rule for every environment that is both edge-fronted and unstartable — which is every remote environment a typical project declares. | +| `{{QA_SERVICES_BLOCK}}` | `skills.dev-desk-check.services[]` (dev-desk-check only) | The whole `### Services pre-check` section body — **h3, because it nests inside `## Phase 3 — Preflight`, not h2** — one bullet per service with its `probe`, its `required` verdict (`true` → stop the run and offer `start`; `false` → continue and record the degradation in the report), and the three fixed rules (probe-cannot-run is `unknown` not `down`; auto-start is local-only and asks first; probes run before the frontend and backend checks, because a dependency that is down produces failures indistinguishable from product bugs and a run that reaches the browser with a dead queue spends its findings section describing the outage). **The hosting template's Phase 3 prose names no services step of its own** — the fixed "Fixed order" line lists `frontend → backend → data preconditions` only, so a project declaring no `services` ships a Phase 3 that does not advertise a step it will never run. Keep it that way: an ordering or rationale sentence about services belongs in this block, which vanishes with them, never in the template around it. **Empty string when `services` is absent or empty — the whole section disappears, heading included.** | +| `{{QA_WRITE_POLICY_BLOCK}}` | `environments..writes` (dev-desk-check + qa-review) | Renders **only the per-environment write-policy bindings** — one line per declared environment naming its `free` / `confirm` / `forbidden` value and what that value means there. Must **not** re-emit a fixed write-gate rule the hosting template already carries as literal prose: guard rails are data, the seven rules from the spec's **Production write gate** are prose, and a rule shipped twice is worse than shipped once — a reader cannot tell which copy binds. **`dev-desk-check`**: the block supplies **none** of the seven — its template already carries all seven verbatim (rules 1–7 in order), including rule 6's `Writes carry synthetic values only. Never real client or patient data as input. Under `phi`, the ledger records opaque IDs, not field values.` clause. **`qa-review`**: its template already carries rules 1, 3, 4 and 6 in reviewer-language prose, plus the UI-reversal rule as its own bullet — so the block supplies **only rules 2, 5 and 7**, in reviewer language: rule 2 (tracking what changed applies no matter what the answer above was), rule 5 (a change the policy above refuses gets written down and proposed, never made), and rule 7 (anything whose effects reach beyond the one thing being tested is proposed and left for an engineer, in every environment) — avoiding `ledger`, `seed`, `migration`, `pull request` and `branch` in all three. | +| `{{QA_DATA_SENSITIVITY_BLOCK}}` | `environments..data_sensitivity` (dev-desk-check + qa-review) | The evidence-handling table restricted to the sensitivity levels actually declared, plus one line per environment binding it: `production → phi: opaque ids only, never field values; no Artifact deliverable.` When every environment is `none`, render the one-line form: `No environment carries client or patient data. Screenshots and report contents are unrestricted.` | +| `{{QA_AUTH_BLOCK}}` | `skills.dev-desk-check.auth` (dev-desk-check + qa-review) | Renders **only** the `accounts[]` table — `\| Role \| Credential location \|`, `where` verbatim as a pointer, never a secret. The manual-login rule itself is already fixed template prose, not something this block supplies: `### Manual login` in the dev-desk-check template, and `**You sign in yourself.**` sitting directly next to the placeholder in the qa-review template. When `accounts` is empty: **dev-desk-check** emits `No account roles are declared — ask the operator which account to use.` **qa-review** must not reuse that engineer-facing fallback — its reader *is* the person signing in — and instead addresses them directly, e.g. `No account role is declared for this project — ask {{QA_ESCALATION_CONTACT}} which account to use.` | +| `{{QA_FEATURE_FLAGS_BLOCK}}` | `skills.dev-desk-check.feature_flags` (dev-desk-check + qa-review) | Names the declared `system` and the `check` pointer. **dev-desk-check variant owns its own `### Feature flags` heading** — do not add a literal heading above it. **Empty string when `feature_flags` is absent or `system: none` — the whole section disappears, heading included.** In the `qa-review` render, the block instead brings its **own** heading in reviewer language (e.g. `### Settings that turn features on`), naming the *setting* rather than the flag system behind it — never `waffle` / `launchdarkly` by name. The hosting template's own fixed prose already carries the blocked-by-a-setting outcome in reviewer language — `Blocked by the setting — ask an engineer to enable it` — so this block must **not** re-append a second, differently-worded copy of that line. | +| `{{QA_DESIGN_SOURCE_BLOCK}}` | `skills.dev-desk-check.design_source` (dev-desk-check + qa-review — **two variants**) | `type: figma` → compare against the live node, citing `file_url`. `type: exports` → compare against the designer-supplied exports at `export_dir` and **label them as exports in the report**. **dev-desk-check variant**: empty string when `design_source` is absent or `type: none` — the whole `### The design pass` heading disappears with it, **heading included** — and the skipped pass is recorded under *Not tested* with the reason; names the tooling: "via the Figma MCP tools available at runtime". **qa-review variant is never the empty string** and carries **no heading of its own** — the numbered step heading above it in the template covers that job, so a vanished region would leave a hole in the sequence the reader follows. On `type: none` it renders exactly: `There is no design to compare against on this project. Note that under *Not tested* and move on.` It also never names the tooling — "open the design file at `` and compare side by side" instead, because its reader is the designer and "MCP" is engineer vocabulary. This is the same two-variant shape `{{QA_FEATURE_FLAGS_BLOCK}}` and `{{QA_USE_CASES_BLOCK}}` already use, and it is what keeps the rendered `qa-review` inside the jargon ban enforced by the template-only placeholder/jargon grep (and, on the rendered output, the derive-time portability check above). | +| `{{QA_USE_CASES_BLOCK}}` | `skills.dev-desk-check.use_cases_path` + `foundation_skills.create-qa-use-cases` (dev-desk-check + qa-review) | **dev-desk-check:** points at the path and says use-case ids are the strongest plan source because the document already states the steps and the expected outcome. Links `[create-qa-use-cases](../create-qa-use-cases/SKILL.md)` **only when that skill is enabled**. **qa-review:** never inlines the checklist and never links a path — renders `Paste the use case, the ticket, or a description of the feature. The checklist this project keeps at changes often; a snapshot of it would be worse than none.` | +| `{{QA_E2E_IMPACT_BLOCK}}` | `foundation_skills.add-e2e-test` (dev-desk-check only) | When `enabled`: a Phase 1 step reporting which `use_cases_path` ids the change touches, and a closing pointer to `[add-e2e-test](../add-e2e-test/SKILL.md)` for automating them. **Empty string when disabled or absent** — the link would dangle. | +| `{{QA_BASE_BRANCH_DETECTION_BLOCK}}` | `project.code_host` + `project.default_branch` (dev-desk-check only) | Host-specific PR-base lookup: `github` → `gh pr view --json baseRefName -q .baseRefName`; `gitlab` → `glab mr view --output json` then read `target_branch`; `bitbucket` / `self-hosted` / `other` → ask the operator. Always followed by the fixed rule: **"no PR" and "the CLI failed" are different answers. Only the first may fall back to `{{DEFAULT_BRANCH}}`. A CLI error stops and reports — a blanket fallback turns an expired token into a plan built from the wrong diff with nothing on screen saying so.** | +| `{{QA_REPORT_DIR}}` | `skills.dev-desk-check.report_dir` (dev-desk-check + qa-review) | Plain value, e.g. `.vinta-ai-workflows/qa`. **In the `qa-review` render this value is the right location only when the reader happens to have a copy of the project on the machine they're working on** — it is a project-relative, deliberately hidden folder, and the normal holder of a `qa-review` file has no checkout at all. Without that check, the render would tell a reader with no repo that a report was saved somewhere they can never reach, and their agent would create the folder wherever it happened to be running. The `qa-review` template's sentence naming this value is therefore conditional on purpose: inside a checkout, the report goes there; without one, it goes wherever it can be handed directly to whoever asked. | +| `{{QA_ARTIFACT_DELIVERABLE_BLOCK}}` | max `data_sensitivity` across declared environments (qa-review only) | `none` → offer the choice between a markdown report and a published Artifact carrying screenshots inline. `client` → offer the Artifact only behind an explicit confirmation naming the environment. `phi` → **do not offer it at all**; render `Markdown only. Publishing to hosted storage would move records outside the managed platform's access controls and audit logging.` Markdown is always available in every case. **This block is the sole authority on where a `qa-review` report may go** — the template's fixed closing line ("this never touches a pull request…") is explicitly bounded by whatever this block offered, so the block itself must carry the actual policy rather than stand in for one. | +| `{{QA_ESCALATION_CONTACT}}` | `skills.dev-desk-check.escalation_contact` (dev-desk-check + qa-review) | The declared contact verbatim. Renders `the engineer who gave you this file` when absent. | +| `{{QA_REVIEW_PROVENANCE_BLOCK}}` | derive-time (qa-review only) | `Generated from {{PROJECT_NAME}} at commit by vinta-derive-skills .` plus the fixed refresh paragraph naming `vinta-update-project-skills` and telling the holder to ask {{QA_ESCALATION_CONTACT}} for a fresh copy when the facts above look wrong. | ### `{{COMMIT_STRATEGY_*}}` substitution table @@ -261,11 +281,27 @@ Render each template (expand includes → substitute placeholders → write), pr - `ai-tools/skills/systematic-debugging/SKILL.md` (from `systematic-debugging-template.md`) — only when `foundation_skills.systematic-debugging: enabled` - `ai-tools/skills/handoff-to-client/SKILL.md` (from `handoff-to-client-template.md`) — only when `foundation_skills.handoff-to-client: enabled` - `ai-tools/skills/write-unit-test/SKILL.md` (from `write-unit-test-template.md`) **plus** `ai-tools/skills/write-unit-test/resources/packs/runners/.md` + `.../packs/stacks/.md` (verbatim from `write-unit-test-packs/`, only the detected runner + matched stacks) — only when `foundation_skills.write-unit-test: enabled` (default-on when a test framework was detected) +- `ai-tools/skills/dev-desk-check/SKILL.md` (from `dev-desk-check-template.md`) — only when `foundation_skills.dev-desk-check: enabled` +- `ai-tools/skills/qa-review/SKILL.md` (from `qa-review-template.md`) — only when `foundation_skills.qa-review: enabled`. Rendered self-contained; see the portability check below. The first five are the **plan-execution unit** — render them together (they share partials + placeholders + cross-links); a change to one partial ripples to every shell that includes it. Validate each output before saving: **no `` marker and no `{{PLACEHOLDER}}` may survive** (angle-bracket runtime slots like `` / `` / `` are expected to remain — they are execution-time, not derive-time). If a placeholder survives, the shell/partial needs a substitution rule. Cross-link check: the conductor's links to `implement-phase` / `review-phase` / the integrate variant resolve, and `amend-plan` / `systematic-debugging` → `review-phase` resolve. `systematic-debugging` adds the two MCP-related placeholders; if a project picks an observability tool the catalogue doesn't list, ask the user for a free-form snippet (heading, required calls, fields to extract) and inline it before saving — and consider upstreaming the snippet to [resources/systematic-debugging-mcp-tools.md](resources/systematic-debugging-mcp-tools.md). +**`qa-review` carries one extra validation — the portability check.** Its whole value is that one file works with no checkout, so before saving the render, grep it: + +```bash +F=ai-tools/skills/qa-review/SKILL.md +grep -nE '\]\(\.\.?/' "$F" +grep -nE '^[[:space:]]*(git|npm|pnpm|yarn|docker|curl|make|pytest|python3?|node|npx)[[:space:]]' "$F" +grep -nE '`(git|npm|pnpm|yarn|docker|curl|pytest|npx)[[:space:]]' "$F" +grep -niE 'reducer|migration|API endpoint|waffle|launchdarkly|playwright|\bMCP\b|ledger|seed|pull request|branch' "$F" # jargon that can arrive via a rendered block +``` + +Three greps rather than one alternation, because a single `make ` pattern false-positives on ordinary prose. Any hit means a relative link or a repo-dependent command survived — usually from a config value spliced into a block, not from the template itself. Rewrite the offending value out of the render (or out of the config) before saving. A `qa-review` that links `../create-qa-use-cases/SKILL.md` is a broken file the moment its holder opens it. + +The jargon grep's term set is wider here than what a check over the template source alone can enforce, on purpose: `ledger`, `seed`, `pull request` and `branch` are added because three of the blocks that feed this render — `{{QA_WRITE_POLICY_BLOCK}}`, `{{QA_DATA_SENSITIVITY_BLOCK}}`, `{{QA_ARTIFACT_DELIVERABLE_BLOCK}}` — are shared with the engineer-facing `dev-desk-check` path. Jargon can arrive **through a rendered block** even when the template itself is clean, which is exactly what a grep over the template source cannot catch, because it only ever sees `{{...}}` still unexpanded. This derive-time check is where a config value that leaked engineer language into a shared block actually gets caught. + ### C. Optional — ask the user Two skills depend on whether the project actually does the thing. Ask via `AskUserQuestion`: @@ -276,6 +312,8 @@ Two skills depend on whether the project actually does the thing. Ask via `AskUs | `add-env-var` | "Does this project have a non-trivial env-var propagation flow (multiple files / build configs / CI to update for one new var)?" Options: `Yes`, `No — single .env file is enough` | Ask if user has an existing template. If yes → copy + adapt. If no → draft from scratch interviewing user about every layer the new var must touch (`.env.example`, build tool envPrefix allowlist, build cache hash inputs, app config module, AGENTS.md env section, CI workflows, deploy-time injection). | | `add-one-off-script` | "Does this project ever need one-off operational scripts — backfills, cleanups, ad-hoc data fixes?" Options: `Yes — enable`, `No — skip`. Recommend `Yes` for any project with a relational DB; the skill bundles a `BaseOneOffScript` class enforcing dry-run-by-default, idempotent re-runs, batched DB ops, streamed reads, segmented CSV backups (1M cells/file, never nested across tables), interruption-safe signal handlers, and console + filesystem + S3 logging. | Copy [resources/foundation-skills/add-one-off-script/](resources/foundation-skills/add-one-off-script/) into `ai-tools/skills/add-one-off-script/` (SKILL.md + the bundled `resources/one_off_script_base.{py,ts}` templates). Then ask the user whether they want the base class file pre-staged at `/one_off/_base.{py,ts}` — default `Yes` for projects whose primary lang is Python or TS. The skill itself prompts to copy the base class on first use if the user said `No` here. | | `prepare-worktree` | "Do you ever want to run plans / experiments inside an isolated git worktree (separate runnable copy of the app with its own DB, env, compose stack) instead of swapping branches in the main checkout?" Options: `Yes — enable`, `No — skip`. Recommend `Yes` for projects with long migrations, multi-day plans, or where the user is likely to want parallel concurrent work. | Copy the whole [resources/foundation-skills/prepare-worktree/](resources/foundation-skills/prepare-worktree/) dir into `ai-tools/skills/prepare-worktree/` — both `SKILL.md` and `scripts/sandbox-run.sh` (mark the script executable: `chmod +x ai-tools/skills/prepare-worktree/scripts/sandbox-run.sh`). Then ask three short follow-ups landing in `skills.prepare-worktree.*`: worktree root (default `.claude/worktrees` for claude-code projects, `../-wt-` sibling dirs otherwise), default deps strategy (`symlink` / `copy` / `reinstall`), default test-DB strategy (`always-fork` / `fork-on-schema-change` / `share`). Also ask whether `implement-plan` should default its Step 0 question (c) to `yes` — lands in `run_options.implement-plan.use_worktree`. | +| `dev-desk-check` | "Should agents be able to QA a change by driving a real browser through the app — preflight, walk the flows, report what actually renders?" Options: `Yes — enable`, `No — skip`. Recommend `Yes` for any project with a user-facing frontend, especially one where `run_options.run_e2e` is off (nothing else exercises the browser). Requires a browser skill or MCP in the harness; the skill refuses to run without one. | Render [resources/dev-desk-check-template.md](resources/dev-desk-check-template.md) using `skills.dev-desk-check.*` (see the bootstrap follow-ups). Strip the services region when `services` is absent, the feature-flag region when `feature_flags` is absent or `system: none`, and the design region when `design_source` is absent or `type: none` — heading included, so the section disappears rather than shipping empty. | +| `qa-review` | "Do product managers or designers on this project need to QA a feature themselves, without a code checkout?" Options: `Yes — enable`, `No — skip`. Independent of `dev-desk-check` — asked separately, recorded separately. | Render [resources/qa-review-template.md](resources/qa-review-template.md) from the same `skills.dev-desk-check.*` block. **Then run the portability check before saving** — see the validation paragraph below. | If user answers "No" to any: don't ship that skill. Record nothing — the foundation set is just smaller. diff --git a/skills/vinta-derive-skills/resources/dev-desk-check-template.md b/skills/vinta-derive-skills/resources/dev-desk-check-template.md new file mode 100644 index 00000000..1ec93d27 --- /dev/null +++ b/skills/vinta-derive-skills/resources/dev-desk-check-template.md @@ -0,0 +1,382 @@ +--- +name: dev-desk-check +description: Drive a real browser through {{PROJECT_NAME}} as a user and report what actually renders — the manual pass that complements automated e2e. Takes a plan from the branch diff, from QA use-case ids, or from a ticket; preflights the frontend and backend for the chosen environment ({{QA_ENVIRONMENT_NAMES}}); walks the flows; and writes an evidence-backed report to {{QA_REPORT_DIR}} plus a Q.A. section on the PR. Enforces this project's per-environment write policy and data-sensitivity rules before it opens the browser. Use when the user says "desk check this branch", "QA this branch", "check this in the browser", "does this actually work", "walk through the new flow", or before handing a UI change to review. +--- + +# Dev desk check + +{{PROJECT_NAME}} ({{STACK_SUMMARY}}) ships UI changes that nothing walks end to end. This skill drives a real +browser through the running app the way a user would, judges what actually renders, and leaves evidence +someone who wasn't in the session can trust. It is the manual counterpart to an automated e2e suite, not a +substitute: automation owns the repeatable regression pass, this owns the human-eye pass. + +It exists because the manual checklist has no executor. A QA checklist is written in user language precisely +so that a person can walk it, and then nothing walks it; the automated suite that would cover the same ground +is opt-in and usually skipped. So the whole job is to judge **what renders, not what exists**. A component +that compiles, a route that is registered, a query that returns 200 and a passing unit test are all evidence +about code — none of them is evidence that a user can complete the flow. + +This skill is invoked deliberately. It is **not wired into `implement-plan`'s phase gate**: a browser pass per +phase is too slow to sit in a loop, the same reasoning that keeps `run_options.run_e2e` off by default. Run it +when a UI change is ready to be believed. + +## Before anything else — browser tooling + +Enumerate the browser-driving tools this harness actually exposes right now — a Playwright MCP server, a +Chrome DevTools MCP server, a browser or computer-use tool the runtime ships, a browser skill installed +alongside this one. Pick by preference: + +1. The tool named in `run_options.dev-desk-check.browser`, if it is present and responding. +2. Otherwise any other browser-driving tool the harness exposes. Say which one you fell back to, and why, in + the report's **Run** line — a reader comparing two runs needs to know they used different eyes. + +**If no browser skill and no browser MCP server is available, this skill cannot run.** Say exactly that, name +what you looked for, and stop. Do **not** hand-roll Playwright, Puppeteer, Selenium or a headless-Chrome +script to get around it: writing automation is another skill's job, an unreviewed throwaway harness produces +evidence nobody can reproduce, and a browser pass this skill did not actually drive is worse than no pass. + +## Step 0 — Resolve the run + +Use `AskUserQuestion` for the finite choices, in **one batch**, and ask only what is still unresolved — if the +invoking message already named the environment or the plan source, take it and don't re-ask: + +- **Environment** — options are exactly the declared environment names from the table below. Never invent one, + never point the browser at a URL that is not in the table. +- **Plan source** — `Branch diff`, `QA use-case ids`, `Ticket / description`. Default the highlighted option + to `run_options.dev-desk-check.scope`. +- **Screenshots** — `On` / `Off`, defaulting to `run_options.dev-desk-check.screenshots`. Always asked, even when + the default is obvious (see Phase 2). + +Anything genuinely free-form — which ticket, which use-case ids, which account role — is asked as open prose, +not squeezed into fixed options. + +{{QA_ENVIRONMENTS_TABLE}} + +## Phase 1 — Choose the plan source and build the plan + +The plan is a numbered list of surfaces to reach and, for each, the **expected** result stated before the +browser opens. Writing "expected" after seeing the screen is how a run confirms whatever it happens to find. + +### From the branch diff + +{{QA_BASE_BRANCH_DETECTION_BLOCK}} + +With the base resolved, list the changed files and map each to a **reachable surface** — a route, a modal, a +wizard step, a table column, a gated action, an empty state, an error state. One changed file can produce +several surfaces, and several files can collapse into one. + +A changed file with **no reachable surface** is recorded, not walked: a migration, a management command, a CI +config, a type-only refactor. List those in the report's *Not tested* section with "no user-reachable surface" +as the reason, so a reader can see the diff was fully considered rather than partially read. + +Surfaces the diff reaches only indirectly still count. If a shared component changed, walk at least one other +place it is used — the regression risk of a UI change lives mostly outside the feature it was written for. + +### From QA use-case ids + +{{QA_USE_CASES_BLOCK}} + +This is the **strongest** plan source. The document already states the steps and the expected outcome, so +"expected vs actual" is read off rather than inferred. Quote each use case's id and title into the plan and +keep its wording; paraphrasing a checklist step is how its intent gets lost. + +### From a ticket or description + +The **weakest** source, because "expected" is being inferred rather than read. Derive concrete steps and a +concrete expected result for each, then **state the derived plan back to the operator and wait for a yes** +before opening the browser. Say plainly which parts you inferred. A plan built on a guessed expectation +produces a report that looks authoritative and tests the wrong thing. + +{{QA_E2E_IMPACT_BLOCK}} + +## Phase 2 — Resolve the run flags + +Resolve everything Step 0 left open, and state the resolved set back in one line before Phase 3: + +- **`--screenshots`** — resolved **explicitly, every run**, never assumed. Defaults come from + `run_options.dev-desk-check.screenshots`, but a default is not a decision: a run that silently captured nothing + leaves a report a reader cannot check, and a run that silently captured everything can put restricted + records on disk. Whatever the answer, screenshot handling stays bounded by the environment's + `data_sensitivity`. Captures land under `{{QA_REPORT_DIR}}`, alongside the report and the write ledger. + Under `data_sensitivity` of `client` or `phi` that location is **required, not conventional** — that + directory is gitignored, which is the whole reason it exists: a capture written anywhere else can be + committed, uploaded to a PR or attached to a design tool by someone who never saw this run. Never write a + capture to the repo root, a temp directory or a path of your own choosing. +- **`--scope`** — the plan source chosen in Phase 1. +- **`--browser`** — the tool actually selected above, not the configured preference. +- **`--writes`** — may only relax **within** the environment's declared `writes` policy, never upgrade past + it. `--writes=allow` against a `forbidden` environment is refused, not honoured. +- **`--dry_run`** — from `run_options.dev-desk-check.dry_run` unless the operator overrode it. See the end of + Phase 3b for where a dry run stops. + +## Phase 3 — Preflight + +Fixed order: **frontend → backend → data preconditions**. The order is not cosmetic: a failure at one step is +indistinguishable from a product bug at the next, and a run that reaches the browser on a broken foundation +spends its findings section describing the breakage rather than the change. + + + +{{QA_SERVICES_BLOCK}} + +### Frontend and backend + +{{QA_PREFLIGHT_BLOCK}} + + + +{{QA_FEATURE_FLAGS_BLOCK}} + +### Authentication + +{{QA_AUTH_BLOCK}} + +### Data preconditions + +State, per planned surface, what data must already exist for the expected result to be meaningful: a record in +a particular state, a user with a particular role, a non-empty list to test pagination against. Check each +through the UI or a read-only query **before** walking, and treat a missing precondition as a write request — +it goes through Phase 3b like any other, and creating one is never a silent side effect of "just setting up". + +A surface whose precondition cannot be satisfied is *Not tested* with that reason. It is not a Blocker: the +feature was never exercised, so there is nothing to call broken. + +### When the preflight blocks the run + +When a required service, the backend, or a feature flag blocks the run, **stop and name who to ask — +{{QA_ESCALATION_CONTACT}} — rather than testing around it.** A run that works around a blocker reports on +something other than the feature: a mocked response, a second dev server, a flag you flipped yourself, or a +switch to a different environment all produce a green report about a system nobody is going to ship. Record +what blocked it, what you tried, and what you need. + +Proposing the fix is not working around it. Rule 7 in Phase 3b still applies: hand the operator the +ready-to-run command plus its reverse — the flag flip, the seed, the service start — and **wait**. What +invalidates a run is flipping the flag yourself and walking on, not offering the command and letting the +operator decide. Escalating without also proposing the change leaves them with a blocker and no next step. + +## Phase 3b — Write authorization + +{{QA_WRITE_POLICY_BLOCK}} + +{{QA_DATA_SENSITIVITY_BLOCK}} + +Before the browser opens, enumerate every write the plan requires. Each entry states what changes, how it will be identifiable afterwards, and its exact reversal: + +``` +PRODUCTION — app.acme.com — 1 write required: + + CREATE draft invoice, title "QA 2026-09-09 — do not process" + REVERSE delete draft # (captured at creation) + +Nothing else in the plan mutates. Confirm the environment and scope to proceed. +``` + +1. **The confirmation must name the environment.** A "go ahead" from earlier in the conversation, or approval of a different run, does not carry. Approving a staging write and approving a production write read identically in a transcript. +2. **`free` still keeps the ledger.** Authorisation to write is not a reason to stop tracking what was written. Ledger plus end-of-run revert offer apply in every environment. +3. **The ledger is written incrementally** to `{{QA_REPORT_DIR}}`, so a run that dies halfway still leaves a record. +4. **An unlisted write stops the run** for its own confirmation. Reversals are pre-authorised — they were named in the approved block. +5. **`forbidden` refuses and proposes.** State what would be required, record it, let the operator decide. +6. **Writes carry synthetic values only.** Never real client or patient data as input. Under `phi`, the ledger records opaque IDs, not field values. +7. **Setup writes stay propose-only in every environment.** Feature flags, seeds and migrations have a blast radius unrelated to the test — flipping a flag in production turns a feature on for every user it applies to, which is a release decision, not a QA step. Hand the operator a ready-to-run command plus its reverse and wait. App-data writes relax; setup writes do not. + +When `run_options.dev-desk-check.dry_run` is true, print this block — the full plan, the resolved flags, the +preflight results and every write the run would require, with its reversal — and **stop here**. Do not open +the browser. A dry run's whole value is that it shows what a real run would touch while nothing has been +touched yet, so say explicitly that nothing was walked and nothing was written. + +## Phase 4 — Open the browser + +Open the selected tool at the chosen environment's `base_url`. Then, before walking anything: + +- Confirm you are on the expected origin and the app has actually booted — a framework error page, a proxy + interstitial and a blank white body all return quickly and all look like "the page loaded". +- Set a viewport wide enough for the layout the change targets, and say which viewport the run used. A + responsive bug and a wrong-viewport bug are indistinguishable in a screenshot with no stated width. +- If the tool cannot reach `base_url`, retry once, then stop and report. Do not switch environments to get a + page to load — an environment change makes the whole plan a different run and it must go back to Step 0. + +When the preferred tool starts and then misbehaves, fall back per the rules in **Before anything else**, and +restart the walk from step 1 rather than stitching one plan across two tools. + +## Phase 5 — Walk it as a user + +Walk the plan in order, one numbered step at a time. For each step record: what you did, what you expected, +what you actually saw, and the result. Never batch several clicks and then interpret one screenshot. + +### When the browser half-works + +Browser tooling fails partially far more often than it fails cleanly, and a partial failure is +indistinguishable from a product bug unless you are looking for it. + +- **A stale screenshot means blind clicks.** If the capture you are reading predates your last action, every + coordinate you derive from it is a guess about a page that has moved on. +- **Confirm every state change you depend on.** After a navigation, a submit, a modal open or a tab switch, + verify the new state before acting on it. Assume nothing landed because a click reported success. +- **Retire a sub-tool that fails twice.** One flake is noise; two is a broken sub-tool. Switch to another way + of getting the same evidence — a different capture mode, reading the DOM, a fresh page load — instead of + retrying the same call and accumulating uncertainty. +- **A stalled capture is not a stalled page.** The screenshot pipeline hanging says nothing about the app. + Establish whether the page is alive by another route before reporting it as hung. + +### Judge what renders, not what exists + +- **Suspect your own tooling before filing a layout bug.** Overlap, clipping, zero-height containers, + missing images and unstyled text are all classic artefacts of a partial or mid-paint capture. Reproduce the + symptom — reload, re-capture, resize, read the computed geometry — before it becomes a finding. +- **Read the tool's evidence, not its generated description.** A summary, alt text, an accessibility tree + label or a snapshot title is the tool's account of the page, not the page. Base every finding on what you + can see or query directly, and quote that in the report. +- **Code is not a result.** Do not resolve a step by reading the component, the route table or the API + response. If the flow could not be completed on screen, the step failed regardless of what the source says. + +### Manual login + +**The agent can never type a password, in any environment** — not local, not a throwaway staging account, not +one whose credential is sitting in a file it can read. When a step reaches a login form, or a session +expires mid-walk, **stop before the form**, name the account role the plan needs, and ask the operator to log +in. Resume only once they confirm. Never read a credential out of a `.env`, a vault export, a password +manager, a fixture or the operator's earlier messages, and never echo one anywhere. + + + +{{QA_DESIGN_SOURCE_BLOCK}} + +## Phase 6 — Findings + +Every finding gets a severity and an attribution. Both are judgements a reader will act on, so make them +explicit rather than implied by tone. + +| Severity | Means | +|---|---| +| **Blocker** | The flow cannot be completed. Data loss, a hard error, an unreachable primary action. | +| **Major** | The flow completes but the outcome is wrong, or a supported path is broken. | +| **Minor** | Correct outcome, degraded experience — a bad message, a missing state, an awkward edge case. | +| **Cosmetic** | Visual or copy defect with no functional impact. | + +Every finding carries **introduced by this change: yes | no | unknown**. + +- **yes** — you have evidence it is new: the diff explains it, or you reproduced its absence on the base. +- **no** — you reproduced it on the base branch or in an untouched environment. +- **unknown** — you did not check. This is an honest, useful answer; a guessed **yes** turns a pre-existing + bug into a review blocker for the wrong author, and a guessed **no** ships a regression. + +Uncertainty belongs in the finding, not in a hedge around it: state what you saw, what you checked, and what +you could not check. + +## Phase 7 — PR deliverable + +Add or update a `## Q.A.` section on the PR, using `{{CODE_HOST}}`'s CLI or API. Update the existing section in +place on a re-run rather than appending a second one — two Q.A. sections make the reader guess which is +current. + +The section carries a checkbox list of what was walked (checked = passed, unchecked = failed or not tested, +each with a one-line result), the findings by severity with their *introduced by this change* verdict, the +environment and plan source, and a link to the full report. + +Under `data_sensitivity` of `client` or `phi`, **screenshots are not attached to the PR** and record contents +are not pasted into it. Link the gitignored report path instead, and say in the section that evidence is held +there because of this environment's data sensitivity — a reader who cannot find the screenshots must be able +to see why they are absent rather than conclude the run captured nothing. Under `phi`, the PR section and the +report both refer to records by opaque id only. + +If there is no PR for this branch, say so and stop after Phase 8 — do not open one. + +## Phase 8 — Report + +Write to `{{QA_REPORT_DIR}}/-.md`, creating the directory if missing, using exactly this +skeleton: + +```markdown +# QA report — — — + +**Run**: dev-desk-check · **Browser**: · **Viewport**: x · **Screenshots**: · **Dry run**: +**Plan source**: | use cases | ticket/description> +**Environment**: — · writes: · data sensitivity: +**Services**: , … (omit this line when no services are declared) + +## Walked + +| # | Step | Expected | Actual | Result | +|---|---|---|---|---| + +## Findings + +### Blocker +### Major +### Minor +### Cosmetic + +One bullet each: **** — what you saw and where. Introduced by this change: yes | no | unknown. + +## Not tested + +- — + +## Writes performed + +| # | Environment | What changed | Identifier | Reversal | Reverted? | +|---|---|---|---|---|---| + +## Evidence + +- "> +``` + +**A report listing only what passed is indistinguishable from one where nothing was tested. The *Not tested* +section is never omitted, and never empty when something was skipped.** Keep the empty severity headings too: +"no Blockers found" and "Blockers were never looked for" must not render the same way. + +The *Writes performed* table is the ledger from Phase 3b, carried through — appended as each write happens, +not reconstructed at the end. Offer to reverse everything still unreverted before closing the run, and record +the verdict per row either way. + +## Pitfalls + +- **Reading an edge-proxy 403 as downtime.** A WAF or edge proxy answers non-browser traffic with 403 or 503 + by design. Behind one, `curl` says nothing about the app's health and the browser is the preflight. Killing + a healthy run on that evidence is the same mistake as the next bullet, in a different costume. +- **Treating a missing probe binary as `down`.** A probe that cannot execute is **`unknown`**. A missing + client binary is a fact about this machine, not about the service. +- **Starting a second dev server on an occupied port.** It binds a different port, or silently loses to the + running one, and the whole run then tests a build nobody asked about. Probe first. Starting anything is + **local-only and asks first**: only in an environment named `local`, only where that environment declares a + start command, only with the operator's approval, and never a second copy. Everywhere else — and for a + non-local environment that happens to declare one — propose the command and wait. +- **Letting an earlier "go ahead" carry into a production write.** Approval is per environment and per scope. + Re-confirm, naming the environment, every time the write set changes. +- **Filing a layout bug that is really a stale screenshot.** Reproduce every visual defect on a fresh capture + before it becomes a finding. This is the single most common false positive in a browser pass. +- **Falling back to `{{DEFAULT_BRANCH}}` when the PR CLI errored** rather than when there is genuinely no PR. + "No PR exists" and "the CLI is broken or the token expired" are different answers, and only the first may + fall back. A blanket fallback builds the plan from the wrong diff with nothing on screen saying so. +- **Reporting the environment you meant instead of the one you used.** Every header line, every finding and + every ledger row names the environment actually driven. +- **Transcribing records into the report.** Under `client`, describe the shape of a record; under `phi`, use + opaque ids only. Never paste field values into a report, a PR comment, a commit message or a log line. + +## Verification + +The run is complete when: + +- **Every planned surface is either in *Walked* or in *Not tested* with a reason.** This covers passes as well + as surfaces: a design pass that was skipped, or that this project declares no source for, is a *Not tested* + entry with its reason — never silently dropped. +- **Every write in the ledger has a reversal recorded and a reverted verdict** (`yes`, `no — `, or + `n/a`). No write appears in the report that was not in an approved authorization block, and no approved + write is missing from the report. +- **The report exists** at `{{QA_REPORT_DIR}}`, follows the skeleton, and states the browser tool, viewport + and environment actually used. +- **The PR carries a `## Q.A.` section** reflecting this run — or the report says explicitly that no PR + exists. +- **Every finding carries a severity and an *introduced by this change* verdict**, and every visual finding + was reproduced on a fresh capture. diff --git a/skills/vinta-derive-skills/resources/qa-review-template.md b/skills/vinta-derive-skills/resources/qa-review-template.md new file mode 100644 index 00000000..7e8001ee --- /dev/null +++ b/skills/vinta-derive-skills/resources/qa-review-template.md @@ -0,0 +1,290 @@ +--- +name: qa-review +description: Walk a feature in {{PROJECT_NAME}} in a real browser and write up what you find, for product managers and designers who have no code checkout. Takes a use case, a ticket, or a plain description — never a diff. Covers {{QA_ENVIRONMENT_NAMES}}. Confirms the app is reachable, walks the flow as a real user would, compares against the design where one exists, and produces a shareable report of what worked, what did not, and what could not be tested. Use when someone says "check the new flow works", "QA this before we announce it", "does this match the design", or "walk through this feature and tell me what's broken". +--- + +# QA review + +Someone needs to know whether a feature in {{PROJECT_NAME}} actually works before it is announced, demoed or +put in front of a client. This skill walks the app in a real browser the way a person would, one step at a +time, and writes up what happened: what worked, what did not, and what could not be checked. It is for the +people who judge the product by using it — product managers, designers, support leads — not for the people +who read its code. The job is to report, never to fix: a finding that lands in a ticket with what was seen, +where, and what was expected is worth more than a guess at the cause. + +## What this file is + +This file is self-contained. Everything it needs is written into it: which environments exist, where they +are, which account to use, what may be changed and what may not, and what to compare against. There is +**nothing here that needs a code checkout, a terminal, or access to the project's code**, and nothing to install. + +What is written below is a snapshot taken on the day this file was generated. When a fact in it looks wrong — +an address that has moved, an account that no longer exists — do not work around it: ask +{{QA_ESCALATION_CONTACT}}, who is also who to ask whenever a review is blocked, and see **Where this file came +from** at the end for a fresh copy. Being blocked is a normal outcome; it gets recorded, not solved. + +## Before you start + +You need three things and only three: + +1. A browser. +2. An account on the environment you are going to test. **You sign in yourself** — see Step 3. +3. The thing you are checking: a use case, a ticket, or a description of the feature in your own words. + +## Before anything else — the browser itself + +Walking a feature means really opening a window on the app, clicking through it and reading what comes back. +So your assistant needs some way to do that itself: open a page, click, type, and look at what appears. + +Settle that before anything below. **If it has no way to open and click through a real browser window, this +review cannot happen.** It should say so plainly and stop — not describe what a screen probably looks like, +not read the product's insides and report on those, and not ask you to click everything and narrate it back. +None of those is a review, and a review nobody walked is worse than no review. Tell +{{QA_ESCALATION_CONTACT}}: someone has to set your assistant up so it can use a browser. A browser it *can* +open that then misbehaves partway through is a different problem — Step 5 covers that. + +## Step 0 — What are you checking, and where + +Use `AskUserQuestion` for the fixed choices, in **one batch**, and ask only what is still open — if the +person already named the environment or handed over a ticket, take it and do not ask again. + +- **Environment** — the options are exactly the names in the table below. Never invent one, and never point + the browser at an address that is not in the table. +- **What to check** — `A use case`, `A ticket`, `A description of the feature`. +- **Screenshots** — `On` / `Off`. **Always asked**, even when the answer seems obvious. A review that + silently captured nothing leaves a report nobody else can check, and one that silently captured everything + can put real records into a picture that then gets shared. Either way, Step 4's rules about what may appear + in a report apply to every capture. + +Anything genuinely open — which ticket, which account, which part of the flow matters most — ask as plain +prose rather than squeezing it into fixed options. + +{{QA_ENVIRONMENTS_TABLE}} + +## Step 1 — Get the steps + +{{QA_USE_CASES_BLOCK}} + +Then, whatever the source: **state the steps and the expected outcome back before walking.** A numbered list, +one line per step, each with what should happen if the feature works. Wait for a yes before opening anything. + +When the steps came from a description rather than a written use case, that restatement is the only place +"expected" ever gets pinned down, so say which parts you worked out yourself. A review that decides what +"expected" meant after seeing the screen confirms whatever it happens to find. + +## Step 2 — Check the app is up + +**Browser only. There is nothing here to start, and nothing to install.** + +Open the chosen environment's address from the table above. Before walking anything, confirm the app really +loaded: an error page, a holding page from whatever sits in front of the app, and a blank white page all +arrive quickly and all look like "the page loaded". + +If the page does not load, **reload it once** — one slow or half-finished first attempt is not an app that is +down, and reloading is the only recovery there is, since nothing here gets restarted. + +If it fails the second time too, **that is the result.** Write down the address you opened, the time and what +both attempts showed, then stop and tell {{QA_ESCALATION_CONTACT}}. Do not move to a different environment to +get something on screen — that makes it a different review, and it goes back to Step 0. + +## Step 3 — Sign in + +{{QA_AUTH_BLOCK}} + +**You sign in yourself.** It is your own account, so this is simple: when a step reaches a sign-in screen, or +the app signs you out partway through, the agent stops before the form and waits for you. It never types a +password, in any environment, and never reads one out of a file, a note or an earlier message. + + + +{{QA_FEATURE_FLAGS_BLOCK}} + +## Step 4 — Before you change anything + +{{QA_WRITE_POLICY_BLOCK}} + +{{QA_DATA_SENSITIVITY_BLOCK}} + +Before anything is saved, created, sent or deleted, list every such action the walk needs. Each entry says +what it changes, how you will find it again afterwards, and exactly how it gets undone. Then get a yes that +**names the environment** — an approval given earlier, or for a different environment, does not carry. If the +walk turns out to need a change nobody listed, stop and ask before making it: a yes for one change is not a +yes for another. + +- **Only make a change you can undo yourself, through the app.** The full write policy above still applies, but a reviewer cannot run a revert script. Anything you cannot undo through the screen in front of you is not yours to authorise — write it down as "this needs an engineer" and move on. +- **Blocked is a result, not a failure.** You cannot turn a feature on. "Blocked by the setting — ask an engineer to enable it" is the correct outcome, recorded under *Not tested*, not reported as broken. +- **Use obviously fake values.** Never a real person's details as input. Name test records so anyone finding them later knows what they are — for example, a title ending in "— QA, do not process". + +Keep the list as you go, not at the end — a review that stops halfway should still say what it changed. + +## Step 5 — Walk it as a user + +Walk the steps in order, one at a time. For each one, record what you did, what you expected, what actually +happened, and the result. Never click through five things and then read a single screenshot backwards. + +### When the browser only half-works + +Browser tooling goes wrong partly far more often than it goes wrong cleanly, and a partial failure looks +exactly like a bug in the product unless you are watching for it. + +- **If the screen you are looking at might be out of date, refresh and confirm before you click on.** A + picture taken before your last action tells you where things used to be, and every click aimed from it is a + guess about a page that has moved on. +- **Confirm every change of state you then rely on.** After moving to a new page, saving, opening a dialog or + switching tabs, check the new state is really there before acting on it. A click that reported success is + not proof that anything landed. +- **If a tool fails twice, stop using it rather than fighting it.** One glitch is noise; twice is broken. Get + the same evidence another way — reload the page, capture it differently, read the screen yourself. Retrying + the same failing thing only piles up uncertainty about what you are looking at. +- **A capture that will not finish is not a page that has stopped.** When taking a picture hangs, that says + nothing about the app. Find out another way whether the page is still alive before reporting it as stuck. + +### Judge what you can see + +- **Rule out your own screen or connection before filing something as broken.** Overlapping text, cut-off + boxes, zero-height gaps, missing images and unstyled text are the classic marks of a picture taken while + the page was still drawing. Reproduce it — reload, resize, capture again — before it becomes a finding. +- **Trust the screen, not a description of it.** A summary or a label attached to a capture is somebody's + account of the page, not the page. Base findings on what you can see, and quote it in the report. +- **"It is there" is not "it works".** If the flow could not be completed on screen, the step failed. +- **Say which window size you used.** A layout that breaks only at one width and a layout that is simply + broken look identical in a picture with no stated size. + +## Step 6 — Compare against the design + + + +{{QA_DESIGN_SOURCE_BLOCK}} + +When you compared against exported images rather than the live design file, **say so in the report** — +exports can be older than the design, so a difference may mean the export is out of date rather than the app +being wrong. + +If there was no comparison — no design, no access, no time — it goes under *Not tested* with that reason, +never silently dropped. + +## Step 7 — What you found + +Every finding gets a severity and a design verdict. Both are judgements someone will act on, so state them +rather than leaving them to the tone of the writing. + +| Severity | Means | +|---|---| +| **Blocker** | The flow cannot be finished. Lost work, a hard error, a main action you cannot reach. | +| **Major** | The flow finishes but the outcome is wrong, or a supported path is broken. | +| **Minor** | Right outcome, worse experience — a confusing message, a missing state, an awkward edge case. | +| **Cosmetic** | It looks or reads wrong, and nothing is broken. | + +Every finding also carries **matches design: yes | no | n/a**. + +- **yes** — you compared it against the design and it matches. +- **no** — you compared it and it does not. Name the part that differs. +- **n/a** — there was nothing to compare against, or you did not compare. Honest and useful; a guessed + **no** sends someone to change a screen nobody asked about, a guessed **yes** lets a real difference through. + +Uncertainty goes inside the finding, not in a hedge around it: what you saw, what you checked, what you did not. + +## Step 8 — Share it + +{{QA_ARTIFACT_DELIVERABLE_BLOCK}} + +**This never touches a pull request. The report goes wherever the team reads it — a ticket, a channel, a +shared link.** Those destinations are bounded by the choice above and never widen it: where a shareable page +is not on offer for this environment, "a shared link" is not on offer either — the report stays text you hand +over directly, however much easier something circulable would be, because of the records this environment +holds. + +## Step 9 — The report + +Write it for someone who was not there and who will read it a week later. Use exactly this skeleton: + +```markdown +# QA review — — — + +**Checked by**: · **Screenshots**: +**What was checked**: +**Environment**: —
+ +## Walked + +| # | Step | Expected | What actually happened | Result | +|---|---|---|---|---| + +## Findings + +### Blocker +### Major +### Minor +### Cosmetic + +One bullet each: **** — what you saw and where. Matches design: yes | no | n/a. + +## Not tested + +- — + +## Changes made + +| # | What changed | How to find it | Undone? | +|---|---|---|---| + +## Evidence + +- "> +``` + +**A report listing only what worked is indistinguishable from one where nothing was tested. The *Not tested* +section is never omitted.** Keep the empty severity headings too: "no Blockers found" and "Blockers were +never looked for" must not read the same way. + + + +When a report is kept as a file rather than shared as a link, where it belongs depends on where you are +working. Inside a copy of the project, it goes in `{{QA_REPORT_DIR}}` — the folder set aside for exactly this, +and deliberately kept out of what the project shares. Without a copy of the project, which is the normal case +for whoever holds this file, **do not create that folder.** Keep the report wherever you can hand it straight +to whoever asked for the review, and say in your message where you put it. + +## Pitfalls + +- **Filing a layout bug that is really an out-of-date picture.** Reproduce it on a fresh capture first — the + single most common false alarm in a browser pass. +- **Letting an earlier "go ahead" carry into a change somewhere real.** Approval is per environment and per + list of changes — ask again, naming the environment, whenever either changes. +- **Reporting the environment you meant instead of the one you used.** Name the one actually opened. +- **Copying record contents into the report.** Describe the shape of what you saw — "a client record with + no billing address" — rather than pasting field values into a report, a ticket or a message. +- **Calling something broken because you could not switch it on.** That is *Not tested*, with who to ask. +- **Working around a blocker to keep the review moving.** A different environment, a different account or a + setting somebody switched on for you gives a clean report about something nobody will ship. Record what + blocked it, what you tried, what you need. + +## Verification + +The review is finished when: + +- **Every step is either in *Walked* or in *Not tested* with a reason** — including the design comparison. +- **Every finding carries a severity and a matches-design verdict**, visual ones re-checked on a fresh capture. +- **Every change you made is listed** with how to find it and whether it was undone. +- **The report follows the skeleton** exactly, and names the environment actually opened. The window size + goes wherever it matters — in the step or the finding it affected, as Step 5 asks — never as an extra + header line the skeleton does not have. +- **Nothing in the report is a real person's details** or the contents of a real record. + +## Where this file came from + +{{QA_REVIEW_PROVENANCE_BLOCK}} diff --git a/skills/vinta-update-project-skills/SKILL.md b/skills/vinta-update-project-skills/SKILL.md index ac1d7d68..203bff63 100644 --- a/skills/vinta-update-project-skills/SKILL.md +++ b/skills/vinta-update-project-skills/SKILL.md @@ -69,6 +69,8 @@ For each project skill, decide its bucket: For `templated` and `stack-derived`, in-place diffing is misleading because the project version contains interpolated values (commands, branch names, etc.) that won't appear in the template. The right refresh path for those is to re-run [vinta-derive-skills](../vinta-derive-skills/SKILL.md) so the user can re-supply the inputs. +**`qa-review` needs re-rendering on a config change, not just a template change.** Unlike every other templated skill, its value depends on facts baked into the body — environment addresses, which account role to use, the design source, who to ask when blocked — because its holder has no repo to look them up in. When `skills.dev-desk-check` in `.vinta-ai-workflows.yaml` has changed since the file was rendered, the project's copy is stale even though the upstream template has not moved. The rendered body carries a generation date and source commit in its closing section; compare those against the config's last change and surface `qa-review` as **regenerate needed** when the config is newer. Same for `dev-desk-check`, though a stale copy there is less costly — its reader can check the config themselves. + ### 4. Diff verbatim skills For each `verbatim` skill: