Skip to content

feat(user-experience): add the user-experience plugin - #6669

Merged
kyle-sexton merged 21 commits into
mainfrom
feat/user-experience
Oct 9, 2026
Merged

kyle-sexton merged 21 commits into
mainfrom
feat/user-experience

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Closes #6306

Summary

Adds the user-experience plugin: a front door for UX work on the app being built, at any stage from an idea with no code to a legacy app. /user-experience:shape detects the app's stage and the project's own research, personas and analytics, states that context, then chains four model-invoked job skills through the Skill tool: plan-user-research → synthesize → structure → evaluate. /user-experience:setup checks and writes the team file. Every deliverable carries an evidence label (evidence-based, assumption-based or mixed), sources inline, and an AI-use disclosure where the ICC/ESOMAR code calls for one.

It is the first implementer of the routing-as-data team layer (<home>/user-experience.yaml), and moves installed() into a shared lib/installed.mjs that user-interface now reads too.

Fix

  • Plugin (plugins/user-experience/, version 0.1.0): skills shape, plan-user-research, synthesize, structure, evaluate, setup; a fresh-context evaluator agent (Read, Grep, Glob only, effort: high); nine reference files plus deliverable.md; routing.json (19 rows over nine jobs) with its schema; scripts/detect.mjs.
  • Team file reader: one teamSurface() resolver (convention-home pointer, default docs/conventions), a block-form YAML subset port of plugins/code-metrics/scripts/yaml_subset.py, merge keyed by job and id with disable and deny as unions, and two named policy rules in scripts/lib/team-policy.mjs that apply docs(conventions): limit which route rows a routing-as-data team file may add #6642's limits now (added ids must match a bundled row or an installed plugin or skill; no new kind: tool rows). A missing, malformed or unreadable file falls back to the built-in routes, names the file, says overrides and the deny floor were not applied, and suggests /user-experience:setup. Path values must stay inside the project, symlinks included.
  • Shared installed(): canonical lib/installed.mjs, generated copies in both plugins via scripts/shared-copies.txt; user-interface switches with no behavior change (lazy readers keep claude mcp list after the plugin list parses) and gets a patch fragment.
  • Cross-plugin: /planning:design points at /user-experience:shape and /user-interface:design; /planning:brainstorm's ## Next names /user-experience:shape; the routing-as-data Adopters table records user-experience (1.0.1); the plugin philosophy's pinned-agents list names the evaluator.
  • Release mechanics: main moved every existing plugin to changelog fragments (chore(release): move the remaining 68 plugins to changelog fragments #6661) while this branch was open, so user-interface's bump became a fragment. user-experience is the one plugin not in scripts/fragment-plugins.txt: listing it fails FRAGMENT-MODE RELEASE on its initial 0.1.0 heading. Follow-up: add it to the list after its first release.
  • Catalog: the marketplace entry leaves defaultEnabled unset, like user-interface, so cloud sessions install the plugin (user decision).

Verification

  • Tests: node --test 'plugins/user-experience/tests/*.test.mjs': 117 pass, 0 fail on 1d5a3fca3f; user-interface's tests unchanged and passing; lib/installed.test.sh passes. Full scripts/run-plugin-tests.sh --jobs 4 after merging main: every suite passed except three this branch does not touch, each failing on this machine's environment: scripts/check-read-caller-keys.test.sh (the runner-policy yaml package is not installed: npm ci --prefix .github/standards/runner-policy), scripts/hook-census.test.sh and scripts/selection-audit.test.sh (strace missing). CI runs them with their dependencies.

  • Gates after merging main (all exit 0): validate-plugins.sh --only "user-experience user-interface planning", sync-shared-copies.sh --check and --check-bump origin/main, check-cross-plugin-source-drift.sh --check, check-skill-leaf-names.sh --check, check-changelog-parity.sh --check and --check-bump origin/main, check-changelog-fragments.sh --check-required origin/main, check-declared-prerequisites.mjs, generate-catalog.mjs --check, generate-cheatsheet.mjs --check, sync-plugin-options-docs.py --check, check-changed-skills.sh origin/main (8 checked, 0 failed), check-skill.sh --require-evals (6 passed), listing budget 1990/8000 chars, both untrusted-content conformance greps (7 files each).

  • Runtime probe: claude -p --plugin-dir from a clean repo outside this one, with --setting-sources project,local: user-experience:shape fired before user-experience:plan-user-research; the isolation question answered ABSENT.

  • Each phase passed a fresh-context phase verifier; the security-sensitive team-file reader was re-probed (symlink and dangling-link escapes dropped, hostile YAML parsed in under 6 ms, path-like detects never stat'd).

  • Eval suite result (run-3, head 86885937af, Claude Code 2.1.296, model claude-opus-5-5, judge Sonnet, --runs 3, with-versus-without ablation): run-validity.py: verdict: VALID (warnings: judge votes, ceiling): 126 of 126 traces, no permission denials, skill fired in all 57 should-trigger with-arm runs; 5 split judge votes; five cases already at 1.00 without the plugin are excluded from the delta. Noise report: delta (with minus without), paired over 21 cases, +0.33, 95% interval +0.17 to +0.50, so the interval excludes 0. Per use case (judge-grader delta, skill-fired is an indicator, not scored; per-case judge agreement):

    Case Delta Judge agreement
    uc1-idea-who-is-it-for +0.56 unanimous 5/6 and 6/6 runs
    uc2-plan-interviews +0.67 6/6, 6/6
    uc3-signup-first-success-flow +0.44 6/6, 6/6
    uc4-test-prototype +0.33 6/6, 6/6
    uc5-structure-navigation +0.67 6/6, 6/6
    uc6-checkout-drop-off +0.11 4/6 (fair-choice), 6/6
    uc7-tickets-and-survey +0.67 6/6
    uc9-cancel-flow +1.00 5/6, 6/6
    uc12-legacy-redesign +0.44 6/6, 6/6

    How it got here: run-1 was INVALID (baseline permission denials in team-file-degrade; prompt fixed in 99fdb37be4); run-2 (Haiku judge) was VALID but uc3 scored −0.22; diagnosis showed the uc3 judges were uncalibrated, so both got calibration samples (b9fb35bbb8→e7d5e20622, 86885937af), the skills now put nothing before the record header, and the judge moved to Sonnet, where both uc3 judges calibrate at 18/18 (100%) against the 90% bar (Haiku: states-and-exits 66.7%). The rubrics' PASS and FAIL bars were not loosened.

  • Eval suite (21 cases, read-only: Read, Glob, Grep, Skill): run outside Claude Code (below). The session's worktree guard refuses any command containing eval, and the CLI refuses a plugin inside a repository with many linked worktrees (docs(evals): document that plugin eval refuses a plugin inside a repo with many linked worktrees #6374). The three detect-dependent cases (project-first, team-file-degrade, nothing-installed-degrade) grade shape's degraded path; node:test covers detect itself. The pass gate for uc1-uc7, uc9 and uc12 is a VALID run with a judge-grader delta above 0 per case (skill-fired at weight 0.1 there), noise verdict reported beside each. Command to run outside Claude Code:

    copy="$(mktemp -d)" && rsync -a --exclude .git --exclude 'evals/results' <repo>/plugins/user-experience/ "$copy/" && claude plugin eval "$copy" --trust-plugin --scaffold --keep-temp --runs 3 --threshold 0.8 --max-cost-usd <your ceiling> --no-publish --json <repo>/plugins/user-experience/evals/results/run-1.json

    then python3 plugins/evals/skills/plugin-eval/scripts/run-validity.py plugins/user-experience/evals/results/run-1.json --runs 3 and python3 plugins/evals/skills/plugin-eval/scripts/noise-report.py plugins/user-experience/evals/results/run-1.json --threshold 0.8 --interval-method normal --grader-agreement.

Review on the ready head

  • Both Claude CI lanes ended green without posting findings (each stopped with "post findings" unticked), so local stand-ins ran: /review:code-review (6 findings) and a security review (5 findings plus 1 LOW earlier). All were fixed in adfa43fa8e (YAML nesting limit) and 1d5a3fca3f: a team row reusing a bundled id may not change kind, detect or account (new named rule bundled-id-keeps-kind-detect-account), presence is per row, team paths under .claude or .git (as written or through a symlink) are rejected, the job skills frame the team file and its warnings as data and ask before sending project files to an MCP or plugin route, setup refuses a home resolving outside the project, YAML quote and duplicate-key-line fixes, and eval grader fixes. 117 node tests pass.
  • Codex P2 (job skills lacked the installed: null skill-listing fallback): fixed in be1f68c2a4, thread resolved.
  • Outcome verification (/verification:confirm, fresh context): every Brief criterion implemented as written except the eval baseline (not run, see above) and ci-status on the ready head.

Follow-ups not filed here

  • plugins/code-metrics/scripts/yaml_subset.py, which yaml-subset.mjs ports, likely has the same apostrophe-comment and duplicate-key-line bugs.
  • Team research_paths still accept other dotfiles such as .env.
  • Add user-experience to scripts/fragment-plugins.txt after its first release.
  • plugins/evals/skills/plugin-eval/scripts/calibrate-judge.py yaml_quoted() writes a literal --- when a sample contains a markdown table or rule; the CLI's frontmatter regex (/^---\s*\n([\s\S]*?)---\s*\n?/, unanchored) then truncates the frontmatter and the case silently fails to load. validate-cases.py split_frontmatter only closes on a whole --- line, so it passes these cases.

Findings recorded for review

  • Reference-file evidence: four verifier rounds; no claim is presented as verified that is not. Remaining minor qualifier gaps: research-methods mixing clause cites digital.gov (defines the split only), synthetic-user line lacks "market-research bodies, not UX-specific"; journeys point-2 paraphrase broader than the claim, blueprint line positions unlabeled; flows-ia click-count sentence is a 2026 excerpt qualifier; evaluation model set narrowed and one Basis line off the three outcomes; discovery-phase F2 UK-government qualifier only in the pointer, Torres lists five categories; measurement NPS scope and verifier caveat.
  • Eval graders that may not separate the arms or are missing: project-first cites-project-evidence, uc5 validation-method, nothing-installed names-access, team-file-degrade names-team-file, uc7 no-identifying-data; no AI-disclosure grader in uc2/uc4 and no evidence-label grader in uc3/uc5/uc9/uc12; four evidence-label regexes depend on the model reading reference/deliverable.md.
  • The evaluator agent has one dispatch site (accepted at design).
  • The pinned-agents paragraph already omitted review brief-reviewer and performance go-faster-sweeper before this branch.

Related

🤖 Generated with Claude Code

kyle-sexton and others added 15 commits October 9, 2026 08:54
…research and detect

Walking skeleton for the user-experience plugin: manifest, marketplace
entry, the shape front door chaining to plan-user-research through the
Skill tool, detect.mjs with the shared lib/installed.mjs canonical and its
generated copy, prerequisites, seed routes, the deliverable envelope and
the fires-shape eval case.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…b/installed.mjs copy

user-interface now imports installed() from its generated copy of the
repo-root lib/installed.mjs canonical, shared with user-experience. No
behavior change: the plugin and MCP lists are passed as lazy readers, so
claude mcp list still runs only after the plugin list parses.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Full routing-as-data rows for the nine UX jobs (19 rows: own
user-interface rows confirmed, Anthropic design and product-management
skills unconfirmed, account-bound research-tool and analytics routes
deferred until install-tested), each external row with pointer, as_of and
recheck. Contract tests port the user-interface rules keyed by job.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…s and degradation

detect.mjs reads <home>/user-experience.yaml through one teamSurface()
resolver (the vendored convention-home resolver, --root <project>), parses
it with a block-form YAML subset port, merges team rows keyed by job and
id with disable and deny as unions, and enforces two named policy rules in
scripts/lib/team-policy.mjs: added ids must match a bundled row or an
installed plugin or skill, and no new kind: tool rows. A missing,
malformed or unreadable file falls back to the built-in routes and says
so. Path values must stay inside the project, symlinks included; unknown
keys drop with a warning; parsing is linear on hostile input.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Nine reference files the job skills load on demand: research methods,
synthesis, journeys, flows and IA, the discovery phase, evaluation,
measurement, first run and AI features, plus the shared
recommendation-basis copy. Each claim is either a verifier-passed research
claim with its qualifier or labeled judgment; volatile upstream specifics
are pointer records with an as-of date and a recheck trigger.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
plan-user-research (study plans, discussion and facilitator guides,
usability test scripts), synthesize (themes as AI-suggested codes for a
named analyst, proto-personas, JTBD by the team's school), structure
(flows and IA as mermaid flowcharts, journeys as tables) and evaluate
(evaluation plans, expert review, measure selection), each with its eval
set, plus a fresh-context evaluator agent caged to Read, Grep and Glob.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…r-interface skills

/planning:design now names /user-experience:shape for user research,
flows and information architecture and /user-interface:design for look
and response, each when enabled; /planning:brainstorm's Next gains
/user-experience:shape.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The routing-as-data Adopters table gains user-experience (group field
job, the first team-layer implementer, applying the two #6642 limits),
released as 1.0.1. The plugin philosophy's pinned-agents list and per-pin
table name the user-experience evaluator at effort high.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
shape now states the app context (stage with signals, audience,
evidence, team settings) and chains plan-user-research, synthesize,
structure and evaluate through the Skill tool, passing each the route
jobs it reads, asking for access it lacks and degrading without node or
the claude CLI. setup checks and writes the team file in block form under
the convention home. prerequisites cover every skill that runs detect;
the README documents the skills, the evaluator agent and the team file.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Twenty more claude plugin eval cases (21 with fires-shape): use cases
uc1-uc7, uc9 and uc12, a fires case per job skill, quiet cases for
UI-only and DX-only asks, and behaviour cases for project-first evidence,
team-file degradation, nothing-installed degradation, assumption labels
and an embedded instruction. Every case is read-only (Read, Glob, Grep,
Skill); the use-case deltas come from judge graders, with skill-fired at
low weight. Six cases need --scaffold.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Main moved every existing plugin to changelog-fragment mode (#6661), so
the 0.2.1 hand bump from the installed() refactor becomes a patch
fragment under .changes/user-interface/ and plugin.json returns to 0.2.0.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…d changelog

The plugin.json description and the 0.1.0 CHANGELOG entry still described
the Phase 1 skeleton; they now name the full chain, the setup skill, the
evaluator agent, the reference files, the route rows, the team-file
reader and the eval suite. The catalog is regenerated.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…with its line

The YAML subset parser had no depth limit: empty sequence items at one
indent nest without indenting, so a long run of them overflowed the stack
and the team file degraded with no line number. Nesting past 64 levels
now raises the parser's own line-bearing error.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
… scaffold

The repository's typos config is en-US, so "neighbour" fails the typos
step; the scaffold now says "neighbor" (shfmt also tidied its redirects).

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton marked this pull request as ready for review October 9, 2026 14:49
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-09T14:54:59.203024Z 04ce457 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@claude

claude Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 3m 22s —— View job


Security review in progress

  • Gather context on PR feat(user-experience): add the user-experience plugin #6669
  • Manual pass over the security-relevant surface: team-file reader (detect.mjs's inside() path/symlink containment check), the hand-rolled YAML subset parser, team-policy.mjs's allowlist, the shared installed() detector, resolve-convention-home.sh, and the prompt-injection/"data, not instructions" framing in every skill body and the evaluator agent
  • Independent adversarial hunt pass to try to break the above (in progress)
  • Post final findings

@claude

claude Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 3m 48s —— View job


Code review in progress

  • Gather context (PR body, CLAUDE.md, REVIEW.md, skill instructions)
  • Run skip-gate checks — PR is open, ready for review, non-trivial (8382 additions); proceeding
  • Get PR diff (gh pr diff, since this checkout is shallow and lacks origin/main history)
  • Manually review the security-sensitive core: scripts/lib/team-policy.mjs, scripts/lib/yaml-subset.mjs, scripts/detect.mjs, lib/installed.mjs and its two generated copies — no issues found, path/symlink containment and the YAML subset parser look sound
  • Fan out hunters across remaining slices (skills/orchestration + evaluator agent; cross-plugin references + release mechanics; team-file reader tests/schema) — running
  • Adversarially verify candidate findings
  • Post inline comments for findings anchored to changed lines
  • Post final summary

@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review has reviewed this pull request through 5ede5c4; a later push is reviewed from there.

@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

claude-review has reviewed this pull request through 5ede5c4; a later push is reviewed from there.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 04ce4577ea

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread plugins/user-experience/skills/plan-user-research/SKILL.md Outdated
kyle-sexton and others added 2 commits October 9, 2026 10:58
…ob skills

When detect reports installed as null (no claude CLI), every route's
present is null, so a job skill invoked directly skipped installed skills
that the session can see. Each job skill's Route step now counts a skill
row as present when its id is in the session's skill listing, as shape
and prerequisites.json already describe.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
… review

The CI review lanes posted nothing, so local security and code reviews
stood in. Fixes:
- A team row reusing a bundled id may not change kind, detect or account
  (new named rule bundled-id-keeps-kind-detect-account), and presence is
  computed per row from its own detect, so a team file cannot point a
  route's research data at another connected server.
- Team paths under .claude or .git, as written or through a symlink, are
  rejected, so deliverables cannot become standing project rules; the job
  skills refuse such an output home too and ask before sending project
  files to an MCP or plugin route.
- The job skills frame the team file and the detect warnings that quote it
  as data and carry the no-node team-file degradation line; setup refuses
  a convention home that resolves outside the project.
- YAML subset: a quote opens only where a value starts, and a duplicate key
  in a sequence item is reported on its own line.
- Evals: the planted install points at example.invalid, team-file-degrade
  targets shape, and the quiet UI case's regex needs a recommendation.
- The plugin philosophy's pinned-agent counts match the agents (12 high,
  7 medium).

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@claude

claude Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 6m 59s —— View job


Code review complete

Incremental review of PR #6669 at 1d5a3fc, scoped to the 20 files changed since the last-reviewed commit 04ce4577 (read from .claude-lane/incremental.diff; gh pr diff used only for surrounding context).

  • Gather context (PR body, CLAUDE.md, REVIEW.md, .claude-lane/incremental.diff)
  • Skip-gate check — PR open, not draft, new commits since the last review; proceeding
  • Read the incremental diff and the in-scope files in full
  • Trace the substantive logic changes: detect.mjs's per-row presence(), the new bundled-id-keeps-kind-detect-account policy rule, yaml-subset.mjs's stripComment/mapping() rework, and the reserved-path (.claude/.git, symlink-aware) check
  • Cross-check each change against its accompanying test in detect.test.mjs / team-policy.test.mjs / yaml-subset.test.mjs
  • Check the SKILL.md prose fixes against the Codex finding and @kyle-sexton's own review comment
  • Post final summary

Findings

None. This incremental commit is a focused set of fixes responding to prior review (including the Codex comment on the installed: null fallback), and each one checks out:

  • detect.mjs / team-policy.mjs (presence fix): the old mergeRoutes computed one shared installed list for all candidate rows, so two different candidate rows sharing an id (different job, different detect) could leak presence from one to the other via Set-deduped ids. The new presence() suffixes each row's id with its array index before calling installed(), so presence is resolved strictly per-row-object. Traced through the new "a row is admitted and marked present by its own detect, not by another row with its id" test (detect.test.mjs) and it holds.
  • New bundled-id-keeps-kind-detect-account rule (team-policy.mjs:317-327): correctly blocks a team row from repointing a bundled id's kind/detect/account while still allowing rank/status/pointer overrides; verified against all cases in team-policy.test.mjs.
  • yaml-subset.mjs stripComment (yaml-subset.mjs:34-49): now only opens a quote at a genuine value-start position, fixing the it's fine # comment false-positive comment-strip; escape handling (\", doubled '') traced consistently with unquote().
  • mapping(indent, result) refactor (yaml-subset.mjs:188,241): fixes a real bug — a duplicate key between a sequence item's dash-line entry and one of its indented continuation lines used to report the error at the dash line's line number instead of the duplicate's own line; now correct, matches the new test.
  • Reserved-path check (RESERVED/resolvesReserved/usable, detect.mjs:293-332): correctly drops research_paths/persona_paths/output_home values under .claude/.git, including via a symlink that resolves there; traced the symlink test by hand (loop termination is bounded since rel is already validated non-escaping by inside()).
  • SKILL.md changes (evaluate, plan-user-research, structure, synthesize): each now adds the installed: null → session-skill-listing fallback the Codex comment flagged was missing, matching the pattern shape/SKILL.md already had, plus a new MCP/plugin consent step before sending project file contents.
  • Eval asset changes (quiet-ui-only/answers-white, team-file-degrade/*, embedded-instruction/scaffold.sh): each narrowed regex/sample/prompt change is internally consistent — e.g. the new answers-white regex correctly no longer false-matches "Use black rather than white," and the embedded-instruction scaffold now points its injected command at example.invalid instead of a real-looking npm package name.
  • docs/plugin-philosophy.md and plugins/user-experience/README.md: doc-only updates; verified the agent counts ("Twelve"/"Seven") and the new policy-rule wording match the code exactly.

I was unable to execute node --test in this sandbox (commands were denied), so the above is a hand-traced static verification against the included unit tests, not an independently-run test pass — the PR body's own "94 pass, 0 fail" claim is author-reported and unverified by me here.

@claude

claude Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 4m 36s —— View job


Security review complete

Incremental scope (last reviewed 04ce4577): reviewed only what changed since then, per .claude-lane/incremental.diff, against /review:security-review's criteria.

  • Read the incremental diff and gather context (changed skill bodies, detect.mjs, team-policy.mjs, yaml-subset.mjs, tests, docs)
  • Check for instruction-surface deletions against the instruction exception register — none found; every change in scope adds a guardrail, none removes or softens one
  • Manual review pass over the security-relevant surface (team-file reader, YAML parser, team-policy allowlist, job-skill route gating)
  • Adversarial validation of candidate findings (producer ≠ verifier) — no candidates survived to need dispatch
  • Post inline comments for confirmed findings — none to post
  • Post final summary

No security issues found. This slice is itself security hardening, closing gaps this PR's own review history had already flagged:

  • scripts/detect.mjs: presence() now namespaces each row's id (`${r.id}\u0000${i}`) before calling installed(), so a team row's own kind/detect decides its present flag — not another row that happens to share its id across a different job. lib/installed.mjs's present() keys strictly off row.detect, so the namespacing doesn't disturb detection, only the id-collision surface. Covered by the new tests/detect.test.mjs case "a row is admitted and marked present by its own detect, not by another row with its id."
  • scripts/lib/team-policy.mjs:L42-L48: the new bundled-id-keeps-kind-detect-account rule stops a team row from re-using a bundled id while swapping its kind, detect, or account — closing the repointing path the rule's own comment names. The job-mismatch fallback (bundled.find((b) => b.id === row.id)) is sound given routing.json's invariant that a reused id keeps the same detect across jobs.
  • scripts/detect.mjs:L305-L331: teamSettings() now rejects research_paths/persona_paths/output_home entries that land under .claude/ or .git/, checked both as written and as resolved through an existing symlink (resolvesReserved), so a project symlink can't smuggle a team-controlled path into either directory. The containment logic composes correctly with the pre-existing inside() escape check it builds on.
  • Job skill bodies (plan-user-research, synthesize, structure, evaluate): now carry the installed: null → session-skill-listing fallback Codex flagged on the prior commit, and a new "name the route's detect and ask before sending it project file contents" gate before an MCP/plugin route (not a skill route, which stays inside this plugin's vetted routing policy) receives project data.
  • scripts/lib/yaml-subset.mjs's reworked stripComment quote-tracking and the sequence-item mapping() duplicate-key fix don't change the trust model: whatever string the parser produces is still re-validated by violations()/DETECT_NAME downstream, so a parsing edge case can misparse a value but can't bypass the schema/allowlist checks. New tests (yaml-subset.test.mjs) cover the comment/quote edge cases the refactor targets.

Nothing in this slice touches GitHub Actions configuration, so no zizmor-lane findings apply.

kyle-sexton and others added 4 commits October 9, 2026 13:05
… workspace

"Following our team's UX conventions" sent every no-plugin baseline run
searching the sandbox's own config and memory folders, which the eval
denies, so run-validity.py marked the whole run INVALID. The prompt now
says the conventions are in this repository.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ate the uc3 judges

The job skills let the stage line and tool or access notes open the
answer, before the record header deliverable.md requires first; they now
follow the record, and structure keeps the PRD's order in its diagram. The
uc3 judges get calibration samples (three must-pass and three must-fail
answers each), and structure-not-surface says the Brief's evidence header,
assumptions and tool notes are not surface; its FAIL clause is unchanged.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…a mislabeled pass

The CLI ends a case's frontmatter at the first '---' anywhere, so the
markdown table separators in nine calibration samples cut the generated
prompt.md short and those cases never loaded. The separators now use two
hyphens. One must-pass sample quoted button wording while saying wording
is later screen work; it now describes the prompt instead.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	.claude-plugin/marketplace.json
#	scripts/shared-copies.txt
@claude

claude Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 2m 26s —— View job


Security review complete

Invoked /review:security-review for PR #6669 at 5ede5c47 per the trigger's header (REPO/PR NUMBER/HEAD SHA).

  • Gather context (PR body, AGENTS.md, custom instructions, prior review history)
  • Skip-gate check — PR open, not closed; scoped the review to the true PR diff (origin/main...HEAD, matching the PR's own 8694/-104 stat) rather than the raw commit range, since the latter also pulled in unrelated commits merged from main
  • Identify what changed since the last completed security review (through 1d5a3fca3f): only 8 files — plan-user-research/synthesize/structure/evaluate SKILL.md ("nothing before the record header" ordering rule), one rule added to structure's diagram-ordering guidance, and team-file-degrade eval fixture/grader/sample tweaks. All purely additive (no deletion, narrowing, or softening of an existing instruction), and none touch the security-sensitive surface.
  • Independently re-verify the security-sensitive core unchanged since that review — read in full rather than taking prior review comments at face value:
    • scripts/detect.mjs — inside()'s per-segment symlink containment (L119-143) and teamSettings()'s reserved-path check (L293-332, written-form and resolved-through-symlink) both hold up; resolvesReserved's while (!existsSync(at)) at = dirname(at) loop is bounded since opts.project itself always exists.
    • scripts/lib/team-policy.mjs — the three rules (id-matches-bundled-or-installed, no-new-tool-rows, bundled-id-keeps-kind-detect-account) correctly block an unvetted id, a new kind: tool row, and repointing a bundled id's kind/detect/account.
    • lib/installed.mjs — spawnSync calls use fixed argument arrays, never team-file-derived strings, so no injection path even where shell: win is set for the Windows PATH-hijack guard; __proto__ as a team-file key becomes an own data property via Object.defineProperty, never touching the prototype chain.
    • scripts/lib/yaml-subset.mjs — recursion is bounded by MAX_DEPTH (64) in nested(); no regex in the module has nested quantifiers that could cause catastrophic backtracking on hostile input.

No security issues found. This synchronize event's own diff is non-security prose/test-fixture changes; the security-relevant surface (team-file reader, YAML parser, team-policy allowlist, shared installed() detector) is unchanged since the last completed review and holds up under independent re-reading.

Branch: feat/user-experience

@claude

claude Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 3m 23s —— View job


Code review complete

Incremental review of PR #6669 at 5ede5c4, scoped to the changes since the last-reviewed commit 1d5a3fca3f (git diff 1d5a3fc..HEAD), per /review:code-review's skip-gate ("this head already has a successful review that still applies" for everything before that point).

  • Gather context (PR body, CLAUDE.md, REVIEW.md, prior review history on this PR)
  • Skip-gate check — PR open, not draft, new commits since the last code-review pass (99fdb37, e7d5e20, 8688593, plus the merge commit 5ede5c4); proceeding
  • Get the diff (git diff 1d5a3fc..HEAD; filtered the diffstat to drop main-merge noise unrelated to this plugin — marketplace.json, docs/catalog.md, lib/context-zone.*, etc. — and confirmed no merge-conflict markers survived in plugins/user-experience/)
  • Review changed files for correctness and maintainability
  • Adversarially verify candidate findings
  • Post inline comments for findings anchored to changed lines
  • Post final summary

Findings

None. This slice is a small, targeted set of fixes responding to the author's own local review, and each one checks out:

  • evaluate/SKILL.md, plan-user-research/SKILL.md, structure/SKILL.md, synthesize/SKILL.md (each +4/+6 lines): adds "Nothing comes before the record's header block. Put the stage line and every note from Context and Route … after the record." This fixes the real ordering bug the commit message names — reference/deliverable.md's "Open the record with these lines, before any content" was being violated by the Context section's instruction to state the stage/evidence first. The new line doesn't contradict the earlier Context-section wording (which never specified ordering); it resolves the ambiguity in deliverable.md's favor. Checked all four skills land the new paragraph in the same place (right after ## Deliverable, before ## Gotchas/## Next) and with identical wording, so they stay interchangeable.
  • structure/SKILL.md Rules (new line): "Draw the flow in the order the PRD or code gives; put a proposed reorder under recommendations, not in the diagram." Consistent with the existing Rules bullet on reading the app's existing flow before proposing a new one.
  • evals/uc3-signup-first-success-flow/graders/structure-not-surface.md: adds "Evidence labels, an AI-use header, assumptions to test, recommendations and notes about tools or access are allowed and are not surface," leaving the FAIL clause (colors/fonts/pixel sizes/component styles) unchanged. Matches the new calibration samples below.
  • evals/uc3-signup-first-success-flow/samples/{states-and-exits,structure-not-surface}.json (new, 3 pass + 3 fail each): calibration samples for the uc3 judges. Hand-checked each pass/fail label against its grader text — the pass cases carry the full deliverable envelope (evidence line, AI-use, recommendations, assumptions table) with no visual styling, the fail cases each introduce real surface detail (hex colors, px sizes, named fonts, component styling). Labels are correct.
  • 8688593's fix to those same two sample files: replaced |---|---|---| markdown table separators with |--|--|--|, and reworded one pass-case line that had quoted literal button text ("Add your first recipe") into a paraphrase, matching the commit's stated rationale (the eval CLI apparently cuts a case's generated frontmatter at the first --- substring anywhere in the file, which a triple-dash table separator was tripping). That claim is about the closed-source claude plugin eval CLI, external to this repo, so I can't independently verify the parser behavior — but the fix itself (removing every 3-consecutive-dash sequence from the sample text) is a sound, low-risk way to satisfy it either way.
  • team-file-degrade/case.yaml: reworded the eval prompt from "our team's UX conventions" to "the team's UX conventions in this repository," per 99fdb37's stated reason (the old wording sent the no-plugin baseline run searching the sandbox's own config/memory, which the eval denies, causing an INVALID run). Self-contained, no interaction with other eval files.

Nothing in this slice touches runtime logic (detect.mjs, team-policy.mjs, yaml-subset.mjs), only skill prose and eval test fixtures, so the blast radius is limited to documentation correctness and eval calibration — consistent with the low-risk bar for this kind of change.

I could not run node --test or any linter in this sandbox (commands were denied), so this is a hand-traced static review, not an independently-run test pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: add the user-experience plugin for user research and strategy (/user-experience:shape)

1 participant