Repository navigation
Conversation
🤖 PR meta 🤖🎉 PropsIf you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message. To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. Props Bot commandsYou can trigger Props Bot actions by commenting on this PR:
Updated as activity occurs, without notifying anyone named here. Add the 📦 Bundle sizeSize Change: 0 B Total Size: 8.69 MB
🏁 Flaky testsSome tests passed with failed attempts. The failures may not be related to this commit but are still reported for visibility. See the documentation for more information. As a user I want to be able to create a navigation overlay for a specific navigation block in
|
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues block the changes introducing Storybook-derived context into review routing.
No open issues 🎉
What changed in this PR
Adds dynamically generated reference data to review-routing assessments, replacing the hard-coded design-system pattern list with names drawn from Storybook documentation.
Changes:
- Loads and validates optional
.mtscontext modules and includes their data in model requests and dry-run output. - Generates pattern names from Storybook and updates routing criteria.
- Adds test coverage, documentation, TypeScript configuration, and workflow checkout support.
| File | Description |
|---|---|
tools/review-routing/tsconfig.json |
Includes routing context modules in TypeScript checks. |
tools/review-routing/src/routing-rules.ts |
Loads and validates optional context modules. |
tools/review-routing/src/decision-api.ts |
Combines questions and context in structured instructions. |
tools/review-routing/src/cli.ts |
Displays structured instructions in dry-run output. |
tools/review-routing/src/__tests__/routing-rules.test.ts |
Covers context validation and loading. |
tools/review-routing/src/__tests__/decision-api.test.ts |
Covers requests containing context. |
tools/review-routing/README.md |
Documents context modules and execution requirements. |
.github/workflows/review-routing.yml |
Includes pattern documents in the sparse checkout. |
.github/review-routing/design-system-patterns.mts |
Extracts pattern names from Storybook headings. |
.github/review-routing/design-system-patterns.md |
Uses generated pattern data and the default threshold. |
.github/review-routing/design-system-maintenance.md |
Clarifies routing for dependency updates affecting behavior. |
Review effort: Balanced
There was a problem hiding this comment.
These TS files should ideally be in tools/ so that they are type checked.
I saw many other PRs tried to add node scripts in .github. So, I think we should ban any JS/TS files outside tools/ and packages/
There was a problem hiding this comment.
These files are type-checked. They're included in the include pattern for the @wordpress/review-routing package.
That being said, the fact that the package is breaking out of itself to cover these additional files probably isn't great. The whole point of separating the configuration from the tool was that the tool could consume any folder of configuration, and the routings relevant for CI workflows could be colocated to CODEOWNERS and the rest of the workflow setup.
There may be a couple alternatives:
- We could keep the routing files where they're at, and add a
tsconfig.jsonhere. That keeps them self-contained, but it goes against your own desire for banning files here, and it starts to litter the directory with non-configuration related files. Moving the folder could also cause some disruption since I've already been referencing it as a place people can contribute (Slack example). - Or, we could move the rules elsewhere, like the
tools/review-routingpackage itself, or their own workspace. This goes against the intended separation, though practically speaking I don't know that it matters so much, particularly since the tool itself is already internal.
There was a problem hiding this comment.
That being said, the fact that the package is breaking out of itself to cover these additional files probably isn't great. The whole point of separating the configuration from the tool was that the tool could consume any folder of configuration, and the routings relevant for CI workflows could be colocated to
CODEOWNERSand the rest of the workflow setup.
This isn't entirely true. The tool knows to look for .github/review-routing, and with how the tool dynamically imports these files it makes sense that it should be covered by the tool's own TypeScript configuration.
Now that still leaves the question of whether we'd be better off just putting the rules in tools/review-routing itself. I still see some advantage to it staying where it is in .github to be colocated to CODEOWNERS and other CI workflow configurations, and because it's primarily a GitHub concern in how reviews are assigned. It's not a hill I'd die on though, if we'd rather just have it be in tools/.
There was a problem hiding this comment.
Another aspect here is that if we want to make the review routing a "GitHub App" that can be installed to other WordPress repositories (some related Slack discussion), we need the configuration to live independently from the tool. Developing it as an app might pose some challenges around what these "dynamic context" scripts need to look like from a perspective of what the tool is expecting to be able to handle, but at the very least it seems more future-proof to keep the configuration in a place like where it's at today.
There was a problem hiding this comment.
Just to put a more definitive / conclusive point to the thread: I'd argue that this is fine as-is, and this is a "strong opinion loosely held".
| @@ -1,6 +1,5 @@ | |||
| --- | |||
| reviewer: [aduth, ciampo] | |||
| threshold: 0.8 | |||
There was a problem hiding this comment.
This is a good one to surface. In truth, I did have to lower the threshold here to be able to match expected pull requests. For example, #86490 stopped matching above 0.8 after the changes here, which was unacceptable to me as it was very much a good example of review routing working effectively, as I commented in #86490 (comment) .
My thinking with this change was: We only need 0.8 because the previous language was so broad that it would flag a lot of false positives, such that we needed to be particularly strict with the confidence of the decision model. With these changes now giving us more precision about what should be matched, we no longer need to worry as much about the false positives and can revert back to the default threshold.
Separately, I'll comb through my agent conversation for the raw data, as this was something I discussed quite intentionally with it.
There was a problem hiding this comment.
Here's the raw data from my agent:
Raw data (AI-generated)
Here's the data. I scored a fixed set of PRs with Jev (
jev-latest), two runs per PR for each version of the rule. All versions use thepatternslist from the new.mtsfile, except the trunk version.
PR Should match Trunk wording PR wording At 0.8 At 0.7 #85775 Quick edit Yes 0.96 0.93 ✅ ✅ #86725 Save & Submit docs Yes 0.93 0.92–0.93 ✅ ✅ #86490 Media editor card Yes 0.85–0.86 0.74–0.76 ❌ ✅ #84428 Copy bound blocks No 0.89 0.59 ✅ ✅ #85117 Pattern creation error No 0.58–0.60 0.34–0.36 ✅ ✅ #84389 Copy bound text No 0.74–0.75 0.17 ✅ ✅ #82247 Status tab counts No 0.13 0.12–0.13 ✅ ✅ #86827 Widgets settings label No 0.04 0.09 ✅ ✅ The "At 0.8" and "At 0.7" columns show whether the PR wording gets the expected result at that threshold.
- With the trunk wording at 0.8, Block Editor: Include resolved bound content when copying blocks #84428 was a false positive, and it scored higher (0.89) than Media Editor: Add a placeholder state with download and copy buttons #86490 (0.85). No threshold could separate them.
- With the PR wording, the negatives drop a lot (Block Editor: Include resolved bound content when copying blocks #84428 0.89 → 0.59, Block Editor: Check supported bindings when copying selected text #84389 0.75 → 0.17), but Media Editor: Add a placeholder state with download and copy buttons #86490 also drops to 0.74–0.76. At 0.8, we'd miss it. At 0.7, all three positives match and all five negatives don't.
To answer your question directly: I couldn't find a wording that keeps #86490 above 0.8 without also bringing #84428 back up. These are the other versions I tried:
Wording #86490 #84428 "implement one of the patterns in patterns" + composition sentence0.54–0.58 0.34–0.35 "implement one of the patterns in patterns"0.82 0.74–0.77 "build an interface that one of the patterns in patternsapplies to"0.85 0.79–0.81 "implement a documented design system pattern, as listed in patterns"0.78 0.77 PR wording: "applies to" + composition sentence 0.74–0.76 0.59 The composition sentence ("A pattern describes how to compose the interface components…") is what separates the two PRs. Without it, they score within about 0.05 of each other. It also lowers #86490 a bit, which is why the threshold needs to come down.
A caveat: the gap is real, but not wide. It runs from 0.59 (the highest negative) to 0.74 (the lowest positive). At 0.7, #86490 is only about 0.04–0.06 above the threshold, and scores vary by about 0.05 between identical runs. The set is also small (three positives), so the live results over the next few weeks will tell us more. If #86490-like PRs start to miss, 0.65 would put the threshold in the middle of the gap.
|
|
||
| The model then reads the question and the data as one object, `{ "question": "...", "stores": [ ... ] }`, so the question can name the field: "Does this change add an action to one of the stores in `stores`?" `--dry-run` prints this object. | ||
|
|
||
| The file runs with Node.js, from the base branch in CI, so it can use only Node.js built-in modules. To read a file of the repository, make its path from `import.meta.url`, and add its directory to the sparse checkout in `.github/workflows/review-routing.yml`. |
There was a problem hiding this comment.
Could we mention that a local --dry-run executes the current checkout's .mts files? Reviewers should inspect those files before running it on a contributor's branch.
There was a problem hiding this comment.
Clarified this a bit in bcd4fb8 . Arguably it should be expected that running anything from a branch should come with a caution about what the branch might contain, though I suppose it's a bit more surprising here: (a) things in .github are often more "configuration" than runnable code, and (b) --dry-run is usually assumed to be "safer".
| ); | ||
| const api = rules.find( ( rule ) => rule.id === 'design-system-api' ); | ||
|
|
||
| expect( patterns?.context?.patterns ).toContain( 'Copy to Clipboard' ); |
There was a problem hiding this comment.
Could we pass a fixture directory to loadRoutingRules with one rule that has a companion module and one that doesn't? That would test both cases without making a Storybook heading or the current setup of design-system-api part of the loader's contract.
There was a problem hiding this comment.
Could we pass a fixture directory to
loadRoutingRuleswith one rule that has a companion module and one that doesn't? That would test both cases without making a Storybook heading or the current setup ofdesign-system-apipart of the loader's contract.
Yeah I think that's better. I did that in 0ebe1d0 along with making the folder names follow our project conventions a bit better (do we have conventions?). My agent took an earlier request a bit too literally, as I'd discouraged it from writing temporary files in the test, but fixtures are a better happy medium between (a) files for tests and (b) not relying too heavily on something external that could change under the tests (like the actual live .github/review-routing contents).
|
|
||
| ### Reference data | ||
|
|
||
| The model knows only what the question and the pull request tell it. To give it reference data, such as a list of names that changes as the project grows, add a TypeScript module with the same name and the `.mts` extension next to the routing file. Its default export takes no arguments and returns an object: |
There was a problem hiding this comment.
Could we say that the function returns a plain object with JSON-compatible values and that question is reserved? For example, a rule that collects names in a Set needs to convert it to an array before returning it.
What?
Adds support for review routing configurations in
.github/review-routingto supplement assessment requests with additional, dynamically-generated context.Why?
To improve accuracy of results in a way which lowers overall maintenance burden.
The existing "design system patterns" routing configuration hard-codes the name of patterns. This increases maintenance burden as we add more design system patterns. Or, more likely, it risks falling out of sync. Reusing a single source of truth for patterns ensures that this is kept accurate and complete.
This is also well-supported and documented by decision models (see Jev's documentation on Noul "Structured instructions"), referencing "supplementary data" in a question.
This was mentioned as one of the original shortcomings of the initial implementation in #86325 as something to explore further: "generated context for rules like component or tokens lists". And the revisions to routing text follow from action items raised in #86677 (comment).
How?
The implementation is built with flexibility in mind: Any rule in
.github/review-routingcan optionally include an accompanying.mtsfile, and that file's default export function will be evaluated when performing the assessment. The object returned by the function is added to the question instructions.A note on security: The workflows associated with this assessment use
pull_request_target, which runs in the context oftrunkand not in the pull request itself. So scripts are only evaluated once they're merged totrunk. This also makes testing difficult here, as the CI checks in this pull request will not use the supplementary data.Testing Instructions
As mentioned in "How?", this is not directly possible to test until merged to
trunk, since it usespull_request_target.You can perform a dry-run locally:
Alternatively, I could add a temporary "demo" workflow as I've done in previous pull requests #86677 and #86449. I may look at testing this tomorrow.
Use of AI Tools
Researched and implemented in coordination with Claude Code + Opus 5.5.