Skip to content

Automated Testing: Add support for dynamic context to review routing - #86921

Open
aduth wants to merge 3 commits into
trunkfrom
update/review-routing-patterns-dependencies
Open

aduth wants to merge 3 commits into
trunkfrom
update/review-routing-patterns-dependencies

Conversation

@aduth

@aduth aduth commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

What?

Adds support for review routing configurations in .github/review-routing to supplement assessment requests with additional, dynamically-generated context.

Why?

To improve accuracy of results in a way which lowers overall maintenance burden.

The existing "design system patterns" routing configuration hard-codes the name of patterns. This increases maintenance burden as we add more design system patterns. Or, more likely, it risks falling out of sync. Reusing a single source of truth for patterns ensures that this is kept accurate and complete.

This is also well-supported and documented by decision models (see Jev's documentation on Noul "Structured instructions"), referencing "supplementary data" in a question.

This was mentioned as one of the original shortcomings of the initial implementation in #86325 as something to explore further: "generated context for rules like component or tokens lists". And the revisions to routing text follow from action items raised in #86677 (comment).

How?

The implementation is built with flexibility in mind: Any rule in .github/review-routing can optionally include an accompanying .mts file, and that file's default export function will be evaluated when performing the assessment. The object returned by the function is added to the question instructions.

A note on security: The workflows associated with this assessment use pull_request_target, which runs in the context of trunk and not in the pull request itself. So scripts are only evaluated once they're merged to trunk. This also makes testing difficult here, as the CI checks in this pull request will not use the supplementary data.

Testing Instructions

As mentioned in "How?", this is not directly possible to test until merged to trunk, since it uses pull_request_target.

You can perform a dry-run locally:

npm run --workspace @wordpress/review-routing-tools route -- --pr 86490 --dry-run

Alternatively, I could add a temporary "demo" workflow as I've done in previous pull requests #86677 and #86449. I may look at testing this tomorrow.

Use of AI Tools

Researched and implemented in coordination with Claude Code + Opus 5.5.

@aduth
aduth requested a review from ciampo November 16, 2026 21:33
@aduth aduth added the [Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests. label Nov 16, 2026
Copilot AI balanced review requested due to automatic review settings November 16, 2026 21:33
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

🤖 PR meta 🤖

Preview in WordPress Playground Preview in Storybook

🎉 Props

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: aduth <aduth@git.wordpress.org>
Co-authored-by: manzoorwanijk <manzoorwanijk@git.wordpress.org>
Co-authored-by: ciampo <mciampini@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.


Props Bot commands

You can trigger Props Bot actions by commenting on this PR:

  • @propsbot add <username> to give props to a GitHub user. Requires write access.
  • @propsbot remove <username> to leave a user out of the props. Requires write access.
  • @propsbot refresh to refresh this list. Adding the props-bot label works too.

Updated as activity occurs, without notifying anyone named here. Add the props-bot label to refresh.

📦 Bundle size

Size Change: 0 B

Total Size: 8.69 MB

bcd4fb8 Run

🏁 Flaky tests

Some tests passed with failed attempts. The failures may not be related to this commit but are still reported for visibility. See the documentation for more information.

As a user I want to be able to create a navigation overlay for a specific navigation block in /test/e2e/specs/site-editor/navigation-overlay-template-part.spec.js, passed after 1 failed attempt.
TimeoutError: locator.click: Timeout 10000ms exceeded.
Call log:
  - waiting for getByRole('region', { name: /(Editor publish|Save panel)/ }).getByRole('button', { name: 'Save', exact: true })

    at Editor.saveSiteEditorEntities (/home/runner/work/gutenberg/gutenberg/packages/e2e-test-utils-playwright/src/editor/site-editor.ts:48:5)
    at /home/runner/work/gutenberg/gutenberg/test/e2e/specs/site-editor/navigation-overlay-template-part.spec.js:91:4
should select synced pattern in /test/e2e/specs/editor/various/writing-flow.spec.js, passed after 1 failed attempt.
Error: expect(locator).toBeFocused() failed

Locator:  locator('[name="editor-canvas"]').contentFrame().locator('[data-type="core/block"]')
Expected: focused
Received: inactive
Timeout:  5000ms

Call log:
  - Expect "toBeFocused" locator('[name="editor-canvas"]').contentFrame().locator('[data-type="core/block"]') with timeout 5000ms
  - waiting for locator('[name="editor-canvas"]').contentFrame().locator('[data-type="core/block"]')
    14 × locator resolved to <div tabindex="0" role="document" draggable="true" data-title="Pattern" data-type="core/block" data-is-drop-zone="true" aria-label="Block: Pattern" id="block-df6a0e3c-f789-4df7-9ae5-536d77ae4207" data-block="df6a0e3c-f789-4df7-9ae5-536d77ae4207" class="block-editor-block-list__block wp-block is-selected is-reusable block-library-block__reusable-block-container wp-block-block block-editor-block-list__layout">…</div>
       - unexpected value "inactive"

    at /home/runner/work/gutenberg/gutenberg/test/e2e/specs/editor/various/writing-flow.spec.js:1464:5

bcd4fb8 Run

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues block the changes introducing Storybook-derived context into review routing.

No open issues 🎉

What changed in this PR

Adds dynamically generated reference data to review-routing assessments, replacing the hard-coded design-system pattern list with names drawn from Storybook documentation.

Changes:

  • Loads and validates optional .mts context modules and includes their data in model requests and dry-run output.
  • Generates pattern names from Storybook and updates routing criteria.
  • Adds test coverage, documentation, TypeScript configuration, and workflow checkout support.
File Description
tools/​review-routing/​tsconfig.json Includes routing context modules in TypeScript checks.
tools/​review-routing/​src/​routing-rules.ts Loads and validates optional context modules.
tools/​review-routing/​src/​decision-api.ts Combines questions and context in structured instructions.
tools/​review-routing/​src/​cli.ts Displays structured instructions in dry-run output.
tools/​review-routing/​src/​__tests__/​routing-rules.test.ts Covers context validation and loading.
tools/​review-routing/​src/​__tests__/​decision-api.test.ts Covers requests containing context.
tools/​review-routing/​README.md Documents context modules and execution requirements.
.github/​workflows/​review-routing.yml Includes pattern documents in the sparse checkout.
.github/​review-routing/​design-system-patterns.mts Extracts pattern names from Storybook headings.
.github/​review-routing/​design-system-patterns.md Uses generated pattern data and the default threshold.
.github/​review-routing/​design-system-maintenance.md Clarifies routing for dependency updates affecting behavior.

Review effort: Balanced

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These TS files should ideally be in tools/ so that they are type checked.

I saw many other PRs tried to add node scripts in .github. So, I think we should ban any JS/TS files outside tools/ and packages/

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These files are type-checked. They're included in the include pattern for the @wordpress/review-routing package.

That being said, the fact that the package is breaking out of itself to cover these additional files probably isn't great. The whole point of separating the configuration from the tool was that the tool could consume any folder of configuration, and the routings relevant for CI workflows could be colocated to CODEOWNERS and the rest of the workflow setup.

There may be a couple alternatives:

  • We could keep the routing files where they're at, and add a tsconfig.json here. That keeps them self-contained, but it goes against your own desire for banning files here, and it starts to litter the directory with non-configuration related files. Moving the folder could also cause some disruption since I've already been referencing it as a place people can contribute (Slack example).
  • Or, we could move the rules elsewhere, like the tools/review-routing package itself, or their own workspace. This goes against the intended separation, though practically speaking I don't know that it matters so much, particularly since the tool itself is already internal.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That being said, the fact that the package is breaking out of itself to cover these additional files probably isn't great. The whole point of separating the configuration from the tool was that the tool could consume any folder of configuration, and the routings relevant for CI workflows could be colocated to CODEOWNERS and the rest of the workflow setup.

This isn't entirely true. The tool knows to look for .github/review-routing, and with how the tool dynamically imports these files it makes sense that it should be covered by the tool's own TypeScript configuration.

Now that still leaves the question of whether we'd be better off just putting the rules in tools/review-routing itself. I still see some advantage to it staying where it is in .github to be colocated to CODEOWNERS and other CI workflow configurations, and because it's primarily a GitHub concern in how reviews are assigned. It's not a hill I'd die on though, if we'd rather just have it be in tools/.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another aspect here is that if we want to make the review routing a "GitHub App" that can be installed to other WordPress repositories (some related Slack discussion), we need the configuration to live independently from the tool. Developing it as an app might pose some challenges around what these "dynamic context" scripts need to look like from a perspective of what the tool is expecting to be able to handle, but at the very least it seems more future-proof to keep the configuration in a place like where it's at today.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to put a more definitive / conclusive point to the thread: I'd argue that this is fine as-is, and this is a "strong opinion loosely held".

@@ -1,6 +1,5 @@
---
reviewer: [aduth, ciampo]
threshold: 0.8

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we keep the 0.8 threshold while changing the question? If 0.7 is intentional, I'd be interested in the new scores for #84428 and #86490.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a good one to surface. In truth, I did have to lower the threshold here to be able to match expected pull requests. For example, #86490 stopped matching above 0.8 after the changes here, which was unacceptable to me as it was very much a good example of review routing working effectively, as I commented in #86490 (comment) .

My thinking with this change was: We only need 0.8 because the previous language was so broad that it would flag a lot of false positives, such that we needed to be particularly strict with the confidence of the decision model. With these changes now giving us more precision about what should be matched, we no longer need to worry as much about the false positives and can revert back to the default threshold.

Separately, I'll comb through my agent conversation for the raw data, as this was something I discussed quite intentionally with it.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's the raw data from my agent:

Raw data (AI-generated)

Here's the data. I scored a fixed set of PRs with Jev (jev-latest), two runs per PR for each version of the rule. All versions use the patterns list from the new .mts file, except the trunk version.

PR Should match Trunk wording PR wording At 0.8 At 0.7
#85775 Quick edit Yes 0.96 0.93 ✅ ✅
#86725 Save & Submit docs Yes 0.93 0.92–0.93 ✅ ✅
#86490 Media editor card Yes 0.85–0.86 0.74–0.76 ❌ ✅
#84428 Copy bound blocks No 0.89 0.59 ✅ ✅
#85117 Pattern creation error No 0.58–0.60 0.34–0.36 ✅ ✅
#84389 Copy bound text No 0.74–0.75 0.17 ✅ ✅
#82247 Status tab counts No 0.13 0.12–0.13 ✅ ✅
#86827 Widgets settings label No 0.04 0.09 ✅ ✅

The "At 0.8" and "At 0.7" columns show whether the PR wording gets the expected result at that threshold.

To answer your question directly: I couldn't find a wording that keeps #86490 above 0.8 without also bringing #84428 back up. These are the other versions I tried:

Wording #86490 #84428
"implement one of the patterns in patterns" + composition sentence 0.54–0.58 0.34–0.35
"implement one of the patterns in patterns" 0.82 0.74–0.77
"build an interface that one of the patterns in patterns applies to" 0.85 0.79–0.81
"implement a documented design system pattern, as listed in patterns" 0.78 0.77
PR wording: "applies to" + composition sentence 0.74–0.76 0.59

The composition sentence ("A pattern describes how to compose the interface components…") is what separates the two PRs. Without it, they score within about 0.05 of each other. It also lowers #86490 a bit, which is why the threshold needs to come down.

A caveat: the gap is real, but not wide. It runs from 0.59 (the highest negative) to 0.74 (the lowest positive). At 0.7, #86490 is only about 0.04–0.06 above the threshold, and scores vary by about 0.05 between identical runs. The set is also small (three positives), so the live results over the next few weeks will tell us more. If #86490-like PRs start to miss, 0.65 would put the threshold in the middle of the gap.

Comment thread tools/review-routing/README.md Outdated

The model then reads the question and the data as one object, `{ "question": "...", "stores": [ ... ] }`, so the question can name the field: "Does this change add an action to one of the stores in `stores`?" `--dry-run` prints this object.

The file runs with Node.js, from the base branch in CI, so it can use only Node.js built-in modules. To read a file of the repository, make its path from `import.meta.url`, and add its directory to the sparse checkout in `.github/workflows/review-routing.yml`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we mention that a local --dry-run executes the current checkout's .mts files? Reviewers should inspect those files before running it on a contributor's branch.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clarified this a bit in bcd4fb8 . Arguably it should be expected that running anything from a branch should come with a caution about what the branch might contain, though I suppose it's a bit more surprising here: (a) things in .github are often more "configuration" than runnable code, and (b) --dry-run is usually assumed to be "safer".

);
const api = rules.find( ( rule ) => rule.id === 'design-system-api' );

expect( patterns?.context?.patterns ).toContain( 'Copy to Clipboard' );

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we pass a fixture directory to loadRoutingRules with one rule that has a companion module and one that doesn't? That would test both cases without making a Storybook heading or the current setup of design-system-api part of the loader's contract.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we pass a fixture directory to loadRoutingRules with one rule that has a companion module and one that doesn't? That would test both cases without making a Storybook heading or the current setup of design-system-api part of the loader's contract.

Yeah I think that's better. I did that in 0ebe1d0 along with making the folder names follow our project conventions a bit better (do we have conventions?). My agent took an earlier request a bit too literally, as I'd discouraged it from writing temporary files in the test, but fixtures are a better happy medium between (a) files for tests and (b) not relying too heavily on something external that could change under the tests (like the actual live .github/review-routing contents).

@ciampo ciampo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚀


### Reference data

The model knows only what the question and the pull request tell it. To give it reference data, such as a list of names that changes as the project grows, add a TypeScript module with the same name and the `.mts` extension next to the routing file. Its default export takes no arguments and returns an object:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we say that the function returns a plain object with JSON-compatible values and that question is reserved? For example, a rule that collects names in a Set needs to convert it to an array before returning it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

[Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants