Skip to content

Automated Testing: Refine design system review routing rules - #86609

Merged
aduth merged 3 commits into
trunkfrom
update/design-system-review-routing-rules
Nov 10, 2026
Merged

aduth merged 3 commits into
trunkfrom
update/design-system-review-routing-rules

Conversation

@aduth

@aduth aduth commented Nov 10, 2026

Copy link
Copy Markdown
Member

What?

Refines review routing documents for how the @WordPress/gutenberg-components would be pinged for changes relevant to design systems.

This follow-up from the initial introduction of the review routing implementation in #86325, using real-world data available as of #86449 that's running on every pull request to inform improvement opportunities.

Why?

As described in #86325, the intent of these configurations will be to replace @WordPress/gutenberg-components as default reviewer for many CODEOWNERS paths to an approach that uses judgment to determine whether any change (regardless of where it occurs) warrants further review from the design systems team. As such, it's important that the actual results of this judgment reflect the reality of what we want to be looped in for.

How?

The revisions are meant to reflect a few specific limitations:

  • Nothing covered the internal behavior and styling of components in these packages. The "API" documentation was focused too much on public exports, and the decision to denote "styles" as not being part of a public API meant that many changes with user-facing stylistic changes were not being flagged.
    • Examples of pull request not flagged under previous language: Modal: Preserve scroll-area focus outlines across dialog layouts #86023
    • Improvement: Add a new document which describes maintenance of the design system itself, which extends to cover anything that affects the end-user. This includes styles and component behavior. It excludes basic refactors or testing / typings improvements. The separate API document was kept, but updated to reflect how the concern here is in terms of maintenance contracts.
  • The migration documentation didn't effectively cover adoption of new components, because it was too focused on direct migration from @wordpress/components to @wordpress/ui.
  • Design system tooling was mentioned too briefly in migration routing that it both missed examples, and also didn't encompass the full slate of design system tooling.
    • Example of pull request not flagged under previous language: Use WCIcon alias for component Icon imports #78366
    • Improvement: Carve out a separate document for tooling specifically, that incorporates both linting as well as additional build tooling support of the design system and how it's used.

Testing Instructions

Repeat Testing Instructions setup procedure in #86325, and run the tool against pull requests mentioned above, verifying that at least one of the documents now matches the team for review.

Examples: (abbreviated to show only the matching rule)

npm run --workspace @wordpress/review-routing-tools route -- --pr 86023
> | design-system-maintenance | `WordPress/gutenberg-components` | 0.94 | 0.70 | Request review |

npm run --workspace @wordpress/review-routing-tools route -- --pr 86501
> | design-system-migration | `WordPress/gutenberg-components` | 0.92 | 0.70 | Request review |

npm run --workspace @wordpress/review-routing-tools route -- --pr 85799
> | design-system-migration | `WordPress/gutenberg-components` | 0.85 | 0.70 | Request review |

npm run --workspace @wordpress/review-routing-tools route -- --pr 78366
> | design-system-tooling | `WordPress/gutenberg-components` | 0.89 | 0.70 | Request review |

Note: Locally my AI agent has been using a pool of over 100 pull requests whose contents were analyzed using Opus 5.5 for understanding of relevance to our team based on hand-written guidance. Separately, I'd like to explore some sort of eval testing that can let us effectively iterate on these documents while protecting against regressions. This is made challenging because decision models aren't completely deterministic in their output and can very by confidence between calls.

Use of AI Tools

Researched and iterated in coordination with Claude Code + Opus 5.5.

@aduth
aduth requested a review from a team November 10, 2026 15:56
@aduth
aduth requested a review from desrosj as a code owner November 10, 2026 15:56
@aduth aduth added the [Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests. label Nov 10, 2026
@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

🤖 PR meta 🤖

Preview in WordPress Playground Preview in Storybook

🎉 Props

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: aduth <aduth@git.wordpress.org>
Co-authored-by: ciampo <mciampini@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.


Props Bot commands

You can trigger Props Bot actions by commenting on this PR:

  • @propsbot add <username> to give props to a GitHub user. Requires write access.
  • @propsbot remove <username> to leave a user out of the props. Requires write access.
  • @propsbot refresh to refresh this list. Adding the props-bot label works too.

Updated as activity occurs, without notifying anyone named here. Add the props-bot label to refresh.

@ciampo ciampo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wasn't involved in the first iteration, but overall the changes LGTM 🚀

This is probably an area that will need a few iterations + observations before we land on a stable and reliable heuristic

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we consider a DS migration also migrating CSS from / to WPDS tokens, or using ThemeProvider from the theme package?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a separate routing configuration specifically for tokens design-system-tokens.md, which should cover how new code uses design tokens.

That being said, there are some opportunities:

  • Your observation could reflect how having these specified in separate documents can be confused for us as maintainers. There's nothing to say it has to be one or many documents, and the current approach came from a reflection after initially having one document for everything. But there might also be some middle-ground, like "migration" encompassing both components and tokens.
  • The tokens document doesn't mention anything about ThemeProvider. I tested this locally with some revised phrasing†, but specifically calling out ThemeProvider can also water down the targetedness for token usage specifically. It might be good to try to find specific pull requests that involve the sort of work we're looking for and refine around that.
  • The tokens language can produce some false positives as-is, like Notes: Add search and a status filter to the All Notes sidebar #84060 and Style revisions: Compare the selected revision with the current version side by side #85288. In these cases, it might be getting tripped upon tagging any styling change. I think the language makes it sound like "colors, spacing, or typography" are mutually exclusive with "design tokens".

I pushed a couple small improvements to address some of this:

  • adb9922 expands migration to cover more packages of the design system, not just claiming @wordpress/ui and @wordpress/components as the only packages. It intentionally excludes DataViews for now; I don't know that the components team is prepared to be looped into all of these right now, and may deserve its own configuration similar to how it's set up in CODEOWNERS today.
  • 597c9ad refines the language of tokens routing to clarify what design tokens actually are and where they come from.

† Refined wording in migration:

Does this change start to use a component from @wordpress/components or @wordpress/ui in interface code, in place of another component or of code written by hand? This includes moving from @wordpress/components to @wordpress/ui. It also includes starting to use the design tokens of @wordpress/theme in interface code, in place of hard-coded values or of @wordpress/base-styles variables, and adding a ThemeProvider so that interface code follows a theme.

More detailed analysis on theme tokens usage findings against real PRs (AI-generated)

migration:

Does this change start to use a component of the design system in interface code, in place of another component or of code written by hand? Such components include those of @wordpress/ui, @wordpress/components and @wordpress/admin-ui, and the icons of @wordpress/icons. This includes moving from @wordpress/components to @wordpress/ui.

The three known positives still pass: 80500 0.89, #86501 0.95 and #85799 0.92.

tokens:

Does this change affect how interface code uses the design tokens of @wordpress/theme, which are the --wpds-* CSS custom properties? Answer no when …

The "Answer no" sentence is the same as before. I tested three wordings on the same 19 PRs (threshold 0.8):

PR Kind Old "such as color, spacing or typography" --wpds-* (chosen) "such as color, spacing, typography, motion and cursors"
86402 Image Editor token adoption 0.93 0.90 0.91 0.89
83074 wp-admin colors token adoption 0.94 0.81 0.89 0.83
79002 edit-site color token adoption 0.93 0.80 0.86 0.83
80300 Notes mention chips token adoption 0.92 0.73 0.84 0.74
75204 DataViews token adoption 0.87 0.83 0.80 0.83
77376 Editor cursor token adoption 0.92 0.55 0.79 0.81
78204 Dashboard motion token adoption 0.90 0.56 0.79 0.78
77358 Commands cursor token adoption 0.83 0.37 0.56 0.54
81112 Post editor ThemeProvider 0.94 0.91 0.91 0.92
81173 Widgets editor ThemeProvider 0.93 0.87 0.88 0.87
80557 Notices ThemeProvider 0.87 0.76 0.81 0.85
81653 Inverted seed ThemeProvider 0.91 0.64 0.76 0.69
84060 Notes search new UI, no tokens 0.84 0.57 0.21 0.55
85288 Style revisions new UI, no tokens 0.82 0.46 0.25 0.46
86154 Media header new UI, 1 token 0.90 0.67 0.65 0.60
85819 Emoji reactions new UI, 11 tokens 0.89 0.72 0.82 0.72

Why I chose the --wpds-* wording:

  • It separates the cases most clearly. The two new-UI PRs with no tokens fell from 0.82–0.84 to 0.21–0.25. The other wordings only took them down to about 0.5.
  • It keeps adoption cases better. A list of token kinds acts as a closed list, even with "such as": the cursor and motion PRs dropped when they were not in it. The --wpds-* name matches text that the model can see in the diff. This is the same lesson as the @wordpress/components to file path question.

Costs to know about:

  • 77376 and 78204 are at 0.79, on the threshold. 77358 (a one-line change from cursor: pointer to the token) and 81653 (a change to an existing ThemeProvider) are below it. These are small mechanical adoptions, so I think losing them is acceptable. If you disagree, a 0.75 threshold would catch three of them and still exclude the no-token PRs. But that would put the threshold just above the scores we observed, and I advised against that before.
  • 85819 still passes at 0.82. It writes a lot of new token CSS, so "affects how interface code uses the design tokens" is arguably true.
  • tokens fell on 80500 and UI: Use LoadingIndicator in remaining popup consumers #86501 (0.24–0.27). That is correct: they are component migrations, and migration catches them.

Other notes:

  • Token adoption PRs now get 0.12–0.51 from migration (before: about 0.05). "A component of the design system" picks up a little of the token meaning. All are below the threshold, and tokens catches them anyway.
  • bespoke-ui still has the closed pair "@wordpress/components and @wordpress/ui". I did not change it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But there might also be some middle-ground, like "migration" encompassing both components and tokens.

That, or even cross-referencing across documents for easier discovery?

The tokens document doesn't mention anything about ThemeProvider

Yeah, I guess consuming ThemeProvider is a different task type than consuming WPDS tokens

Thank you!

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That, or even cross-referencing across documents for easier discovery?

It might not be exactly what you had in mind, but one of the improvements I was thinking about in #86325 was "generated context for rules like component or tokens lists", which in my mind would be related external resources that could be attached to the request context, like a full list of tokens or components, so that the decision model can know if e.g. a pull request could leverage an existing token or component.

@aduth

aduth commented Nov 10, 2026

Copy link
Copy Markdown
Member Author

This is probably an area that will need a few iterations + observations before we land on a stable and reliable heuristic

Yep, absolutely. This pull request is part of that feedback loop stemming from observations after seeing the results that #86449 made available, and I expect we'll continue to do that based on our own anecdotal observations of pull requests we are or aren't being flagged for.

@aduth
aduth merged commit 6aa2c65 into trunk Nov 10, 2026
57 checks passed
@aduth
aduth deleted the update/design-system-review-routing-rules branch November 10, 2026 18:45
@github-actions github-actions Bot added this to the Gutenberg 24.5 milestone Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

[Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants