Repository navigation
Automated Testing: Refine design system review routing rules - #86609
Conversation
🤖 PR meta 🤖🎉 PropsIf you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message. To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. Props Bot commandsYou can trigger Props Bot actions by commenting on this PR:
Updated as activity occurs, without notifying anyone named here. Add the |
ciampo
left a comment
There was a problem hiding this comment.
I wasn't involved in the first iteration, but overall the changes LGTM 🚀
This is probably an area that will need a few iterations + observations before we land on a stable and reliable heuristic
There was a problem hiding this comment.
Should we consider a DS migration also migrating CSS from / to WPDS tokens, or using ThemeProvider from the theme package?
There was a problem hiding this comment.
There's a separate routing configuration specifically for tokens design-system-tokens.md, which should cover how new code uses design tokens.
That being said, there are some opportunities:
- Your observation could reflect how having these specified in separate documents can be confused for us as maintainers. There's nothing to say it has to be one or many documents, and the current approach came from a reflection after initially having one document for everything. But there might also be some middle-ground, like "migration" encompassing both components and tokens.
- The tokens document doesn't mention anything about ThemeProvider. I tested this locally with some revised phrasing†, but specifically calling out ThemeProvider can also water down the targetedness for token usage specifically. It might be good to try to find specific pull requests that involve the sort of work we're looking for and refine around that.
- The tokens language can produce some false positives as-is, like Notes: Add search and a status filter to the All Notes sidebar #84060 and Style revisions: Compare the selected revision with the current version side by side #85288. In these cases, it might be getting tripped upon tagging any styling change. I think the language makes it sound like "colors, spacing, or typography" are mutually exclusive with "design tokens".
I pushed a couple small improvements to address some of this:
- adb9922 expands migration to cover more packages of the design system, not just claiming
@wordpress/uiand@wordpress/componentsas the only packages. It intentionally excludes DataViews for now; I don't know that the components team is prepared to be looped into all of these right now, and may deserve its own configuration similar to how it's set up in CODEOWNERS today. - 597c9ad refines the language of tokens routing to clarify what design tokens actually are and where they come from.
† Refined wording in migration:
Does this change start to use a component from @wordpress/components or @wordpress/ui in interface code, in place of another component or of code written by hand? This includes moving from @wordpress/components to @wordpress/ui. It also includes starting to use the design tokens of @wordpress/theme in interface code, in place of hard-coded values or of @wordpress/base-styles variables, and adding a ThemeProvider so that interface code follows a theme.
More detailed analysis on theme tokens usage findings against real PRs (AI-generated)
migration:
Does this change start to use a component of the design system in interface code, in place of another component or of code written by hand? Such components include those of
@wordpress/ui,@wordpress/componentsand@wordpress/admin-ui, and the icons of@wordpress/icons. This includes moving from@wordpress/componentsto@wordpress/ui.
The three known positives still pass: 80500 0.89, #86501 0.95 and #85799 0.92.
tokens:
Does this change affect how interface code uses the design tokens of
@wordpress/theme, which are the--wpds-*CSS custom properties? Answer no when …
The "Answer no" sentence is the same as before. I tested three wordings on the same 19 PRs (threshold 0.8):
| PR | Kind | Old | "such as color, spacing or typography" | --wpds-* (chosen) |
"such as color, spacing, typography, motion and cursors" |
|---|---|---|---|---|---|
| 86402 Image Editor | token adoption | 0.93 | 0.90 | 0.91 | 0.89 |
| 83074 wp-admin colors | token adoption | 0.94 | 0.81 | 0.89 | 0.83 |
| 79002 edit-site color | token adoption | 0.93 | 0.80 | 0.86 | 0.83 |
| 80300 Notes mention chips | token adoption | 0.92 | 0.73 | 0.84 | 0.74 |
| 75204 DataViews | token adoption | 0.87 | 0.83 | 0.80 | 0.83 |
| 77376 Editor cursor | token adoption | 0.92 | 0.55 | 0.79 | 0.81 |
| 78204 Dashboard motion | token adoption | 0.90 | 0.56 | 0.79 | 0.78 |
| 77358 Commands cursor | token adoption | 0.83 | 0.37 | 0.56 | 0.54 |
| 81112 Post editor | ThemeProvider | 0.94 | 0.91 | 0.91 | 0.92 |
| 81173 Widgets editor | ThemeProvider | 0.93 | 0.87 | 0.88 | 0.87 |
| 80557 Notices | ThemeProvider | 0.87 | 0.76 | 0.81 | 0.85 |
| 81653 Inverted seed | ThemeProvider | 0.91 | 0.64 | 0.76 | 0.69 |
| 84060 Notes search | new UI, no tokens | 0.84 | 0.57 | 0.21 | 0.55 |
| 85288 Style revisions | new UI, no tokens | 0.82 | 0.46 | 0.25 | 0.46 |
| 86154 Media header | new UI, 1 token | 0.90 | 0.67 | 0.65 | 0.60 |
| 85819 Emoji reactions | new UI, 11 tokens | 0.89 | 0.72 | 0.82 | 0.72 |
Why I chose the --wpds-* wording:
- It separates the cases most clearly. The two new-UI PRs with no tokens fell from 0.82–0.84 to 0.21–0.25. The other wordings only took them down to about 0.5.
- It keeps adoption cases better. A list of token kinds acts as a closed list, even with "such as": the cursor and motion PRs dropped when they were not in it. The
--wpds-*name matches text that the model can see in the diff. This is the same lesson as the@wordpress/componentsto file path question.
Costs to know about:
- 77376 and 78204 are at 0.79, on the threshold. 77358 (a one-line change from
cursor: pointerto the token) and 81653 (a change to an existing ThemeProvider) are below it. These are small mechanical adoptions, so I think losing them is acceptable. If you disagree, a 0.75 threshold would catch three of them and still exclude the no-token PRs. But that would put the threshold just above the scores we observed, and I advised against that before. - 85819 still passes at 0.82. It writes a lot of new token CSS, so "affects how interface code uses the design tokens" is arguably true.
tokensfell on 80500 and UI: Use LoadingIndicator in remaining popup consumers #86501 (0.24–0.27). That is correct: they are component migrations, andmigrationcatches them.
Other notes:
- Token adoption PRs now get 0.12–0.51 from
migration(before: about 0.05). "A component of the design system" picks up a little of the token meaning. All are below the threshold, andtokenscatches them anyway. bespoke-uistill has the closed pair "@wordpress/componentsand@wordpress/ui". I did not change it.
There was a problem hiding this comment.
But there might also be some middle-ground, like "migration" encompassing both components and tokens.
That, or even cross-referencing across documents for easier discovery?
The tokens document doesn't mention anything about ThemeProvider
Yeah, I guess consuming ThemeProvider is a different task type than consuming WPDS tokens
Thank you!
There was a problem hiding this comment.
That, or even cross-referencing across documents for easier discovery?
It might not be exactly what you had in mind, but one of the improvements I was thinking about in #86325 was "generated context for rules like component or tokens lists", which in my mind would be related external resources that could be attached to the request context, like a full list of tokens or components, so that the decision model can know if e.g. a pull request could leverage an existing token or component.
Yep, absolutely. This pull request is part of that feedback loop stemming from observations after seeing the results that #86449 made available, and I expect we'll continue to do that based on our own anecdotal observations of pull requests we are or aren't being flagged for. |
What?
Refines review routing documents for how the @WordPress/gutenberg-components would be pinged for changes relevant to design systems.
This follow-up from the initial introduction of the review routing implementation in #86325, using real-world data available as of #86449 that's running on every pull request to inform improvement opportunities.
Why?
As described in #86325, the intent of these configurations will be to replace @WordPress/gutenberg-components as default reviewer for many CODEOWNERS paths to an approach that uses judgment to determine whether any change (regardless of where it occurs) warrants further review from the design systems team. As such, it's important that the actual results of this judgment reflect the reality of what we want to be looped in for.
How?
The revisions are meant to reflect a few specific limitations:
@wordpress/componentsto@wordpress/ui.@wordpress/componentsto@wordpress/ui.Testing Instructions
Repeat Testing Instructions setup procedure in #86325, and run the tool against pull requests mentioned above, verifying that at least one of the documents now matches the team for review.
Examples: (abbreviated to show only the matching rule)
Note: Locally my AI agent has been using a pool of over 100 pull requests whose contents were analyzed using Opus 5.5 for understanding of relevance to our team based on hand-written guidance. Separately, I'd like to explore some sort of eval testing that can let us effectively iterate on these documents while protecting against regressions. This is made challenging because decision models aren't completely deterministic in their output and can very by confidence between calls.
Use of AI Tools
Researched and iterated in coordination with Claude Code + Opus 5.5.