Skip to content

Automated Testing: Add GitHub Actions workflow for review routing (diagnostic only) - #86449

Merged
aduth merged 4 commits into
trunkfrom
add/review-routing-ci
Nov 9, 2026
Merged

aduth merged 4 commits into
trunkfrom
add/review-routing-ci

Conversation

@aduth

@aduth aduth commented Nov 6, 2026 •

Copy link
Copy Markdown
Member

What?

Continuation of #86325

Adds a GitHub Actions workflow that uses the @wordpress/review-routing-tools package added in #86325 to request and output the results of a pull request change using the configuration model configured in repository settings and based on review routing configurations.

Why?

As an incremental step toward the goals outlined in #86325, this starts to provide us with real data about how pull requests would be evaluated to ensure that we see expected results before starting to request review from affected groups, and before making any changes to CODEOWNERS.

How?

The Action outputs the results in the actions log. I considered adding this as a detail to the "PR Meta" comment, but the comment is quite bloated, and I think someone interested in these diagnostics values can easily get them from the logs.

The changes in the commit b4dbf6d using the pull_request event is temporary and exists only to demonstrate the results here in this pull request, since pull_request_target would evaluate based on trunk, where these changes don't yet exist. Edit: After demonstrating this as functional in #86449 (comment), the pull_request event was removed in ed9ef3b.

Initially I am running this without the decision model variables yet configured in GitHub to demonstrate the failure case of being unable to reach the API. Edit: After demonstrating failure scenario in #86449 (comment), these values are now configured.

This adds new repository variables and secrets to configure Jev as the decision model, with a token sponsored by Automattic and managed by myself as an administrator. The token has a limited budget, but given the low costs associated with this model, it should be sufficient for a while. One of the objectives with this work is to understand the overall usage and costs associated with this, and moving this to be managed by the WordPress organization itself.

Testing Instructions

Verify the outputs associated with the "Review routing" action check on this pull request.

Use of AI Tools

Planned and implemented in coordination with Claude Code + Opus 5.5.

@aduth aduth added the [Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests. label Nov 6, 2026
Comment thread .github/workflows/review-routing.yml Dismissed
@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

🤖 PR meta 🤖

Preview in WordPress Playground Preview in Storybook

🎉 Props

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: aduth <aduth@git.wordpress.org>
Co-authored-by: lancewillett <lancewillett@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.


Props Bot commands

You can trigger Props Bot actions by commenting on this PR:

  • @propsbot add <username> to give props to a GitHub user. Requires write access.
  • @propsbot remove <username> to leave a user out of the props. Requires write access.
  • @propsbot refresh to refresh this list. Adding the props-bot label works too.

Updated as activity occurs, without notifying anyone named here. Add the props-bot label to refresh.

@aduth

aduth commented Nov 6, 2026

Copy link
Copy Markdown
Member Author

Initially I am running this without the decision model variables yet configured in GitHub to demonstrate the failure case of being unable to reach the API.

Run 1 is expected result: https://github.com/WordPress/gutenberg/actions/runs/42243753359/job/127980466185?pr=86449

The process failed with the following message, but the check continues:

Set DECISION_API_URL, DECISION_API_KEY and DECISION_MODEL, or use --dry-run. See tools/review-routing/README.md.

Next I'll configure these values and re-run it.

@aduth

aduth commented Nov 6, 2026

Copy link
Copy Markdown
Member Author

After adding the configuration values (and updating variable names in 5f596f0), scores are now shown as expected.

Rule Team Probability Threshold Decision
design-system-api WordPress/gutenberg-components 0.05 0.70 No request
design-system-bespoke-ui WordPress/gutenberg-components 0.02 0.70 No request
design-system-docs WordPress/gutenberg-components 0.04 0.70 No request
design-system-migration WordPress/gutenberg-components 0.02 0.70 No request
design-system-patterns WordPress/gutenberg-components 0.03 0.80 No request
design-system-style-sync WordPress/gutenberg-components 0.07 0.80 No request
design-system-tokens WordPress/gutenberg-components 0.04 0.80 No request

Model jev-latest read 2 files, about 1,608 tokens. Diffs: 0 generated, 0 trimmed, 0 left out for budget.

See: https://github.com/WordPress/gutenberg/actions/runs/42245605761/job/127986898176?pr=86449

I pushed ed9ef3b to remove the temporary action event so that this now only targets pull_request_target, so this is ready for final review now.

@aduth
aduth requested review from lancewillett and mirka November 6, 2026 15:52

@lancewillett lancewillett left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like a good starting point. No comments or blockers.

@aduth
aduth merged commit 5c13e00 into trunk Nov 9, 2026
75 checks passed
@aduth
aduth deleted the add/review-routing-ci branch November 9, 2026 15:24
@github-actions github-actions Bot added this to the Gutenberg 24.5 milestone Nov 9, 2026
@aduth

aduth commented Nov 9, 2026

Copy link
Copy Markdown
Member Author

I've been monitoring this since it was merged a few hours ago, and already starting to get some good data. GitHub's brownouts today for pull_request_target actions caused some failures, but that should already be addressed moving forward as it is included in the repository policy now (see related Core-66139).

Some specific examples:

Pull requests which are now flagged for review by the components team but would previously have been missed by CODEOWNERS:

Pull requests which would have been flagged by CODEOWNERS, but which (correctly) do not flag for additional team review:

Pull requests which would be flagged correctly by both CODEOWNERS and the review routing system:

Potential gaps where neither CODEOWNERS nor the review routing catches:

These two gaps could potentially hint at an opportunity to improve the language in the "migration" routing.

@aduth

aduth commented Nov 9, 2026

Copy link
Copy Markdown
Member Author

Pull requests which would have been flagged by CODEOWNERS, but which (correctly) do not flag for additional team review:
[...]

This might be one we want to be flagged on, and it's something I'm already preparing improvements for separately. The issue is that design-system-api specifically touches on changes to the public API and explicitly excludes "internal styles". While there might be certain changes in these packages we feel okay to be hands-off for (e.g. refactors), I think styles are an area we do want to be flagged for review, even if it's not technically a public API. In this way it might be better to think of it in terms of maintenance contracts and changes with a user-facing impact.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

[Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants