Repository navigation
Automated Testing: Add GitHub Actions workflow for review routing (diagnostic only) - #86449
Conversation
🤖 PR meta 🤖🎉 PropsIf you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message. To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. Props Bot commandsYou can trigger Props Bot actions by commenting on this PR:
Updated as activity occurs, without notifying anyone named here. Add the |
Run 1 is expected result: https://github.com/WordPress/gutenberg/actions/runs/42243753359/job/127980466185?pr=86449 The process failed with the following message, but the check continues:
Next I'll configure these values and re-run it. |
|
After adding the configuration values (and updating variable names in 5f596f0), scores are now shown as expected.
See: https://github.com/WordPress/gutenberg/actions/runs/42245605761/job/127986898176?pr=86449 I pushed ed9ef3b to remove the temporary action event so that this now only targets |
lancewillett
left a comment
There was a problem hiding this comment.
Looks like a good starting point. No comments or blockers.
|
I've been monitoring this since it was merged a few hours ago, and already starting to get some good data. GitHub's brownouts today for Some specific examples: Pull requests which are now flagged for review by the components team but would previously have been missed by CODEOWNERS:
Pull requests which would have been flagged by CODEOWNERS, but which (correctly) do not flag for additional team review:
Pull requests which would be flagged correctly by both CODEOWNERS and the review routing system:
Potential gaps where neither CODEOWNERS nor the review routing catches:
These two gaps could potentially hint at an opportunity to improve the language in the "migration" routing. |
This might be one we want to be flagged on, and it's something I'm already preparing improvements for separately. The issue is that |
What?
Continuation of #86325
Adds a GitHub Actions workflow that uses the
@wordpress/review-routing-toolspackage added in #86325 to request and output the results of a pull request change using the configuration model configured in repository settings and based on review routing configurations.Why?
As an incremental step toward the goals outlined in #86325, this starts to provide us with real data about how pull requests would be evaluated to ensure that we see expected results before starting to request review from affected groups, and before making any changes to CODEOWNERS.
How?
The Action outputs the results in the actions log. I considered adding this as a detail to the "PR Meta" comment, but the comment is quite bloated, and I think someone interested in these diagnostics values can easily get them from the logs.
The changes in the commit b4dbf6d using theEdit: After demonstrating this as functional in #86449 (comment), thepull_requestevent is temporary and exists only to demonstrate the results here in this pull request, sincepull_request_targetwould evaluate based ontrunk, where these changes don't yet exist.pull_requestevent was removed in ed9ef3b.Initially I am running this without the decision model variables yet configured in GitHub to demonstrate the failure case of being unable to reach the API.Edit: After demonstrating failure scenario in #86449 (comment), these values are now configured.This adds new repository variables and secrets to configure Jev as the decision model, with a token sponsored by Automattic and managed by myself as an administrator. The token has a limited budget, but given the low costs associated with this model, it should be sufficient for a while. One of the objectives with this work is to understand the overall usage and costs associated with this, and moving this to be managed by the WordPress organization itself.
Testing Instructions
Verify the outputs associated with the "Review routing" action check on this pull request.
Use of AI Tools
Planned and implemented in coordination with Claude Code + Opus 5.5.