Repository navigation
Automated Testing: Add support for dynamic context to review routing #86921
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: trunk
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,6 +1,5 @@ | ||
| --- | ||
| reviewer: [aduth, ciampo] | ||
| threshold: 0.8 | ||
| --- | ||
|
|
||
| Does this change implement a documented design system pattern: destructive actions, error messages, save and submit, or copy to clipboard? | ||
| Does this change build an interface that one of the design system patterns in `patterns` applies to? A pattern describes how to compose the interface components of `@wordpress/ui` and `@wordpress/components` for a common task. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. These TS files should ideally be in I saw many other PRs tried to add node scripts in
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. These files are type-checked. They're included in the That being said, the fact that the package is breaking out of itself to cover these additional files probably isn't great. The whole point of separating the configuration from the tool was that the tool could consume any folder of configuration, and the routings relevant for CI workflows could be colocated to There may be a couple alternatives:
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
This isn't entirely true. The tool knows to look for Now that still leaves the question of whether we'd be better off just putting the rules in
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Another aspect here is that if we want to make the review routing a "GitHub App" that can be installed to other WordPress repositories (some related Slack discussion), we need the configuration to live independently from the tool. Developing it as an app might pose some challenges around what these "dynamic context" scripts need to look like from a perspective of what the tool is expecting to be able to handle, but at the very least it seems more future-proof to keep the configuration in a place like where it's at today.
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Just to put a more definitive / conclusive point to the thread: I'd argue that this is fine as-is, and this is a "strong opinion loosely held". |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| import { readdir, readFile } from 'node:fs/promises'; | ||
|
|
||
| const PATTERNS_DIRECTORY = new URL( | ||
| '../../storybook/stories/design-system/patterns/', | ||
| import.meta.url | ||
| ); | ||
|
|
||
| /** | ||
| * Lists the names of the documented design system patterns, from the heading | ||
| * of each pattern document in Storybook. | ||
| * | ||
| * @return The pattern names. | ||
| */ | ||
| export default async function () { | ||
| const names = ( await readdir( PATTERNS_DIRECTORY ) ) | ||
| // The introduction describes patterns in general, not one pattern. | ||
| .filter( | ||
| ( name ) => name.endsWith( '.mdx' ) && name !== 'introduction.mdx' | ||
| ) | ||
| .sort(); | ||
|
|
||
| const patterns = await Promise.all( | ||
| names.map( async ( name ) => { | ||
| const source = await readFile( | ||
| new URL( name, PATTERNS_DIRECTORY ), | ||
| 'utf8' | ||
| ); | ||
| const heading = source.match( /^# (.+)$/m ); | ||
| if ( ! heading ) { | ||
| throw new Error( `${ name } has no heading.` ); | ||
| } | ||
| return heading[ 1 ].trim(); | ||
| } ) | ||
| ); | ||
|
|
||
| return { patterns }; | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -21,6 +21,20 @@ Does this change add or change a public API of the design system packages? ... | |
|
|
||
| Ask about one kind of change in each file. A question that combines several conditions gives a less useful probability, and the model can let one condition hide another. To route several kinds of change to the same reviewer, add a file for each kind with the same `reviewer`. The tool decides each rule on its own and does not combine the rules of a reviewer, so the report shows the probability of each rule. To ask a new reviewer, or to opt yourself in to a kind of change, add a file. | ||
|
|
||
| ### Reference data | ||
|
|
||
| The model knows only what the question and the pull request tell it. To give it reference data, such as a list of names that changes as the project grows, add a TypeScript module with the same name and the `.mts` extension next to the routing file. Its default export takes no arguments and returns an object: | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Could we say that the function returns a plain object with JSON-compatible values and that |
||
|
|
||
| ```ts | ||
| export default async function () { | ||
| return { stores: [ 'core', 'core/editor' ] }; | ||
| } | ||
| ``` | ||
|
|
||
| The model then reads the question and the data as one object, `{ "question": "...", "stores": [ ... ] }`, so the question can name the field: "Does this change add an action to one of the stores in `stores`?" `--dry-run` prints this object. | ||
|
|
||
| The tool runs this file each time that it loads the rules, also with `--dry-run`, so treat it like any other code that you run from a checkout. It runs with Node.js, from the base branch in CI, and can use only Node.js built-in modules. To read a file of the repository, make its path from `import.meta.url`, and add its directory to the sparse checkout in `.github/workflows/review-routing.yml`. | ||
|
|
||
| ## What the model reads | ||
|
|
||
| Every rule reads the same evidence: | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| --- | ||
| reviewer: octocat | ||
| --- | ||
|
|
||
| Ask when a store changes. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| export default async function () { | ||
| return { stores: [ 'core' ] }; | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| --- | ||
| reviewer: octocat | ||
| --- | ||
|
|
||
| Ask when a store changes. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,9 @@ | ||
| { | ||
| "$schema": "https://json.schemastore.org/tsconfig.json", | ||
| "extends": "@wordpress/monorepo-tools/tsconfig/node.base.json", | ||
| "include": [ "src" ] | ||
| "compilerOptions": { | ||
| // The `.mts` files of the routing rules are outside this directory. | ||
| "rootDir": "../.." | ||
| }, | ||
| "include": [ "src", "../../.github/review-routing/*.mts" ] | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could we keep the 0.8 threshold while changing the question? If 0.7 is intentional, I'd be interested in the new scores for #84428 and #86490.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a good one to surface. In truth, I did have to lower the threshold here to be able to match expected pull requests. For example, #86490 stopped matching above 0.8 after the changes here, which was unacceptable to me as it was very much a good example of review routing working effectively, as I commented in #86490 (comment) .
My thinking with this change was: We only need 0.8 because the previous language was so broad that it would flag a lot of false positives, such that we needed to be particularly strict with the confidence of the decision model. With these changes now giving us more precision about what should be matched, we no longer need to worry as much about the false positives and can revert back to the default threshold.
Separately, I'll comb through my agent conversation for the raw data, as this was something I discussed quite intentionally with it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here's the raw data from my agent:
Raw data (AI-generated)