Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/review-routing/design-system-maintenance.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@
reviewer: [aduth, ciampo]
---

Does this change how a component, a design token or a Sass mixin of the design system packages `@wordpress/components`, `@wordpress/ui`, `@wordpress/theme`, `@wordpress/base-styles` or `@wordpress/icons` looks or behaves for the people who use it, for example its styles, its interactions or its accessibility? Only changes to these packages count, not changes to application code that uses them. Answer no when what people see and do stays the same, such as for a refactor, or for a change to dependencies, tooling, tests, types or build configuration that keeps the same runtime behavior.
Does this change how a component, a design token or a Sass mixin of the design system packages `@wordpress/components`, `@wordpress/ui`, `@wordpress/theme`, `@wordpress/base-styles` or `@wordpress/icons` looks or behaves for the people who use it, for example its styles, its interactions or its accessibility? Only changes to these packages count, not changes to application code that uses them. Answer no when what people see and do stays the same, such as for a refactor, or for a change to tooling, tests, types or build configuration. A dependency update counts when it changes how these components look or behave.
3 changes: 1 addition & 2 deletions .github/review-routing/design-system-patterns.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
---
reviewer: [aduth, ciampo]
threshold: 0.8

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we keep the 0.8 threshold while changing the question? If 0.7 is intentional, I'd be interested in the new scores for #84428 and #86490.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a good one to surface. In truth, I did have to lower the threshold here to be able to match expected pull requests. For example, #86490 stopped matching above 0.8 after the changes here, which was unacceptable to me as it was very much a good example of review routing working effectively, as I commented in #86490 (comment) .

My thinking with this change was: We only need 0.8 because the previous language was so broad that it would flag a lot of false positives, such that we needed to be particularly strict with the confidence of the decision model. With these changes now giving us more precision about what should be matched, we no longer need to worry as much about the false positives and can revert back to the default threshold.

Separately, I'll comb through my agent conversation for the raw data, as this was something I discussed quite intentionally with it.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's the raw data from my agent:

Raw data (AI-generated)

Here's the data. I scored a fixed set of PRs with Jev (jev-latest), two runs per PR for each version of the rule. All versions use the patterns list from the new .mts file, except the trunk version.

PR Should match Trunk wording PR wording At 0.8 At 0.7
#85775 Quick edit Yes 0.96 0.93 ✅ ✅
#86725 Save & Submit docs Yes 0.93 0.92–0.93 ✅ ✅
#86490 Media editor card Yes 0.85–0.86 0.74–0.76 ❌ ✅
#84428 Copy bound blocks No 0.89 0.59 ✅ ✅
#85117 Pattern creation error No 0.58–0.60 0.34–0.36 ✅ ✅
#84389 Copy bound text No 0.74–0.75 0.17 ✅ ✅
#82247 Status tab counts No 0.13 0.12–0.13 ✅ ✅
#86827 Widgets settings label No 0.04 0.09 ✅ ✅

The "At 0.8" and "At 0.7" columns show whether the PR wording gets the expected result at that threshold.

To answer your question directly: I couldn't find a wording that keeps #86490 above 0.8 without also bringing #84428 back up. These are the other versions I tried:

Wording #86490 #84428
"implement one of the patterns in patterns" + composition sentence 0.54–0.58 0.34–0.35
"implement one of the patterns in patterns" 0.82 0.74–0.77
"build an interface that one of the patterns in patterns applies to" 0.85 0.79–0.81
"implement a documented design system pattern, as listed in patterns" 0.78 0.77
PR wording: "applies to" + composition sentence 0.74–0.76 0.59

The composition sentence ("A pattern describes how to compose the interface components…") is what separates the two PRs. Without it, they score within about 0.05 of each other. It also lowers #86490 a bit, which is why the threshold needs to come down.

A caveat: the gap is real, but not wide. It runs from 0.59 (the highest negative) to 0.74 (the lowest positive). At 0.7, #86490 is only about 0.04–0.06 above the threshold, and scores vary by about 0.05 between identical runs. The set is also small (three positives), so the live results over the next few weeks will tell us more. If #86490-like PRs start to miss, 0.65 would put the threshold in the middle of the gap.

---

Does this change implement a documented design system pattern: destructive actions, error messages, save and submit, or copy to clipboard?
Does this change build an interface that one of the design system patterns in `patterns` applies to? A pattern describes how to compose the interface components of `@wordpress/ui` and `@wordpress/components` for a common task.
37 changes: 37 additions & 0 deletions .github/review-routing/design-system-patterns.mts

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These TS files should ideally be in tools/ so that they are type checked.

I saw many other PRs tried to add node scripts in .github. So, I think we should ban any JS/TS files outside tools/ and packages/

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These files are type-checked. They're included in the include pattern for the @wordpress/review-routing package.

That being said, the fact that the package is breaking out of itself to cover these additional files probably isn't great. The whole point of separating the configuration from the tool was that the tool could consume any folder of configuration, and the routings relevant for CI workflows could be colocated to CODEOWNERS and the rest of the workflow setup.

There may be a couple alternatives:

  • We could keep the routing files where they're at, and add a tsconfig.json here. That keeps them self-contained, but it goes against your own desire for banning files here, and it starts to litter the directory with non-configuration related files. Moving the folder could also cause some disruption since I've already been referencing it as a place people can contribute (Slack example).
  • Or, we could move the rules elsewhere, like the tools/review-routing package itself, or their own workspace. This goes against the intended separation, though practically speaking I don't know that it matters so much, particularly since the tool itself is already internal.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That being said, the fact that the package is breaking out of itself to cover these additional files probably isn't great. The whole point of separating the configuration from the tool was that the tool could consume any folder of configuration, and the routings relevant for CI workflows could be colocated to CODEOWNERS and the rest of the workflow setup.

This isn't entirely true. The tool knows to look for .github/review-routing, and with how the tool dynamically imports these files it makes sense that it should be covered by the tool's own TypeScript configuration.

Now that still leaves the question of whether we'd be better off just putting the rules in tools/review-routing itself. I still see some advantage to it staying where it is in .github to be colocated to CODEOWNERS and other CI workflow configurations, and because it's primarily a GitHub concern in how reviews are assigned. It's not a hill I'd die on though, if we'd rather just have it be in tools/.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another aspect here is that if we want to make the review routing a "GitHub App" that can be installed to other WordPress repositories (some related Slack discussion), we need the configuration to live independently from the tool. Developing it as an app might pose some challenges around what these "dynamic context" scripts need to look like from a perspective of what the tool is expecting to be able to handle, but at the very least it seems more future-proof to keep the configuration in a place like where it's at today.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to put a more definitive / conclusive point to the thread: I'd argue that this is fine as-is, and this is a "strong opinion loosely held".

Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
import { readdir, readFile } from 'node:fs/promises';

const PATTERNS_DIRECTORY = new URL(
'../../storybook/stories/design-system/patterns/',
import.meta.url
);

/**
* Lists the names of the documented design system patterns, from the heading
* of each pattern document in Storybook.
*
* @return The pattern names.
*/
export default async function () {
const names = ( await readdir( PATTERNS_DIRECTORY ) )
// The introduction describes patterns in general, not one pattern.
.filter(
( name ) => name.endsWith( '.mdx' ) && name !== 'introduction.mdx'
)
.sort();

const patterns = await Promise.all(
names.map( async ( name ) => {
const source = await readFile(
new URL( name, PATTERNS_DIRECTORY ),
'utf8'
);
const heading = source.match( /^# (.+)$/m );
if ( ! heading ) {
throw new Error( `${ name } has no heading.` );
}
return heading[ 1 ].trim();
} )
);

return { patterns };
}
2 changes: 2 additions & 0 deletions .github/workflows/review-routing.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,11 +35,13 @@ jobs:
steps:
# On `pull_request_target`, this checks out the base branch. The
# tool has no runtime dependencies, so `npm run` needs no install.
# Add the paths that the `.mts` files of the rules read.
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
sparse-checkout: |
tools/review-routing
.github/review-routing
storybook/stories/design-system/patterns
show-progress: ${{ runner.debug == '1' && 'true' || 'false' }}
persist-credentials: false

Expand Down
14 changes: 14 additions & 0 deletions tools/review-routing/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,20 @@ Does this change add or change a public API of the design system packages? ...

Ask about one kind of change in each file. A question that combines several conditions gives a less useful probability, and the model can let one condition hide another. To route several kinds of change to the same reviewer, add a file for each kind with the same `reviewer`. The tool decides each rule on its own and does not combine the rules of a reviewer, so the report shows the probability of each rule. To ask a new reviewer, or to opt yourself in to a kind of change, add a file.

### Reference data

The model knows only what the question and the pull request tell it. To give it reference data, such as a list of names that changes as the project grows, add a TypeScript module with the same name and the `.mts` extension next to the routing file. Its default export takes no arguments and returns an object:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we say that the function returns a plain object with JSON-compatible values and that question is reserved? For example, a rule that collects names in a Set needs to convert it to an array before returning it.


```ts
export default async function () {
return { stores: [ 'core', 'core/editor' ] };
}
```

The model then reads the question and the data as one object, `{ "question": "...", "stores": [ ... ] }`, so the question can name the field: "Does this change add an action to one of the stores in `stores`?" `--dry-run` prints this object.

The tool runs this file each time that it loads the rules, also with `--dry-run`, so treat it like any other code that you run from a checkout. It runs with Node.js, from the base branch in CI, and can use only Node.js built-in modules. To read a file of the repository, make its path from `import.meta.url`, and add its directory to the sparse checkout in `.github/workflows/review-routing.yml`.

## What the model reads

Every rule reads the same evidence:
Expand Down
9 changes: 7 additions & 2 deletions tools/review-routing/src/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -82,8 +82,13 @@ const { state, stats } = buildEvidence( pull, generated );
const request = buildRequest( state, rules, DECISION_MODEL ?? '' );

if ( values[ 'dry-run' ] ) {
const questions = rules.map(
( rule ) => `### ${ rule.id }\n\n${ rule.instructions }`
const questions = Object.entries( request.questions ).map(
( [ id, { instructions } ] ) =>
`### ${ id }\n\n${
typeof instructions === 'string'
? instructions
: `\`\`\`json\n${ JSON.stringify( instructions, null, '\t' ) }\n\`\`\``
}`
);
process.stdout.write(
`${ state }\n\n---\n\n# Questions\n\n${ questions.join( '\n\n' ) }\n`
Expand Down
16 changes: 13 additions & 3 deletions tools/review-routing/src/decision-api.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,10 @@ export type DecisionApi = {
export type DecisionRequest = {
model: string;
state: string;
questions: Record< string, { type: 'noul'; instructions: string } >;
questions: Record<
string,
{ type: 'noul'; instructions: string | Record< string, unknown > }
>;
};

type DecisionResponse = {
Expand All @@ -32,7 +35,9 @@ type CloudflareEnvelope = {
const TIMEOUT_MS = 120_000;

/**
* Builds a request with one yes or no question for each routing rule.
* Builds a request with one yes or no question for each routing rule. A rule
* with reference data sends an object, with the question in its own field, as
* the System One documentation recommends.
*
* @param state Evidence for the model to read.
* @param rules Routing rules.
Expand All @@ -50,7 +55,12 @@ export function buildRequest(
questions: Object.fromEntries(
rules.map( ( rule ) => [
rule.id,
{ type: 'noul', instructions: rule.instructions },
{
type: 'noul',
instructions: rule.context
? { question: rule.instructions, ...rule.context }
: rule.instructions,
},
] )
),
};
Expand Down
66 changes: 58 additions & 8 deletions tools/review-routing/src/routing-rules.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ export type RoutingRule = {
threshold: number;
/** File body, sent to the model as the question. */
instructions: string;
/** Reference data from the rule's optional `.mts` file. */
context?: Record< string, unknown >;
};

export const DEFAULT_THRESHOLD = 0.7;
Expand Down Expand Up @@ -70,6 +72,45 @@ export function parseRoutingRule( id: string, source: string ): RoutingRule {
return { id, reviewers, threshold, instructions };
}

/** The imported `.mts` file of a rule. */
type ContextModule = { default?: unknown };

/**
* Reads the reference data from the module of a rule's `.mts` file. Its
* default export takes no arguments and returns the data, which the model
* reads in named fields next to the question.
*
* @param id Rule id.
* @param module The imported module.
* @return The reference data.
*/
export async function readContext(
id: string,
module: ContextModule
): Promise< Record< string, unknown > > {
if ( typeof module.default !== 'function' ) {
throw new Error( `${ id }.mts: the default export is not a function.` );
}

const context: unknown = await module.default();
if (
typeof context !== 'object' ||
context === null ||
Object.getPrototypeOf( context ) !== Object.prototype
) {
throw new Error( `${ id }.mts: the function must return an object.` );
}

// The question goes in this field, so the context cannot replace it.
if ( 'question' in context ) {
throw new Error(
`${ id }.mts: the object cannot have a \`question\`.`
);
}

return context as Record< string, unknown >;
}

/**
* Reads every routing file in a directory.
*
Expand All @@ -79,16 +120,25 @@ export function parseRoutingRule( id: string, source: string ): RoutingRule {
export async function loadRoutingRules(
directory: URL = RULES_DIRECTORY
): Promise< RoutingRule[] > {
const names = ( await readdir( directory ) )
.filter( ( name ) => name.endsWith( '.md' ) )
.sort();
const files = await readdir( directory );
const names = files.filter( ( name ) => name.endsWith( '.md' ) ).sort();

return Promise.all(
names.map( async ( name ) =>
parseRoutingRule(
path.basename( name, '.md' ),
names.map( async ( name ) => {
const id = path.basename( name, '.md' );
const rule = parseRoutingRule(
id,
await readFile( new URL( name, directory ), 'utf8' )
)
)
);

if ( files.includes( `${ id }.mts` ) ) {
rule.context = await readContext(
id,
await import( new URL( `${ id }.mts`, directory ).href )
);
}

return rule;
} )
);
}
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,19 @@ describe( 'buildRequest', () => {
},
} );
} );

it( 'puts the question and the context of a rule in an object', () => {
const rule = { ...RULE, context: { components: [ 'Button' ] } };

expect(
buildRequest( 'state', [ rule ], 'jev' ).questions[
'design-system'
].instructions
).toEqual( {
question: 'Ask when UI changes.',
components: [ 'Button' ],
} );
} );
} );

describe( 'readProbability', () => {
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
reviewer: octocat
---

Ask when a store changes.
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
export default async function () {
return { stores: [ 'core' ] };
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
reviewer: octocat
---

Ask when a store changes.
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ import {
DEFAULT_THRESHOLD,
loadRoutingRules,
parseRoutingRule,
readContext,
} from '../routing-rules.ts';

describe( 'parseRoutingRule', () => {
Expand Down Expand Up @@ -72,6 +73,29 @@ describe( 'parseRoutingRule', () => {
} );
} );

describe( 'readContext', () => {
it( 'reads the object that the default export returns', async () => {
expect(
await readContext( 'data', {
default: async () => ( { stores: [ 'core' ] } ),
} )
).toEqual( { stores: [ 'core' ] } );
} );

it.each( [
[ 'a default export that is not a function', { default: {} } ],
[ 'a function that returns an array', { default: () => [] } ],
[
'an object with a question',
{ default: () => ( { question: 'Ask.' } ) },
],
] )( 'rejects a module with %s', async ( _, module ) => {
await expect( readContext( 'data', module ) ).rejects.toThrow(
/^data\.mts: /
);
} );
} );

describe( 'loadRoutingRules', () => {
it( 'reads the routing files in the repository', async () => {
const rules = await loadRoutingRules();
Expand All @@ -80,4 +104,13 @@ describe( 'loadRoutingRules', () => {
'design-system-api'
);
} );

it( 'reads the context of a rule from its .mts file', async () => {
const [ withContext, withoutContext ] = await loadRoutingRules(
new URL( './fixtures/routing-rules/', import.meta.url )
);

expect( withContext.context ).toEqual( { stores: [ 'core' ] } );
expect( withoutContext.context ).toBeUndefined();
} );
} );
6 changes: 5 additions & 1 deletion tools/review-routing/tsconfig.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
{
"$schema": "https://json.schemastore.org/tsconfig.json",
"extends": "@wordpress/monorepo-tools/tsconfig/node.base.json",
"include": [ "src" ]
"compilerOptions": {
// The `.mts` files of the routing rules are outside this directory.
"rootDir": "../.."
},
"include": [ "src", "../../.github/review-routing/*.mts" ]
}
Loading