Repository navigation
Limit wp-build esbuild concurrency - #79889
Conversation
deaf89e to
fe78504
Compare
|
Size Change: 0 B Total Size: 7.81 MB |
20b09fa to
cc68047
Compare
|
Flaky tests detected in 5026013. 🔍 Workflow run URL: https://github.com/WordPress/gutenberg/actions/runs/28867792776
|
Nikschavan
left a comment
There was a problem hiding this comment.
This tested well for me, Thank you!
| ### Build Concurrency | ||
|
|
||
| By default, `wp-build` limits concurrent `esbuild` build jobs to half of the | ||
| available CPU parallelism, clamped between 2 and 8 jobs. This prevents large |
There was a problem hiding this comment.
MAX_DEFAULT_BUILD_CONCURRENCY on the top is defined 18, so I imagine the clamped concurrency is between 2 and 16?
There was a problem hiding this comment.
Aaah this should be updated.
|
It may be a few days before I have a chance to look at this deeply, but I agree with the general premise of what this is proposing, and that unlimited concurrency would likely produce worse outcomes for a project the scale of Gutenberg and others. On the specific approach, I know that As a broader point, I still think spawning a bunch of individual ESBuild processes is likely always going to be less efficient than using their Entry points feature, but last I looked this was rather complicated by the various bespoke customizations we've introduced in the build, and this has likely only gotten even more complex over time. |
I be happy to remove the configuration part! If I do remove this, I wonder if we should change the logic? ( or not ). It kind of looked like that using half of the available CPU's seemed optimal. But that is partly for a local machine, I bet this wouldn't be as accurate for CLI.
I didn't specifically look into this, mostly because we were already spinning up different ESBuild processes for each. I am happy to explore, but this may also be a good follow up? |
75c30e0 to
5026013
Compare
Assuming you mean CI, and if we're not going to tinker that as part of this PR, then we can always add the feature later when we decide it's a problem that we want to tackle. My hunch is that if we settle to a fixed value, and maybe even use a library like
Yeah, definitely not suggesting it as a blocker for this improvement. Though it's a can we've continued to kick down the road for a while now 😅 |
|
I haven't looked, but worth comparing the CI timings impact on this branch vs. a control like |
Oops yeah, I meant CI.
I didn't explore any libraries for this, given the actual changes where not that big. But happy to explore. |
5026013 to
0c50f37
Compare
|
I think this work is valuable and I'd like to see it continued. I think we need some more measurement and tuning. I also find it curious that there's no CI improvement. I see you temporarily overrode the concurrency in 1464560 to 4, but it's not clear if the timings made a difference there? I'm also not clear on the specifics around "warm" builds mentioned in the original comment, because there is no inherent caching to the build. It does seem like the benefit will largely be for development machines with many cores that can maximize the parallelism, but I'd be curious to try to see what the exact effect is with different values and over more runs. Should we be using the full number of cpus rather than half, for example. @louwie17 let me know if I can help try some of this tuning. |
manzoorwanijk
left a comment
There was a problem hiding this comment.
The speedup does not reproduce
My first (naive) pass ran 3x baseline then 3x PR, and reproduced the PR's result almost exactly: ~20.2s baseline vs ~16.9s PR, a 16% win. That was an artifact. Load average on this machine was 61 (Chrome), so sequential blocks drift.
Re-running interleaved, alternating baseline and PR within each round so drift cancels:
| Round | base (uncapped) | PR default (9) | PR --concurrency=4 |
|---|---|---|---|
| 1 | 21739 ms | 20019 ms | 20227 ms |
| 2 | 20291 ms | 20837 ms | 20279 ms |
| 3 | 19305 ms | 19186 ms | 20089 ms |
| 4 | 19739 ms | 19318 ms | 19562 ms |
| 5 | 19167 ms | 20096 ms | 19961 ms |
| median | 19739 ms | 20019 ms | 20089 ms |
Flat. The within-condition spread (~2.5s) is larger than any between-condition difference.
| return typeof os.availableParallelism === 'function' | ||
| ? os.availableParallelism() | ||
| : os.cpus().length; |
There was a problem hiding this comment.
This is available since Node 18, and thus we don't need this typeof check.
The typeof guard and os.cpus() fallback were only needed for Node < 18.14. wp-build requires Node >= 20.10, so os.availableParallelism() is always available. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QWvvMSSnAkdVhFL4JBRf67
Use the full available CPU parallelism as the default instead of half, still clamped between 2 and 16. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QWvvMSSnAkdVhFL4JBRf67
Trunk moved unit tests from Jest to Vitest: import the Vitest APIs explicitly, mock with `vi`, and assert the warning with `toHaveWarnedWith` so the console guard accepts it. Drop the dependency group comments that trunk no longer uses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011b5CpdkcW8GMR9s57P1wsr
Use half of `os.availableParallelism()` (at least one) as the build concurrency, and drop the `--concurrency` flag and `WP_BUILD_CONCURRENCY` environment variable overrides. Move the changelog entry under `## Unreleased`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UHkHDHxL2zAvGw9N8FMrHz
f0b9c33 to
22fc93e
Compare
sirreal
left a comment
There was a problem hiding this comment.
This looks good, thank you.
I believe the printed item timings weren't representative before (everything competing in parallel) and they're likely worse now (adds wait time as well) but that's a minor detail.
Thank you, this will be helpful.
|
I think the configuration may be helpful, especially with environment variables that can be configured on CI, but I agree on adding it later if there's need. |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UHkHDHxL2zAvGw9N8FMrHz
wp-build started every esbuild build at once. Running them all together used far more memory without making the build faster, and large builds often ran out of memory. Queue the builds and run at most half the available CPU cores' worth at a time, rounded up. --------- Co-authored-by: louwie17 <louwie17@git.wordpress.org> Co-authored-by: Nikschavan <nikschavan@git.wordpress.org> Co-authored-by: manzoorwanijk <manzoorwanijk@git.wordpress.org> Co-authored-by: sirreal <jonsurrell@git.wordpress.org> Co-authored-by: aduth <aduth@git.wordpress.org> Co-authored-by: retrofox <retrofox@git.wordpress.org> Co-authored-by: simison <simison@git.wordpress.org>
|
I just cherry-picked this PR to the wp/7.2 branch to get it included in the next release: 67cbadb |
This updates the pinned commit hash of the Gutenberg repository from `be6a50be6834e922b9db12c75c9a2cd817b2db59` to `d72020a651eda64dcc53b5dbd5b9d5bd511c4cc9` (version 24.3.0-rc.1). A full list of changes included in this commit can be found on GitHub: WordPress/gutenberg@be6a50b...d72020a - BorderControl: Remove double focus outline (WordPress/gutenberg#85173) - Snackbar: Use design system focus ring (WordPress/gutenberg#85168) - SelectControl: Fix default options showing the `Select` placeholder (WordPress/gutenberg#85172) - Actions API: Remove the unsupported `single` action context (WordPress/gutenberg#85186) - Client-side media: Upgrade wasm-vips to 0.0.19 and raise memory budgets accordingly (WordPress/gutenberg#84980) - Styles: Fix inconsistent 'Elements' tools panel heading level (WordPress/gutenberg#85247) - `media-utils`/MediaEdit: use design system focus ring (WordPress/gutenberg#84940) - Icons: Add new form based icons. (WordPress/gutenberg#84074) (WordPress/gutenberg#85360) - Navigation: keep custom CSS off inner list containers (WordPress/gutenberg#84881) (WordPress/gutenberg#85367) - Navigation: keep Global Styles custom CSS off inner list containers (WordPress/gutenberg#85366) - Edit as HTML: Show design system focus ring on the HTML textarea (WordPress/gutenberg#85266) (WordPress/gutenberg#85359) - Icons: Restore the missing speech bubble in postComments (WordPress/gutenberg#85275) - Block Directory: Standardize downloadable block focus indicators (WordPress/gutenberg#85307) - Global Styles: Add icons and tweak styling for element and block buttons (WordPress/gutenberg#85363) (WordPress/gutenberg#85414) - Tests: Stop asserting Core lacks the core fields functions (WordPress/gutenberg#85201) - Meta Boxes: Use the design system focus ring for the panel toggle (WordPress/gutenberg#85355) (WordPress/gutenberg#85410) - DataViews: Use the design system focus ring for list items. (WordPress/gutenberg#85358) (WordPress/gutenberg#85435) - Components: Standardize checkbox and radio focus outlines (WordPress/gutenberg#85379) (WordPress/gutenberg#85436) - UI: Use solid brand highlights for popup items (WordPress/gutenberg#84087) - Media: Preserve EXIF metadata when converting HEIC to JPEG client-side (WordPress/gutenberg#82040) - Icons: Add error, info and postComments to the core-admin collection (WordPress/gutenberg#85452) - UnitControl: Use the design system outline for the unit select focus ring (WordPress/gutenberg#85444) - Page List: Keep pages visible in the Navigation sidebar in contentOnly (WordPress/gutenberg#85357) - Icons: Add new asterisk icon (WordPress/gutenberg#85442) (WordPress/gutenberg#85498) - wp-env: Keep Debian security updates in Bullseye-based images (WordPress/gutenberg#85469) - Editor: Guard entity preload filters and share entity actions/fields logic (WordPress/gutenberg#85390) (WordPress/gutenberg#85499) - UI: Scale Spinner stroke width with its size (WordPress/gutenberg#85378) - [WP 7.2] Fix the block-library TypeScript test project (WordPress/gutenberg#85533) - Editor: Vertically center the Categories loading spinner (WordPress/gutenberg#85563) - DataViews: Render table bulk actions outside the table (WordPress/gutenberg#85250) (WordPress/gutenberg#85571) - Record e2e traces only when a failed test is retried (WordPress/gutenberg#85428) - Writing flow: fix triple click in a list item selecting the next item (WordPress/gutenberg#85569) - Build: Limit concurrent wp-build builds (WordPress/gutenberg#79889) - Icons: Promote Dashicon replacements to `core-admin` and map their Dashicons (WordPress/gutenberg#85541) Props peterwilsoncc, mamaduka. See #66340 git-svn-id: https://develop.svn.wordpress.org/trunk@64796 602fd350-edb4-49c9-b593-d223f7449a82
This updates the pinned commit hash of the Gutenberg repository from `be6a50be6834e922b9db12c75c9a2cd817b2db59` to `d72020a651eda64dcc53b5dbd5b9d5bd511c4cc9` (version 24.3.0-rc.1). A full list of changes included in this commit can be found on GitHub: WordPress/gutenberg@be6a50b...d72020a - BorderControl: Remove double focus outline (WordPress/gutenberg#85173) - Snackbar: Use design system focus ring (WordPress/gutenberg#85168) - SelectControl: Fix default options showing the `Select` placeholder (WordPress/gutenberg#85172) - Actions API: Remove the unsupported `single` action context (WordPress/gutenberg#85186) - Client-side media: Upgrade wasm-vips to 0.0.19 and raise memory budgets accordingly (WordPress/gutenberg#84980) - Styles: Fix inconsistent 'Elements' tools panel heading level (WordPress/gutenberg#85247) - `media-utils`/MediaEdit: use design system focus ring (WordPress/gutenberg#84940) - Icons: Add new form based icons. (WordPress/gutenberg#84074) (WordPress/gutenberg#85360) - Navigation: keep custom CSS off inner list containers (WordPress/gutenberg#84881) (WordPress/gutenberg#85367) - Navigation: keep Global Styles custom CSS off inner list containers (WordPress/gutenberg#85366) - Edit as HTML: Show design system focus ring on the HTML textarea (WordPress/gutenberg#85266) (WordPress/gutenberg#85359) - Icons: Restore the missing speech bubble in postComments (WordPress/gutenberg#85275) - Block Directory: Standardize downloadable block focus indicators (WordPress/gutenberg#85307) - Global Styles: Add icons and tweak styling for element and block buttons (WordPress/gutenberg#85363) (WordPress/gutenberg#85414) - Tests: Stop asserting Core lacks the core fields functions (WordPress/gutenberg#85201) - Meta Boxes: Use the design system focus ring for the panel toggle (WordPress/gutenberg#85355) (WordPress/gutenberg#85410) - DataViews: Use the design system focus ring for list items. (WordPress/gutenberg#85358) (WordPress/gutenberg#85435) - Components: Standardize checkbox and radio focus outlines (WordPress/gutenberg#85379) (WordPress/gutenberg#85436) - UI: Use solid brand highlights for popup items (WordPress/gutenberg#84087) - Media: Preserve EXIF metadata when converting HEIC to JPEG client-side (WordPress/gutenberg#82040) - Icons: Add error, info and postComments to the core-admin collection (WordPress/gutenberg#85452) - UnitControl: Use the design system outline for the unit select focus ring (WordPress/gutenberg#85444) - Page List: Keep pages visible in the Navigation sidebar in contentOnly (WordPress/gutenberg#85357) - Icons: Add new asterisk icon (WordPress/gutenberg#85442) (WordPress/gutenberg#85498) - wp-env: Keep Debian security updates in Bullseye-based images (WordPress/gutenberg#85469) - Editor: Guard entity preload filters and share entity actions/fields logic (WordPress/gutenberg#85390) (WordPress/gutenberg#85499) - UI: Scale Spinner stroke width with its size (WordPress/gutenberg#85378) - [WP 7.2] Fix the block-library TypeScript test project (WordPress/gutenberg#85533) - Editor: Vertically center the Categories loading spinner (WordPress/gutenberg#85563) - DataViews: Render table bulk actions outside the table (WordPress/gutenberg#85250) (WordPress/gutenberg#85571) - Record e2e traces only when a failed test is retried (WordPress/gutenberg#85428) - Writing flow: fix triple click in a list item selecting the next item (WordPress/gutenberg#85569) - Build: Limit concurrent wp-build builds (WordPress/gutenberg#79889) - Icons: Promote Dashicon replacements to `core-admin` and map their Dashicons (WordPress/gutenberg#85541) Props peterwilsoncc, mamaduka. See #66340 Built from https://develop.svn.wordpress.org/trunk@64796 git-svn-id: http://core.svn.wordpress.org/trunk@63946 1a063a9b-81f0-0310-95a4-ce76da25c4cd
|
I noticed we have a PR to add config, in case anyone in this thread find a need for it: |
This updates the pinned commit hash of the Gutenberg repository from `be6a50be6834e922b9db12c75c9a2cd817b2db59` to `d72020a651eda64dcc53b5dbd5b9d5bd511c4cc9` (version 24.3.0-rc.1). A full list of changes included in this commit can be found on GitHub: WordPress/gutenberg@be6a50b...d72020a - BorderControl: Remove double focus outline (WordPress/gutenberg#85173) - Snackbar: Use design system focus ring (WordPress/gutenberg#85168) - SelectControl: Fix default options showing the `Select` placeholder (WordPress/gutenberg#85172) - Actions API: Remove the unsupported `single` action context (WordPress/gutenberg#85186) - Client-side media: Upgrade wasm-vips to 0.0.19 and raise memory budgets accordingly (WordPress/gutenberg#84980) - Styles: Fix inconsistent 'Elements' tools panel heading level (WordPress/gutenberg#85247) - `media-utils`/MediaEdit: use design system focus ring (WordPress/gutenberg#84940) - Icons: Add new form based icons. (WordPress/gutenberg#84074) (WordPress/gutenberg#85360) - Navigation: keep custom CSS off inner list containers (WordPress/gutenberg#84881) (WordPress/gutenberg#85367) - Navigation: keep Global Styles custom CSS off inner list containers (WordPress/gutenberg#85366) - Edit as HTML: Show design system focus ring on the HTML textarea (WordPress/gutenberg#85266) (WordPress/gutenberg#85359) - Icons: Restore the missing speech bubble in postComments (WordPress/gutenberg#85275) - Block Directory: Standardize downloadable block focus indicators (WordPress/gutenberg#85307) - Global Styles: Add icons and tweak styling for element and block buttons (WordPress/gutenberg#85363) (WordPress/gutenberg#85414) - Tests: Stop asserting Core lacks the core fields functions (WordPress/gutenberg#85201) - Meta Boxes: Use the design system focus ring for the panel toggle (WordPress/gutenberg#85355) (WordPress/gutenberg#85410) - DataViews: Use the design system focus ring for list items. (WordPress/gutenberg#85358) (WordPress/gutenberg#85435) - Components: Standardize checkbox and radio focus outlines (WordPress/gutenberg#85379) (WordPress/gutenberg#85436) - UI: Use solid brand highlights for popup items (WordPress/gutenberg#84087) - Media: Preserve EXIF metadata when converting HEIC to JPEG client-side (WordPress/gutenberg#82040) - Icons: Add error, info and postComments to the core-admin collection (WordPress/gutenberg#85452) - UnitControl: Use the design system outline for the unit select focus ring (WordPress/gutenberg#85444) - Page List: Keep pages visible in the Navigation sidebar in contentOnly (WordPress/gutenberg#85357) - Icons: Add new asterisk icon (WordPress/gutenberg#85442) (WordPress/gutenberg#85498) - wp-env: Keep Debian security updates in Bullseye-based images (WordPress/gutenberg#85469) - Editor: Guard entity preload filters and share entity actions/fields logic (WordPress/gutenberg#85390) (WordPress/gutenberg#85499) - UI: Scale Spinner stroke width with its size (WordPress/gutenberg#85378) - [WP 7.2] Fix the block-library TypeScript test project (WordPress/gutenberg#85533) - Editor: Vertically center the Categories loading spinner (WordPress/gutenberg#85563) - DataViews: Render table bulk actions outside the table (WordPress/gutenberg#85250) (WordPress/gutenberg#85571) - Record e2e traces only when a failed test is retried (WordPress/gutenberg#85428) - Writing flow: fix triple click in a list item selecting the next item (WordPress/gutenberg#85569) - Build: Limit concurrent wp-build builds (WordPress/gutenberg#79889) - Icons: Promote Dashicon replacements to `core-admin` and map their Dashicons (WordPress/gutenberg#85541) Props peterwilsoncc, mamaduka. See #66340 git-svn-id: https://develop.svn.wordpress.org/trunk@64796 602fd350-edb4-49c9-b593-d223f7449a82


What?
Adds a global concurrency limit for
@wordpress/buildesbuild jobs.buildWithConcurrency()wrapper for allesbuild.build()calls inwp-build.os.availableParallelism(), rounded up.Fixes #79888.
Why?
wp-buildcurrently schedules package, route, widget, style, and worker esbuild jobs without a global cap. Large projects can start many bundles at once, which oversubscribes CPU and memory. In downstream projects with many widgets this can make builds slower and can OOM CI runners.Local Gutenberg build:all benchmark
I re-tested this PR locally on July 6, 2026 with
npm run build, using the same checkout and machine for both runs. Node reported 12 available CPUs, so this PR used a concurrency of 6, the same as the current half-CPU default.npm run buildtotalwp-buildportion/usr/bin/time -p153f003c3fdcc6804713d4153f003c3fdcc6804713d4The first run was a fresh build and showed the larger difference; the second was warm, which accounts for the smaller gap.
Jetpack CI results
Tested in Jetpack CI with a pnpm override of
@wordpress/build: Automattic/jetpack#53394 (this PR) against Automattic/jetpack#53393 (its merge base). This PR lowered peak memory during wp-build in every round, mostly because far fewer Sass compiler processes ran at once, and build time did not change measurably. See the comment below for the setup, per-run links and detailed numbers.Testing Instructions
npm installif dependencies are not already installed.npm run test:unit -- packages/wp-build/lib/test/build-concurrency.js.npm run buildand confirm the output includesUsing build concurrency: <n>, where<n>is half of your available CPUs, rounded up.Testing performed
npm run buildon trunk (153f003c3fd) and on this PR (cc6804713d4) for the benchmark abovenpm run test:unit -- packages/wp-build/lib/test/build-concurrency.jsnpm run lint:js -- packages/wp-build/lib(no errors)npm run typecheck -- packages/wp-build/tsconfig.jsonnode tools/validation/validate-changelogs.ts --require-pr=79889 packages/wp-build/CHANGELOG.md🤖 Generated with Claude Code
https://claude.ai/code/session_01UHkHDHxL2zAvGw9N8FMrHz