Bootstrap: measure how splittable the spine is before splitting it (#22642) - #23018
Merged
Merged
Conversation
…22642) The gen-0 rebuild compiles test/selfhost/compiler_lib_spine_smoke/main.php — 6.5k literal require_once lines — as ONE translation unit, single-threaded, for hours. The obvious remedy is the split-compilation shape already proven for helper units (#15889): per-chunk .o with a content fingerprint, emitted in parallel, merged at link. The risk is not the chunking, it is the edges. JIT\Call\ExternalMethod lowers a method call on a class that is not in the current module to __value__writeNull, silently (#579). Split the spine naively and every cross-chunk call becomes a silent null — a miscompile far worse than a slow build. So measure the edges first. Static analysis over the spine require list: which file declares which class, which classes each file references (new, static call, extends/implements, trait use, instanceof, catch, param/return/property types), and how many references cross a candidate chunk boundary. Measured on 6521 spine files, 8068 declared classes: strategy chunks largest cross-refs ratio top 5 4662 443 2.9% ext 77 2121 529 3.5% dir 93 2121 679 4.4% Intra-spine coupling is far lower than the file count suggests — 3-4% of references cross a directory boundary, and they concentrate into a few core chunks (heaviest pairs are ext/intl > lib/VM, ext/standard > lib/VM, lib/JIT > lib/VM). A further 40825 references leave the spine entirely (builtin/vendor) and are unaffected by chunking. So a split is worth attempting, and `ext` looks like the sweet spot: 77 chunks for parallelism at 3.5% coupling. The remaining obstacle is ext/standard at 2121 files — it would dominate wall time and needs sub-splitting before parallelism pays off. This ships the measurement only. Nothing in the build consumes it yet. Verification: - php -l clean - all three strategies run to completion on the current spine (1 unparsed file) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merged
3 tasks
PurHur
added a commit
that referenced
this pull request
Jul 25, 2026
… not work (#22642) (#23033) #23018 measured directory-level partitioning of the spine at 3-4% cross-chunk coupling and flagged one obstacle: ext/standard is ~2.1k files, a third of the spine, so it caps wall time no matter how many chunks the rest is cut into. This measures whether that chunk can be cut further. It cannot, cheaply. Two sub-partitions of the three oversized directories (ext/standard, lib/JIT, lib/VM): strategy chunks largest cross-refs ratio dir 93 2120 679 4.4% sub 161 504 6193 40.3% letter buckets hub 164 504 6229 40.6% letter buckets, Vm* kept together Cutting them by first letter takes coupling from 4.4% to 40.3% — 9x. The heaviest pairs all point into the V bucket (ext/standard#S > ext/standard#V at 409, lib/VM#R > lib/VM#V at 338), which reads as hub-and-leaf: many one-builtin leaf files calling a small shared core. So I tested keeping the Vm* classes together as a hub chunk. It does not help — 40.6%, marginally worse. The Vm* prefix does not capture whatever the real hub is. Hypothesis refuted, recorded here so nobody re-runs it. What this means for a split build: keep partitioning at directory level. ext/standard stays whole, so the critical path is one 2120-file chunk and the parallel speedup is bounded by 6521/2120, roughly 3x — not the ~14x core count suggests. The larger win is incremental rebuild (edit one ext module, re-emit one small chunk), not raw fan-out. Going below directory granularity needs a partition derived from the actual reference graph rather than from names. That is a bigger piece of work and this says it would have to earn its keep against a 3x ceiling. Verification: all four strategies run to completion on the current spine. Co-authored-by: PurHur <tedyyyyy@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Jul 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Measurement only — nothing in the build consumes it yet. It exists to decide whether split-compiling the spine is worth attempting, before anyone commits to a partitioning.
Why
The gen-0 rebuild compiles
test/selfhost/compiler_lib_spine_smoke/main.php— 6,521 literalrequire_oncelines — as one translation unit, single-threaded, for hours. The obvious remedy is the split-compilation shape already proven for helper units (#15889): per-chunk.owith a content fingerprint, emitted in parallel, merged at link.The risk is not the chunking, it is the edges.
JIT\Call\ExternalMethodlowers a method call on a class that is not in the current module to__value__writeNull, silently (#579). Split the spine naively and every cross-chunk call becomes a silent null — a miscompile far worse than a slow build.What it measures
Static pass over the spine require list: which file declares which class, which classes each file references (
new, static call,extends/implements, traituse,instanceof,catch, param/return/property types), and how many references cross a candidate chunk boundary.Result — the spine is much less entangled than its size suggests
6,521 files, 8,068 declared classes:
topextdirOnly 3–4% of intra-spine references cross a directory boundary, and they concentrate into a few core chunks — heaviest pairs are
ext/intl > lib/VM(82),ext/standard > lib/VM(73),lib/JIT > lib/VM(45). A further 40,825 references leave the spine entirely (builtin/vendor) and are unaffected by chunking.What this says about the next step
A split is worth attempting.
extlooks like the sweet spot — 77 chunks for parallelism at 3.5% coupling, and the coupling is concentrated enough that binding the core chunks' symbols by declaration (asHelperRuntimeCache::tryProvidealready does) would cover most edges.The remaining obstacle is
ext/standardat 2,121 files: it would dominate wall time regardless of how many other chunks run in parallel, so it needs sub-splitting before parallelism actually pays off.When a real chunk build is attempted, count the edges that failed to bind with
PHP_COMPILER_REPORT_EXTERNAL_STUBS=1(#22967) — that turns the silent-null risk into a number.Test plan
php -lclean--jsonoutput parses🤖 Generated with Claude Code