Skip to content

Bootstrap: measure how splittable the spine is before splitting it (#22642) - #23018

Merged
PurHur merged 1 commit into
masterfrom
feat/spine-split-probe
Jul 25, 2026
Merged

PurHur merged 1 commit into
masterfrom
feat/spine-split-probe

Conversation

@PurHur

@PurHur PurHur commented Jul 25, 2026

Copy link
Copy Markdown
Owner

Measurement only — nothing in the build consumes it yet. It exists to decide whether split-compiling the spine is worth attempting, before anyone commits to a partitioning.

Why

The gen-0 rebuild compiles test/selfhost/compiler_lib_spine_smoke/main.php6,521 literal require_once lines — as one translation unit, single-threaded, for hours. The obvious remedy is the split-compilation shape already proven for helper units (#15889): per-chunk .o with a content fingerprint, emitted in parallel, merged at link.

The risk is not the chunking, it is the edges. JIT\Call\ExternalMethod lowers a method call on a class that is not in the current module to __value__writeNull, silently (#579). Split the spine naively and every cross-chunk call becomes a silent null — a miscompile far worse than a slow build.

What it measures

Static pass over the spine require list: which file declares which class, which classes each file references (new, static call, extends/implements, trait use, instanceof, catch, param/return/property types), and how many references cross a candidate chunk boundary.

Result — the spine is much less entangled than its size suggests

6,521 files, 8,068 declared classes:

strategy chunks largest chunk cross-chunk refs ratio
top 5 4,662 443 2.9%
ext 77 2,121 529 3.5%
dir 93 2,121 679 4.4%

Only 3–4% of intra-spine references cross a directory boundary, and they concentrate into a few core chunks — heaviest pairs are ext/intl > lib/VM (82), ext/standard > lib/VM (73), lib/JIT > lib/VM (45). A further 40,825 references leave the spine entirely (builtin/vendor) and are unaffected by chunking.

What this says about the next step

A split is worth attempting. ext looks like the sweet spot — 77 chunks for parallelism at 3.5% coupling, and the coupling is concentrated enough that binding the core chunks' symbols by declaration (as HelperRuntimeCache::tryProvide already does) would cover most edges.

The remaining obstacle is ext/standard at 2,121 files: it would dominate wall time regardless of how many other chunks run in parallel, so it needs sub-splitting before parallelism actually pays off.

When a real chunk build is attempted, count the edges that failed to bind with PHP_COMPILER_REPORT_EXTERNAL_STUBS=1 (#22967) — that turns the silent-null risk into a number.

Test plan

  • php -l clean
  • all three strategies run to completion on the current spine (1 unparsed file)
  • --json output parses

🤖 Generated with Claude Code

…22642)

The gen-0 rebuild compiles test/selfhost/compiler_lib_spine_smoke/main.php — 6.5k
literal require_once lines — as ONE translation unit, single-threaded, for hours.
The obvious remedy is the split-compilation shape already proven for helper units
(#15889): per-chunk .o with a content fingerprint, emitted in parallel, merged at link.

The risk is not the chunking, it is the edges. JIT\Call\ExternalMethod lowers a method
call on a class that is not in the current module to __value__writeNull, silently
(#579). Split the spine naively and every cross-chunk call becomes a silent null — a
miscompile far worse than a slow build. So measure the edges first.

Static analysis over the spine require list: which file declares which class, which
classes each file references (new, static call, extends/implements, trait use,
instanceof, catch, param/return/property types), and how many references cross a
candidate chunk boundary.

Measured on 6521 spine files, 8068 declared classes:

  strategy   chunks  largest   cross-refs  ratio
  top             5     4662          443   2.9%
  ext            77     2121          529   3.5%
  dir            93     2121          679   4.4%

Intra-spine coupling is far lower than the file count suggests — 3-4% of references
cross a directory boundary, and they concentrate into a few core chunks (heaviest
pairs are ext/intl > lib/VM, ext/standard > lib/VM, lib/JIT > lib/VM). A further
40825 references leave the spine entirely (builtin/vendor) and are unaffected by
chunking.

So a split is worth attempting, and `ext` looks like the sweet spot: 77 chunks for
parallelism at 3.5% coupling. The remaining obstacle is ext/standard at 2121 files —
it would dominate wall time and needs sub-splitting before parallelism pays off.

This ships the measurement only. Nothing in the build consumes it yet.

Verification:
- php -l clean
- all three strategies run to completion on the current spine (1 unparsed file)

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@PurHur
PurHur merged commit 70dfcc9 into master Jul 25, 2026
PurHur added a commit that referenced this pull request Jul 25, 2026
… not work (#22642) (#23033)

#23018 measured directory-level partitioning of the spine at 3-4% cross-chunk
coupling and flagged one obstacle: ext/standard is ~2.1k files, a third of the spine,
so it caps wall time no matter how many chunks the rest is cut into. This measures
whether that chunk can be cut further. It cannot, cheaply.

Two sub-partitions of the three oversized directories (ext/standard, lib/JIT, lib/VM):

  strategy  chunks  largest  cross-refs  ratio
  dir           93     2120         679    4.4%
  sub          161      504        6193   40.3%   letter buckets
  hub          164      504        6229   40.6%   letter buckets, Vm* kept together

Cutting them by first letter takes coupling from 4.4% to 40.3% — 9x. The heaviest
pairs all point into the V bucket (ext/standard#S > ext/standard#V at 409,
lib/VM#R > lib/VM#V at 338), which reads as hub-and-leaf: many one-builtin leaf files
calling a small shared core.

So I tested keeping the Vm* classes together as a hub chunk. It does not help —
40.6%, marginally worse. The Vm* prefix does not capture whatever the real hub is.
Hypothesis refuted, recorded here so nobody re-runs it.

What this means for a split build: keep partitioning at directory level. ext/standard
stays whole, so the critical path is one 2120-file chunk and the parallel speedup is
bounded by 6521/2120, roughly 3x — not the ~14x core count suggests. The larger win
is incremental rebuild (edit one ext module, re-emit one small chunk), not raw fan-out.

Going below directory granularity needs a partition derived from the actual reference
graph rather than from names. That is a bigger piece of work and this says it would
have to earn its keep against a 3x ceiling.

Verification: all four strategies run to completion on the current spine.

Co-authored-by: PurHur <tedyyyyy@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
@PurHur
PurHur deleted the feat/spine-split-probe branch July 25, 2026 23:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants