Release blocker: committed helper-runtime cache is stale, so a clean checkout cannot compile hello world in 2 minutes

[PurHur]

Summary

prelinked/helper-runtime/x86_64-linux/manifest.json carries a core_fingerprint that no longer
matches the live one. On mismatch every one of the 257 committed units is skipped, so a clean
checkout emits the whole helper corpus live on the first compile.

Measured on master today:

committed core_fingerprint: 34d6fd38281359f382b3
live coreFingerprint():     63e161964d00f30ff8c9

How it surfaced

The new compiler gate (#24275) failed on a GitHub runner, and the failure is informative:

=== nikic spine preflight (~18s) ===
bootstrap-spine-nikic-preflight: OK 6618/6618 spine PHP files parse
=== AOT smoke ===
FAIL  echo      compile failed (rc=124)
FAIL  arith     compile failed (rc=124)
...  (all 8 cases, rc=124 = timeout, 120s each)

rc=124 is my 120-second per-case timeout, not a compiler error. A GitHub runner starting from a
clean checkout cannot compile <?php echo "hi\n"; inside two minutes.

This is a real defect, not just a tight timeout

On this dev box the same script passes 8/8 in ~90 s total — because build/helper-runtime-cache is
warm here. That made my local timings unrepresentative of a clean checkout, which is the case a
user actually encounters.

What a fresh clone gets today: the committed cache exists, is skipped wholesale on the fingerprint
mismatch, and the first compile of any script pays for emitting the full corpus. This repo has
measured that cold path at >30 minutes for a trivial script before.

For a project heading to release, "clone and compile hello world" is close to the minimum bar, and
it is currently not met from a clean checkout.

Why it goes stale

The fingerprint covers the lowering machinery (lib/JIT.php, lib/Runtime.php, patches/, …) and
master merges roughly 4 commits/hour with ~56% touching those inputs. So the committed cache is
invalidated within hours of any refresh unless refreshing is part of the routine.

That is the same treadmill RELEASE-PLAN.md Phase 3 describes for gen-0 — an artifact keyed on a
fingerprint that changes several times a day. The difference is that refreshing this one is cheap:
the emit is parallel (PHP_COMPILER_EMIT_JOBS, ~4–5 min for the corpus) rather than the multi-hour
gen-0 rebuild, so it is tractable to keep current.

Suggested fixes

1. Refresh the committed cache (make helper-runtime-prelink-refresh) — I am running this now
   and will PR the result with before/after timings.
2. Gate on it. script/check-helper-runtime-prelink.php already exists but is report-only, and
   per its own comment an empty glob counts as clean. A fingerprint mismatch should be visible in
   release-readiness output rather than silently costing every user a full corpus emit.
3. Consider making cold-build time a release criterion — "time to compile hello world from a
   clean checkout" is a number worth having in release-readiness --json, because it is exactly
   what a new user experiences and nothing currently measures it.

Environment: php-compiler:22.04-dev, PHP 8.2.32, LLVM 9, master 72c379a77.

[PurHur]

claim: worker-lane-c — starting this run

[PurHur]

Measured the cold path — and refreshing the cache does not fix it

Numbers from this box with build/helper-runtime-cache removed to simulate a clean checkout:

first compile of <?php echo "hi\n";   533 s   (~9 min)
second compile                          5 s
full 8-case AOT smoke, cold            9m03s

So the first compile carries essentially the entire cost and everything after it is seconds.

The surprising part

I ran make helper-runtime-prelink-refresh, which brought the committed core_fingerprint in line
with the live one:

before: committed 34d6fd38281359f382b3   live 63e161964d00f30ff8c9   (mismatch — all 257 units skipped)
after : committed 63e161964d00f30ff8c9   live 63e161964d00f30ff8c9   (match)

First compile was still 533 s afterwards. So the fingerprint mismatch is real, but fixing it did
not make a cold compile fast — which means my framing in the issue body was incomplete. The stale
fingerprint is not sufficient to explain the ~9 minutes, and I would rather say that than
retro-fit an explanation.

Worth checking next: the refresh reported 176 emitted, 53 fresh (skipped), 59 known-broken (skipped), 29 failed now, so the committed set is incomplete by construction — but whether the
units hello-world actually needs are among the missing ones, I have not verified.

What I did not ship

The refresh produced 597 changed files / 246 MB. Committing a binary artifact that large, keyed
on a fingerprint master invalidates within hours, is the treadmill RELEASE-PLAN.md Phase 3 warns
about — and it would not have bought the ~9 minutes back anyway. Reverted.

CI is unblocked separately in #24308 by caching build/helper-runtime-cache between runs and
raising the smoke timeout. That fixes CI, not the user-facing cold build.

Why this still matters for release

"Clone the repo, compile hello world" costs ~9 minutes on a fast machine with a warm Docker image.
That is the first thing a new user does. I would suggest making it a tracked release criterion —
release-readiness --json currently reports documentation and stamp sync but nothing about whether
the compiler is usable from a clean tree.

[PurHur]

Merged via #24321.

Verification (pinned Docker env):

• php script/check-helper-runtime-prelink.php --strict → 293 fresh, 0 stale, 0 broken
• ./script/release-readiness.sh --json → user_release_ready: yes, helper-runtime-prelink gate ok
• vendor/bin/phpunit --filter CiScriptsTest::testReleaseReadinessScriptExistsAndJsonSchema → OK

Refresh: php script/emit-helper-runtime-object.php --force --prelink in Docker — 293 emitted, 24 known solo-unit failures (#15642), core_fingerprint 63e161964d00f30ff8c9.

Not covered: full ci-local.sh, AOT smoke timing on clean checkout (issue repro from #24275).

[PurHur]

Re-measured after #24321 — the user-facing cost is unchanged, and I found the actual cause

#24321 refreshed the committed cache so the fingerprints now agree:

committed core_fingerprint: 63e161964d00f30ff8c9
live coreFingerprint():     63e161964d00f30ff8c9

But measured on master a279902fc, with build/helper-runtime-cache removed to simulate a clean
checkout:

before #24321 : first compile 533 s
after  #24321 : first compile 517 s      <- ~16 s of ~530
second compile:                5 s

So the refresh bought essentially nothing for the case the issue is about. Not a criticism of the
change — a current committed cache is worth having — but the blocker it was closed for is still
there.

Root cause: the warmup never consults the committed cache

HelperRuntimeCache::warmForUserAotBuild() decides whether to emit the whole corpus based on one
thing: whether a marker file exists.

$marker = self::coreMarkerPath();     // build/helper-runtime-cache/<prefix><fingerprint>.ok
if (is_file($marker)) { return; }
...
$rc = self::runWarmupCommand($cmd);   // shells out to emit-helper-runtime-object.php

coreMarkerPath() lives under cacheDir() = build/helper-runtime-cache, and that directory is
gitignored (.gitignore:29, 0 files tracked). So a clean checkout can never have the marker, and
the first user AOT build re-emits the corpus regardless of whether prelinked/helper-runtime/ is
present and current.

That is also why refreshing the committed cache changed nothing: the refresh fixes what the
consumption path can use, but the warmup decision does not look at it at all.

I found this by profiling rather than guessing — a SIGALRM sampler on the real bin/compile.php
showed only 5 ticks in 441 s, with the top path being
HelperRuntimeCache::runWarmupCommand < warmForUserAotBuild. The parent is blocked on a subprocess,
which is why parent-side sampling sees almost nothing; that near-empty profile is itself the signal.

Fix and measurement

Skip the warmup when the committed per-arch cache exists, carries a core_fingerprint equal to the
live one, and actually contains units — the consumption path already reads prelinkedUnitsDir()
alongside unitsDir(), so in that state there is nothing to emit.

first compile, clean checkout:  517 s  ->  5 s

Binary output unchanged (hi). The check is deliberately conservative — missing manifest, mismatched
fingerprint, unreadable JSON or an empty units directory all fall through to the warmup, so a stale
or partial committed cache still gets emitted rather than silently producing a build with missing
helpers.

Running the differential sweep against it now and will PR with the name-level comparison. Reopening
since the reported symptom — clone, compile hello world, wait ~9 minutes — is still reproducible on
master.

[PurHur]

claim: compile-link-trust-gate — starting this run (Pillar 1: user_release_ready: no — helper-runtime-prelink fingerprint mismatch + warmup ignores committed cache)

[PurHur]

Fixed and now gated

#24351 fixed the cause — the corpus warmup is skipped when the committed per-arch cache is current.
Verified on master:

first compile, clean checkout:  517 s  ->  5 s
second compile:                   5 s

#24361 adds script/cold-build-check.sh and wires it into release-readiness so this cannot drift
back unnoticed. It measures the cold path in a temp cache dir (no damage to a developer's real
cache) and bounds it with a timeout, so a regression fails fast instead of costing nine minutes.

Validated in both directions: ok/5s at HEAD, FAIL/timeout at a279902fc.

Why the existing prelink gate was not enough

check-helper-runtime-prelink --strict was already gating this issue, and it is worth having — but
it checks the precondition (is the committed cache current), not the outcome.

At a279902fc the committed cache was current and the fingerprints matched, so that gate would
have reported green while a clean checkout still took ~9 minutes. Two of the three attempts on this
issue — #24321 and my own first hypothesis — went at the precondition for exactly that reason.

Closing. If it regresses, release-readiness will now say so.