Compliance: 9 VMTest cases flip between two runs of the same commit #24672

Summary

Two complete 24-shard VMTest runs of the same commit, same image, zero timeouts, produced
different failing sets. 9 cases flipped; 422 were stable in both.

run 1: 426 failing        run 2: 427 failing
stable in both: 422       flipped: 9

The nine

Failed in run 1, passed in run 2 — these are in the committed baseline (#24634), so they would
show as phantom fixes:

stdlib/mt_getrandmax_jit
stdlib/stream_resolve_include_path_dot
stdlib/string_builtin_null_coerce_batch2
stdlib/string_builtin_null_coerce_batch2_jit

Passed in run 1, failed in run 2 — these are not in the baseline, so they would show as phantom
regressions, the more damaging direction:

language/reflection_property_is_abstract
stdlib/hrtime_nanosecond_precision
stdlib/mkdir
stdlib/mkdir_jit
stdlib/pcntl_setns

Patterns worth noting

Plain/_jit pairs flip together. string_builtin_null_coerce_batch2 with its _jit variant,
and mkdir with mkdir_jit. Two backends of the same case moving in lockstep points at one shared
cause rather than four independent flakes — likely the case itself, not the lowering.

hrtime_nanosecond_precision confirms a documented suspicion. AGENTS.md §2 already lists
stdlib/hrtime* as order-dependent or flaky. This is the first measurement backing that: it passed
one run and failed the next on identical inputs.

Filesystem and process cases dominate the pass→fail direction. mkdir, mkdir_jit,
pcntl_setns — plausibly environment-sensitive (leftover state, namespace availability) rather than
nondeterminism in the compiler. Worth checking whether they depend on prior cases in shard order.

mt_getrandmax_jit involves randomness and is the least surprising member of the list.

Why this matters

A single-run baseline is wrong in 9 places and looks authoritative. Anyone diffing against it
would chase five regressions that are not regressions, and credit four fixes that did not happen.

That is not hypothetical here: an e08_spread "regression" in the differential corpus measured
8/10 vs 4/10 before 30 runs showed 17/30 vs 17/30 — pure noise that cost real investigation time.

Handling

All nine go into test/compliance/quarantine.txt, excluded from both sides of the baseline diff, and
the baseline is regenerated without them. Quarantine entry is a bug to fix, not a resting place —
each of these is a real defect in either the case or the runtime, and the goal is to empty that file.

Determinism is otherwise strong: 422 of 431 distinct failing names (98%) were identical across two
independent 2.1-hour runs.

Environment: php-compiler:22.04-dev, PHP 8.2.32, LLVM 9, master 36bd525a2.