The question
Now that VMTest has a validated baseline (#24673: 422 failing names, 9 flaky quarantined, stable
across two independent runs), the failures can be located rather than counted. They are not spread
evenly across the language — they cluster in a few extensions.
That turns "422 failing cases" from a vague quality signal into a release-scoping decision.
Measured, from the complete run
Rates are failing ÷ cases actually executed per area (8,023 of 8,844 ran; the rest are withheld
by the suite's own host-library policies):
| area |
failing / executed |
rate |
halt_compiler |
1 / 1 |
100% |
gmp |
6 / 6 |
100% |
intl |
56 / 63 |
88.9% |
types |
5 / 23 |
21.7% |
pcre |
4 / 20 |
20.0% |
string |
7 / 48 |
14.6% |
language |
104 / 1788 |
5.8% |
stdlib |
~230 / ~5089 |
~4.5% |
Completely clean, with meaningful volume:
spl 0 / 121 mbstring 0 / 45
phar 0 / 21 openssl 0 / 21
xmlwriter 0 / 20
Why the shape matters more than the total
stdlib and language carry the largest absolute counts — together ~334 of 422 — but they are the
strongest areas proportionally, at ~5%. A 5% diffuse failure rate across the core language is
hard to ship and hard to describe to a user.
intl and gmp are the opposite: 62 failures concentrated in two extensions that barely work at
all. gmp fails every case that runs. intl fails 56 of 63.
Concentrated debt is shippable; diffuse debt is not. A release that says "intl and gmp are not
supported in this version" is honest and actionable. A release that silently half-implements them is
neither — a user calling numfmt_create() gets a wrong answer rather than a clear "unsupported".
Proposal
Mark intl and gmp explicitly unsupported for the first release, rather than shipping them
silently broken:
- document them as unsupported in the release notes and capability matrix;
- consider making their entry points fail loudly rather than wrongly, so the failure mode is a clear
error rather than a silent miscompile — the project's own worst-bug category;
- keep the compliance cases enabled and in the baseline, so the debt stays visible and progress is
measurable when someone picks them up.
This is a maintainer decision about what the release claims, not something to fix case by case — 62
cases across two extensions is weeks of work, and shipping without deciding is the option that
produces silent wrong answers in the field.
halt_compiler at 1/1 is noted for completeness; a single case is not a scoping question.
Reproducing
for i in $(seq 0 23); do script/shard-compliance.sh --suite=VMTest --shards=24 --shard=$i; done
script/compliance-baseline.sh --diff --suite=VMTest
Concentration is derived from the per-shard .log TeamCity testStarted events (executed counts)
against test/compliance/baselines/VMTest.failing.
Environment: php-compiler:22.04-dev, PHP 8.2.32, LLVM 9, master 9f1817c27.
The question
Now that
VMTesthas a validated baseline (#24673: 422 failing names, 9 flaky quarantined, stableacross two independent runs), the failures can be located rather than counted. They are not spread
evenly across the language — they cluster in a few extensions.
That turns "422 failing cases" from a vague quality signal into a release-scoping decision.
Measured, from the complete run
Rates are failing ÷ cases actually executed per area (8,023 of 8,844 ran; the rest are withheld
by the suite's own host-library policies):
halt_compilergmpintltypespcrestringlanguagestdlibCompletely clean, with meaningful volume:
Why the shape matters more than the total
stdlibandlanguagecarry the largest absolute counts — together ~334 of 422 — but they are thestrongest areas proportionally, at ~5%. A 5% diffuse failure rate across the core language is
hard to ship and hard to describe to a user.
intlandgmpare the opposite: 62 failures concentrated in two extensions that barely work atall.
gmpfails every case that runs.intlfails 56 of 63.Concentrated debt is shippable; diffuse debt is not. A release that says "intl and gmp are not
supported in this version" is honest and actionable. A release that silently half-implements them is
neither — a user calling
numfmt_create()gets a wrong answer rather than a clear "unsupported".Proposal
Mark
intlandgmpexplicitly unsupported for the first release, rather than shipping themsilently broken:
error rather than a silent miscompile — the project's own worst-bug category;
measurable when someone picks them up.
This is a maintainer decision about what the release claims, not something to fix case by case — 62
cases across two extensions is weeks of work, and shipping without deciding is the option that
produces silent wrong answers in the field.
halt_compilerat 1/1 is noted for completeness; a single case is not a scoping question.Reproducing
Concentration is derived from the per-shard
.logTeamCitytestStartedevents (executed counts)against
test/compliance/baselines/VMTest.failing.Environment:
php-compiler:22.04-dev, PHP 8.2.32, LLVM 9, master9f1817c27.