Repository navigation
Docs: record Phase 1 complete with the measured compliance numbers - #24701
Merged
Merged
Conversation
The compliance suite is now a gate rather than merely runnable. Records what two complete independent 24-shard runs of one commit measured: executed 8,023 of 8,844 (rest withheld by host-library policies) failing (stable) 422 flaky 9, quarantined with direction (#24672) determinism 422 of 431 distinct names identical = 98% Marks P1.3 and P1.4 done, and states the finding that justifies the two-run protocol: the single-run baseline was wrong in 9 places while looking authoritative, 5 of them in the phantom-regression direction. The 422-name baseline diffs clean against BOTH runs, including the one it was not built from — two runs that disagreed on 9 cases now both report no regressions. Also records the concentration result (#24697): gmp 6/6 and intl 56/63 failing, together 62 of 422, while spl/mbstring/phar/openssl/xmlwriter are perfectly clean and stdlib/language carry the largest counts at the lowest rates (~5%). That makes the debt a release-scoping decision rather than a quality verdict. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Owner
Author
|
Cleanup: deleted leftover head branch |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 1 — "make correctness observable" — is done. This records it with numbers rather than
adjectives.
What the suite now does
Measured over two complete independent runs of one commit, 2.1 h each, zero timeouts:
The finding that justifies the two-run protocol
A single-run baseline was wrong in 9 places and looked authoritative. Five of those were absent
from it but failing intermittently — they would have surfaced as phantom regressions, sending
someone bisecting a change that did nothing.
The validation that matters is not that the baseline exists, but that it is stable: the 422-name
baseline diffs clean against both runs, including the one it was not built from. Two runs that
disagreed on 9 cases now both report no regressions.
The debt is concentrated (#24697)
gmpfails 6/6 andintl56/63 — 62 of 422 in two extensions — whilespl(121 executed),mbstring(45),phar(21),openssl(21) andxmlwriter(20) are perfectly clean.stdlibandlanguagecarry the largest absolute counts at the lowest rates (~5%).That reframes the total: it is a release-scoping decision about what the release claims to support,
not a diffuse quality problem.
Docs only.