Repository navigation
Compliance: JITTest baseline, and scope the quarantine per suite - #24880
Merged
PurHur merged 1 commit intoJul 29, 2026
Merged
Conversation
JITTest gets its first committed baseline, and the quarantine gains suite scoping — because collecting that baseline exposed the need for it. Two complete 24-shard JITTest collects at the SAME commit e7de997 (runs 30464127933 and 30464157996), same runner image: 491 failing names, 6,912 executed, and ZERO disagreements between the runs on either set. So JITTest is deterministic when the tree is held constant, exactly as VMTest turned out to be (8,112 executed, one flip). Neither suite needs a broad quarantine; the earlier belief that they did came from comparing runs of different commits (#24726). The scoping is not cosmetic. stdlib/hrtime_nanosecond_precision is genuinely flaky under VMTest (#24870) and was quarantined for it — but under JITTest it fails STABLY in both runs of e7de997. A suite-agnostic quarantine dropped that stable failure out of the JITTest baseline, which is precisely the "real failure made invisible" harm the quarantine itself caused in #24726. Lines may now be written as: VMTest:stdlib/hrtime_nanosecond_precision # only under VMTest stdlib/something # every suite Bare names keep their old meaning, so this is backward compatible. A scope prefix is only recognised when it names the suite being run; case names contain no colon, so nothing else parses as scoped. Verified: - JITTest baseline collected from run 30464127933 diffs CLEAN (exit 0) against run 30464157996 — the run it was not built from - scoping raises JITTest from 490 to 491 names (hrtime no longer hidden) - VMTest is unchanged at 428 with quarantined=1, still clean against the run its baseline was not built from - cross-suite behaviour tested on synthetic fixtures: a VMTest-scoped entry is NOT quarantined under JITTest and vice versa, while a bare entry is quarantined under both Collected at e7de997; master has since moved, so the first diff may show drift — the DROPPED / FIXED / NEWLY EXECUTED split will classify it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
4 tasks
PurHur
deleted the
compliance/jittest-baseline-and-suite-scoped-quarantine
branch
July 30, 2026 00:44
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
JITTest gets its first committed baseline, and the quarantine gains suite scoping — because collecting that baseline exposed the need for it.
JITTest is deterministic too
Two complete 24-shard collects at the same commit
e7de99700, same runner image — 30464127933 and 30464157996:Matching what VMTest turned out to be (8,112 executed, one flip). Neither suite needs a broad quarantine; the earlier belief that they did came from comparing runs of different commits (#24726).
Why scoping, and why it is not cosmetic
stdlib/hrtime_nanosecond_precisionis genuinely flaky under VMTest (#24870) and quarantined for it. Under JITTest it fails stably — in both runs ofe7de99700.A suite-agnostic quarantine therefore dropped a stable JITTest failure out of that baseline. That is exactly the "real failure made invisible" harm the quarantine itself caused in #24726, reproduced in miniature by the fix for it.
Bare names keep their old meaning, so this is backward compatible. A prefix is only treated as a scope when it names the suite being run; case names contain no colon, so nothing else parses as scoped.
Verified
quarantined: 1, still clean against the run its baseline was not built fromVMTest:-scoped entry is not quarantined under JITTest and vice versa, while a bare entry is quarantined under bothNote
Collected at
e7de99700; master has since moved, so the first diff may show drift. The DROPPED / FIXED / NEWLY EXECUTED split added in #24835 will classify it rather than silently reporting vanished cases as fixed.🤖 Generated with Claude Code