Repository navigation
fix: stabilize nightly verify-test failures (#1256) - #1567
Merged
Merged
Conversation
Issue #1256 was failing for two separate reasons on main. `FAMILY_ORDER` omitted the newly added Inkling family, which made `tests::family_order_is_exhaustive` fail deterministically. The `temperature_one_support_unchanged` regression was also too strict for Metal snapshots: the saved support and probabilities remained stable, but nightly runs could drift by a final f32 ULP. Add Inkling to the CLI family ordering and compare the saved `T == 1.0` sampling snapshot by exact support plus a two-ULP tolerance for positive probabilities. This keeps the intended regression guard while removing the nightly flake. Validation: `cargo fmt --all --check`; `cargo test -p mlxcel-core sampling::tests::temperature_one_support_unchanged -- --exact --nocapture`; `cargo test --bin mlxcel tests::family_order_is_exhaustive -- --exact --nocapture`; `cargo test --bin mlxcel tests::family_order_has_no_orphans -- --exact --nocapture`.
Follow up the issue #1256 fix with the requested stricter regression boundary. Move Inkling next to the LFM2 family block so the CLI order mirrors the ModelType declaration more closely, and keep the routed `temperature_one_support_unchanged` snapshot at a one-ULP tolerance. The non-routed stock-chain case still preserves exact support, but it now compares values with a small numeric tolerance because the local CPU backend drifted by two ULP while preserving the same filtered distribution. Validation: `cargo test -p mlxcel-core sampling::tests::temperature_one_support_unchanged -- --exact --nocapture`; `cargo test --bin mlxcel tests::family_order_is_exhaustive -- --exact --nocapture`; `cargo test --bin mlxcel tests::family_order_has_no_orphans -- --exact --nocapture`; `cargo fmt --all --check`.
The previous follow-up loosened the non-routed `temperature_one_support_unchanged` stock-chain case to an absolute epsilon, which was broader than intended. Replace that with the same snapshot helper used by the routed cases: exact support plus a `max_ulp` ceiling. Local validation on this Linux host showed why the stock-chain case needs `max_ulp = 2`: `cargo test -p mlxcel-core sampling::tests::temperature_one_support_unchanged -- --exact --nocapture` failed at token 12 of the `(top_k=40, top_p=1.0)` case with `0.0016345085` vs `0.0016345083`, a two-ULP drift on a low-probability entry. The routed nightly failure remains capped at one ULP. Validation: `cargo test -p mlxcel-core sampling::tests::temperature_one_support_unchanged -- --exact --nocapture`; `cargo fmt --all --check`.
inureyes
added
type:bug
This was referenced Sep 1, 2026
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fix the two known
make verify-testfailure modes behind issue #1256: the deterministicFAMILY_ORDERdrift after adding Inkling, and the intermittenttemperature_one_support_unchangedsnapshot flake on nightly Metal runs.What Changed
Inklingfamily toFAMILY_ORDERbeside the relatedLFM2model block somlxcel archremains exhaustive in a deliberate order.sampling::tests::temperature_one_support_unchangedstrict on exact support while comparing positive finite snapshot probabilities by ULP budget instead of full bit identity.Validation
cargo fmt --all --checkcargo test -p mlxcel-core sampling::tests::temperature_one_support_unchanged -- --exact --nocapturecargo test --bin mlxcel tests::family_order_is_exhaustive -- --exact --nocapturecargo test --bin mlxcel tests::family_order_has_no_orphans -- --exact --nocapturefmt,clippy, fullmake verify-test, and ffmpeg video tests all passed on PR HEAD (run 33505774122)The full
make verify-testtarget was also attempted on the local Linux/aarch64 host after allowing its MLX FetchContent download. It reached workspace linking, then failed because the macOS-orientedmetal,acceleratefeature set lacks the MLX symbolsmlxcel_set_qmv_wideandmlxcel_qmv_wideon this Linux target. This is a platform gate, not a test failure or macOS execution proof.Closes #1256