Repository navigation
test: fix intermittent mllama exact-zero assert and webui router race #2467
Copy link
Copy link
Closed
Labels
priority:lowLow priorityLow prioritystatus:doneCompletedCompletedtype:testTest related changesTest related changes
Description
Problem / Background
Two intermittent test failures, both fixable in test code only. Fix both in one PR, run the verification pass once at the end, and keep pushes to a minimum.
tests/mllama_parity.rsragged_multi_image_states_concatenate_real_rowsfailed once on main 89ced0a (full workspace run,--test-threads=1, GB10) withmax_abs_diffof 1.1920929e-7 where the assert requires exactly 0.0. Six reruns passed.- WebUI
webui/tests/router-real.harness.tstest "secured Rust router drives download, load, chat, Stop/drain, unload, remove and negative security cases" intermittently fails atexpect(page.getByLabel(/Session key|세션 키/i)).toBeVisible()(5 s timeout) afterrefresh(page)ininjectRealUnauthorizedCatalog. Seen on PR CI during the auto-impl run and on main 781126a; reruns pass.
Current Behavior
- Item 1: the exact-zero assert is at
tests/mllama_parity.rsnear line 620 (max_abs_diff(&sub, &expected), 0.0). The sibling tile-selection test usesTILE_SELECTION_REASSOCIATION_TOL(line 261, 1e-6) per fix(tests): tolerate f32 reassociation in the mllama tile-selection parity test #1067 for the same reason. - Item 2:
injectRealUnauthorizedCatalog(around line 247) installs a one-shot route (times: 1) that rewrites the next catalog request'sauthorizationheader toBearer expired-router-key, then callsrefresh(page)and waits for the session-key field. Suspects: the field renders after the 401 response and a retry, a second catalog request consumes or bypasses the one-shot route, or the refresh races the route mock.
Proposed Solution
- Replace the exact-zero assert with the same documented reassociation tolerance (reuse the constant or add a sibling with a comment explaining why reduction order can differ by one ulp).
- Root-cause the race: log requests and responses around the refresh, determine which signal means "401 handled and the key prompt is mounted" (for example the catalog response status or a DOM state), and wait on that signal. Do not just raise the timeout.
Scope
In scope: tests/mllama_parity.rs, webui/tests/router-real.harness.ts.
Out of scope: product code, unless the root cause shows the UI genuinely renders the prompt late; then fix that and say so.
Implementation Notes
- Reuse:
TILE_SELECTION_REASSOCIATION_TOL; existing Playwright helpers in the harness. - Constraints: keep the tolerance tight enough to fail on a real row-order defect (the test checks media-major ordering).
- Edge cases: the route mock must still apply to exactly one request.
- Error handling: none new.
Acceptance Criteria
- Item 1: the assert uses a documented tolerance no looser than 1e-6, and a comment states why. A deliberately swapped row order still fails the test.
- Item 2: the root cause is stated in the PR, the wait uses the correct signal, and the harness passes 20 consecutive runs, including at least some under CPU load.
- The harness test still covers the negative security case it was written for (the 401 path is still exercised).
Verification
export MLX_ENABLE_TF32=0
cargo test --profile test-fast --features cuda --test mllama_parity -- --test-threads=1
cd webui && for i in $(seq 1 20); do npx playwright test tests/router-real.harness.ts || echo "FAIL run $i"; done
Check the exact harness invocation in webui/package.json and CI before running. A pass is 0 failures in 20 runs.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
priority:lowLow priorityLow prioritystatus:doneCompletedCompletedtype:testTest related changesTest related changes
Activity
fix(cuda): take the batch stride in sorted gather_mm, make captures b…
docs: add PR 2517 technical report
test: fix flaky mllama exact-zero assert and webui 401 wait (#2517)