Skip to content

test: fix intermittent mllama exact-zero assert and webui router race #2467

Closed

Description

@inureyes

Problem / Background

Two intermittent test failures, both fixable in test code only. Fix both in one PR, run the verification pass once at the end, and keep pushes to a minimum.

  1. tests/mllama_parity.rs ragged_multi_image_states_concatenate_real_rows failed once on main 89ced0a (full workspace run, --test-threads=1, GB10) with max_abs_diff of 1.1920929e-7 where the assert requires exactly 0.0. Six reruns passed.
  2. WebUI webui/tests/router-real.harness.ts test "secured Rust router drives download, load, chat, Stop/drain, unload, remove and negative security cases" intermittently fails at expect(page.getByLabel(/Session key|세션 키/i)).toBeVisible() (5 s timeout) after refresh(page) in injectRealUnauthorizedCatalog. Seen on PR CI during the auto-impl run and on main 781126a; reruns pass.

Current Behavior

  • Item 1: the exact-zero assert is at tests/mllama_parity.rs near line 620 (max_abs_diff(&sub, &expected), 0.0). The sibling tile-selection test uses TILE_SELECTION_REASSOCIATION_TOL (line 261, 1e-6) per fix(tests): tolerate f32 reassociation in the mllama tile-selection parity test #1067 for the same reason.
  • Item 2: injectRealUnauthorizedCatalog (around line 247) installs a one-shot route (times: 1) that rewrites the next catalog request's authorization header to Bearer expired-router-key, then calls refresh(page) and waits for the session-key field. Suspects: the field renders after the 401 response and a retry, a second catalog request consumes or bypasses the one-shot route, or the refresh races the route mock.

Proposed Solution

  1. Replace the exact-zero assert with the same documented reassociation tolerance (reuse the constant or add a sibling with a comment explaining why reduction order can differ by one ulp).
  2. Root-cause the race: log requests and responses around the refresh, determine which signal means "401 handled and the key prompt is mounted" (for example the catalog response status or a DOM state), and wait on that signal. Do not just raise the timeout.

Scope

In scope: tests/mllama_parity.rs, webui/tests/router-real.harness.ts.

Out of scope: product code, unless the root cause shows the UI genuinely renders the prompt late; then fix that and say so.

Implementation Notes

  • Reuse: TILE_SELECTION_REASSOCIATION_TOL; existing Playwright helpers in the harness.
  • Constraints: keep the tolerance tight enough to fail on a real row-order defect (the test checks media-major ordering).
  • Edge cases: the route mock must still apply to exactly one request.
  • Error handling: none new.

Acceptance Criteria

  • Item 1: the assert uses a documented tolerance no looser than 1e-6, and a comment states why. A deliberately swapped row order still fails the test.
  • Item 2: the root cause is stated in the PR, the wait uses the correct signal, and the harness passes 20 consecutive runs, including at least some under CPU load.
  • The harness test still covers the negative security case it was written for (the 401 path is still exercised).

Verification

export MLX_ENABLE_TF32=0
cargo test --profile test-fast --features cuda --test mllama_parity -- --test-threads=1
cd webui && for i in $(seq 1 20); do npx playwright test tests/router-real.harness.ts || echo "FAIL run $i"; done

Check the exact harness invocation in webui/package.json and CI before running. A pass is 0 failures in 20 runs.

Activity

  1. added and removed on Oct 21, 2026
  2. added a commit that references this issue on Oct 21, 2026
  3. added a commit that references this issue on Oct 21, 2026
    93e75e9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions