Skip to content

test(data): fix flaky quantized depth roundtrip tests#159

Merged
janickm merged 1 commit into
NVIDIA:mainfrom
janickm:dev/janickm/fix-flaky-quantized-depth-test
Jun 22, 2026
Merged

test(data): fix flaky quantized depth roundtrip tests#159
janickm merged 1 commit into
NVIDIA:mainfrom
janickm:dev/janickm/fix-flaky-quantized-depth-test

Conversation

@janickm

@janickm janickm commented Jun 22, 2026

Copy link
Copy Markdown
Collaborator

Problem

The test_quantized_depth_roundtrip test in ncore/impl/data/v4/components_test.py fails intermittently in CI (e.g. run 27951090321):

test_quantized_depth_roundtrip
>   np.testing.assert_allclose(dequantized, original, atol=0.5 * quant.scale, rtol=0)
E   AssertionError

Root cause

Two issues compound:

  1. Unseeded RNG -- the three quantized-roundtrip tests used np.random.default_rng() with no seed, so each CI run drew different data and results were non-reproducible.
  2. Too-tight tolerance -- the ideal quantization error is at most 0.5 * scale, but the source original is float32 and the dequantized output is also cast back to float32. Each carries up to ~max_value * eps_f32 of representation error on top of the rounding error. The measured |dequantized - original| therefore occasionally exceeds the exact 0.5 * scale bound (about 0.8% of random seeds for the float64-intermediate case), tripping the assertion with rtol=0.

Fix

  • Seed the RNG (np.random.default_rng(0)) in all three quantized roundtrip tests so runs are deterministic and reproducible.
  • Widen the tolerance for the float64-intermediate uint16 case to 0.5 * scale + 2 * max_value * eps_f32, reflecting the real float32 representation error budget. The other two cases already used looser bounds (scale / 1.0 * scale).

Verified the widened bound holds across 5000 seeds (worst observed error 0.000500 vs atol 0.000514), so the test stays green even if the seed changes later.

Testing

bazelisk test //ncore/impl/data/v4:pytest_components_3_11
//ncore/impl/data/v4:pytest_components_3_11   PASSED

The quantized depth roundtrip tests used an unseeded np.random RNG and an
exact 0.5 * scale tolerance. The float32 source data and the float32 cast of
the dequantized output each carry up to ~max_value * eps_f32 of representation
error on top of the quantization rounding error, so the measured difference
occasionally exceeds 0.5 * scale (about 0.8% of random seeds for the float64
intermediate case), making the test flaky in CI.

Seed the RNG for deterministic, reproducible runs and widen the tolerance to
account for float32 representation error. Verified the widened bound holds
across 5000 seeds.
@janickm janickm self-assigned this Jun 22, 2026
@janickm janickm added this pull request to the merge queue Jun 22, 2026
Merged via the queue into NVIDIA:main with commit 8b8c22b Jun 22, 2026
5 checks passed
@janickm janickm deleted the dev/janickm/fix-flaky-quantized-depth-test branch June 22, 2026 15:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant