Skip to content

Cut test suite runtime in binary roundtrips and integer sweeps - #5519

Merged
nlohmann merged 3 commits into
nlohmann:developfrom
ayush-singh-0601:test-suite-runtime
Sep 27, 2026
Merged

nlohmann merged 3 commits into
nlohmann:developfrom
ayush-singh-0601:test-suite-runtime

Conversation

@ayush-singh-0601

Copy link
Copy Markdown
Contributor

See #5418.

Linux CI runs --no-skip, so the expensive binary roundtrip cases actually run there. Each corpus file was parsed four times (once per input adapter). Parse once and reuse it.

The 16-bit integer sweeps still prove width selection: stride 7 hits every low-byte residue, and the endpoints stay in the loop.

unit-large_json used depth 5,000,000 to check the non-recursive destructor from #1419. 500,000 is enough. jeopardy.json is ~500 MB of serialization output, so it is now its own skipped test; canada/twitter/citm/sample keep running.

Unicode byte sweeps are left to #5426.

  • The changes are described in detail, both the what and why.
  • If applicable, an existing issue is referenced.
  • The Code coverage remained at 100%. A test case for every new line of code.
  • If applicable, the documentation is updated.
  • The source code is amalgamated by running make amalgamate.

@nlohmann

Copy link
Copy Markdown
Owner

Please rebase to the latest develop branch as I had to fix some tests.

The Linux CI jobs pass --no-skip, so skip() does not help there.
Parse each corpus file once in the binary roundtrip loops instead of
four times. Sample the 16-bit integer ranges with stride 7 (still hits
every low byte) and always keep the endpoints.

Also drop the 5M-node parse test to 500k, which still covers the
non-recursive destructor, and move jeopardy.json into its own skipped
test so the cheaper binary-format size checks actually run.

See nlohmann#5418.

Signed-off-by: ayush-singh-0601 <singhayush062006@gmail.com>
ci_test_gcc compiles with -Werror=useless-cast. On that compiler
int32_t is int, so static_cast<int32_t> of the loop bound is an
error. The bounds are already int, and the sampled values do not
change.

Signed-off-by: ayush-singh-0601 <singhayush062006@gmail.com>
@ayush-singh-0601

Copy link
Copy Markdown
Contributor Author

Rebased onto the latest develop.

The gcc job was failing as well, so I fixed that in the same push. static_cast<int32_t> on the sampled loop bounds is a useless cast there (int32_t is just int), and ci_test_gcc builds with -Werror. Those bounds are plain ints now. The ranges, the stride of 7, and the endpoints are unchanged.

@nlohmann nlohmann removed the please rebase Please rebase your branch to origin/develop label Sep 23, 2026

@nlohmann nlohmann left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Most of this PR is fine to merge. The stride-7 integer sweeps still hit every boundary and every byte value, the roundtrip hoisting only removes duplicated parsing, and 500k depth is still plenty for unit-large_json.

I'd like one change: please revert tests/src/unit-binary_formats.cpp to develop's version.

The split doesn't save time, because of how skipped tests run in CI (cmake/test.cmake):

Job type Command Runs skip() tests?
Default (gcc/clang/sanitizer/coverage) test-foo --no-skip yes
JSON_FastTests=ON test-foo no
Valgrind valgrind test-foo no

On develop, "Binary Formats" is skipped, so it runs only in the default jobs. With this PR:

  • Default --no-skip jobs still run both test cases, jeopardy.json included, so there is no saving.
  • FastTests and valgrind jobs never ran any of this before. Now they run canada/twitter/citm/sample. Under valgrind, test-binary_formats_valgrind went from about 2 s to about 223 s (compared with develop CI run 35829411620). That makes the valgrind job slower overall (ctest time 390 s → 527 s), even though the other changes should have made it faster.

So the cheaper files don't "keep running": they never ran in those jobs. It isn't a coverage gain either. The only checks that run in more places are the encoded-size checks for four files, and the default and coverage jobs already run those.

If you'd like something from that file in the fast jobs, un-skipping just sample.json would be acceptable, since it's small. Otherwise, a plain revert is best.

Nit: the comment on next_integer_sample in tests/src/test_utils.hpp says "Walk [first, last]", but the function has no first parameter.


This review was written by Claude Code on behalf of @nlohmann.

Revert tests/src/unit-binary_formats.cpp to its develop state.
The test-case split made valgrind jobs slower instead of faster,
because the cheaper corpus files (canada/twitter/citm/sample)
now ran under valgrind where they never did before.

Fix the next_integer_sample comment: the function has no 'first'
parameter, so describe what the function actually does.

Signed-off-by: ayush-singh-0601 <singhayush062006@gmail.com>
@ayush-singh-0601

Copy link
Copy Markdown
Contributor Author

Addressed both points from the review:

  1. Reverted tests/src/unit-binary_formats.cpp to develop. The split was counterproductive -- the cheaper corpus files never ran in fast/valgrind jobs before, and introducing them there inflated valgrind runtime from ~390 s to ~527 s with no coverage gain. Everything is back to a single skipped test case.

  2. Fixed the next_integer_sample comment in tests/src/test_utils.hpp. The old wording said "Walk [first, last]" but the function has no first parameter. Updated to "Advance i toward last (inclusive) by stride", which matches the actual signature.

No other files were touched. Ready for another look when you have a chance.

@nlohmann nlohmann left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

@nlohmann nlohmann added this to the Release 3.13.0 milestone Sep 27, 2026
@nlohmann nlohmann added the 🚀 ready to merge Ready to merge - just waiting for CI to complete. label Sep 27, 2026
@nlohmann
nlohmann merged commit 98e00d2 into nlohmann:develop Sep 27, 2026
5 of 160 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

aspect: binary formats BSON, CBOR, MessagePack, UBJSON L 🚀 ready to merge Ready to merge - just waiting for CI to complete. tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants