Speed up whitespace skipping in the lexer

[nlohmann]

Summary

Fixes #5412.

Stacked on top of #5484 (fix for #5411) — this PR's diff only makes sense on top of that branch; only the last commit here is new.

lexer::skip_whitespace() calls get() once per whitespace byte. get() itself checks a next_unget flag on every call to see whether it should replay a previously-ungotten character rather than reading a fresh one from the input adapter. That check only ever matters for the first character skip_whitespace() reads (a previous token, e.g. a number, may have ended with unget(), leaving one pending character to replay) — nothing inside skip_whitespace()'s loop itself calls unget(), so next_unget is provably false for every whitespace character after the first.

This PR reads the first character with the existing get() (unchanged), and every subsequent whitespace character with a new get_ignoring_pending_unget() that shares get()'s bookkeeping tail but skips the now-provably-dead next_unget branch. Behavior is unchanged for the first character read for every token as before.

Scope and what was deliberately left out

The issue's suggested direction is a full contiguous-buffer bulk skip: scan a run of whitespace directly in the input adapter's buffer with a SWAR/memspn-style scan, and update the position counters once per run instead of once per byte. That depends on bulk-scan adapter infrastructure (supports_bulk_scan/bulk_data()/bulk_skip()) that the issue explicitly says is introduced by the open, unmerged parser-performance PR #5283. Per the scoping for this fix, I did not build that adapter infrastructure from scratch here, and did not depend on or replicate #5283. I judged writing new bulk-scan/adapter-peeking infrastructure targeted at a security-sensitive parser, without the review #5283 itself is still undergoing, to be out of scope for this PR.

Instead, this PR is limited to the safe, always-correct improvement described in the fallback scoping for this issue: removing redundant per-character bookkeeping from the existing byte-at-a-time loop. Full bulk-skipping is left as future work once #5283 (or equivalent contiguous-adapter support) lands upstream.

Given this narrower scope, the throughput improvement here is real but far more modest than the ~25% of accept() time cited in the issue for the full bulk-skip approach (which comes from amortizing per-byte counter updates across whole whitespace runs, not from removing one branch per byte).

Behavior preservation / testing

• Line/column/byte-offset bookkeeping (chars_read_total, chars_read_current_line, lines_read) is completely untouched by this change — get_ignoring_pending_unget() performs the exact same steps get() does when next_unget is (provably) false.
• Added a differential regression test in tests/src/unit-class_parser.cpp (SECTION("issue #5412 - whitespace skipping bookkeeping (compact vs. pretty-printed)")) asserting the exact byte offset, line, column, and error message for parse errors in both compact and pretty-printed (dump(4)) malformed input, including input with an error after multiple embedded newlines/indentation.
• Verified with a standalone harness that error byte/what() output is byte-for-byte identical before and after this change across compact and pretty-printed inputs (including inputs truncated right before the final closing brace, and inputs with an invalid token after several indented lines).
• Ran the full required test files locally (offline, without the json_test_data submodule available in this environment): unit-class_lexer.cpp, unit-class_parser.cpp, unit-deserialization.cpp, unit-regression1.cpp, unit-regression2.cpp, unit-udt.cpp, unit-class_parser_diagnostic_positions.cpp (9234 assertions, all pass) — all pass, with the same 7 pre-existing unit-regression1.cpp failures present identically on unmodified develop in this offline environment (missing json_test_data fixture files; unrelated to this change).
• make amalgamate was run and single_include/nlohmann/json.hpp is included in this PR.

Breaking change?

No breaking changes. This only adds a new private helper method to nlohmann::detail::lexer and changes the internal implementation of skip_whitespace(); no public API is touched.

— opened by Claude Code on behalf of @nlohmann

[nlohmann]

Verified with a local benchmark (json::accept() over the same ~20k-object document rendered compact vs. dump(4)-pretty-printed, clang++ -O2 -DNDEBUG, Apple Clang 21 / arm64 macOS, median of 150 timed iterations, comparing this branch's base (issue-5411-lexer-skip-conversion tip) vs. this branch's tip):

• Compact (1.27 MB, no whitespace): 4.89 ms before → 4.74 ms after (about 3% faster)
• Pretty-printed (2.27 MB, ~1 MB of added whitespace): 6.13 ms before → 6.87 ms after (about 12% slower); overhead per added whitespace byte went from ~0.00000122 ms/byte to ~0.00000213 ms/byte (worse, not better)

I also tried a deeply-nested document (300 levels, indentation runs up to ~1200 spaces) to specifically stress long contiguous whitespace runs: pretty-printed accept() went from 0.74 ms before to 2.29 ms after (about 3x slower).

So on this machine/compiler I could not reproduce the claimed improvement for the whitespace-heavy case — it looks like a regression here, while compact (whitespace-free) input is marginally faster. Results were highly reproducible (<2% run-to-run variance) across -O2/-O3 and with/without -DNDEBUG. This could be compiler/platform-specific (e.g. how Apple Clang 21 inlines/schedules the new get_ignoring_pending_unget() split), so it may be worth re-checking against your usual benchmark setup — but wanted to report the numbers as measured rather than assume they'd match the PR description.

— posted by Claude Code on behalf of @nlohmann

[nlohmann]

A follow-up benchmark (Apple Clang 21 arm64 and Homebrew GCC 12, both -O2/-DNDEBUG, macOS) found that this PR, as originally written, was a regression, not a speedup, on pretty-printed / deeply-nested input:

case	Clang before (#5484 tip)	Clang PR (before this fix)	GCC before	GCC PR (before this fix)
300-level deep nesting, dump(4)	0.24 ms	0.79 ms (3.2x slower)	0.77 ms	1.34 ms (1.75x slower)
2000 objects, depth 3, dump(4)	1.20 ms	1.50 ms (25% slower)	1.51 ms	2.08 ms (38% slower)
isolated whitespace-skip cost (1M spaces)	0.65 ns/char	2.13 ns/char (3.3x slower)	2.08 ns/char	3.67 ns/char (1.8x slower)

Compact JSON (no meaningful whitespace runs) was unaffected either way (~3.7-4.0 ms, noise-level differences), matching what was already reported.

Root cause: it wasn't the get()/get_ignoring_pending_unget() split itself — that still inlines completely into scan() on both compilers. The problem was the shape of the loop that resulted: rewriting do { get(); } while (cond); into get(); while (cond) { get_ignoring_pending_unget(); } defeated both Clang's and GCC's ability to keep the input adapter's read/end pointers in registers across loop iterations. Instead, both compilers regenerated code that reloaded those pointers from memory on every single character, turning a register-only loop into a memory-bound one — worst exactly on long whitespace runs (deep indentation), which matches what was observed.

Fix (pushed as c2ce707a8): keep the same two-function structure, but restore a do-while shape (guarded by an if for the "first character isn't whitespace" case):

get();

if (current == ' ' || current == '\t' || current == '\n' || current == '\r')
{
    do
    {
        get_ignoring_pending_unget();
    }
    while (current == ' ' || current == '\t' || current == '\n' || current == '\r');
}

This is a pure control-flow reshaping with no behavior change. Verified after the fix (median of repeated runs, -O2/-DNDEBUG):

case	Clang fixed	GCC fixed
300-level deep nesting	0.24 ms (matches pre-#5490 baseline)	0.28 ms (2.7x faster than pre-#5490 baseline)
2000 objects, depth 3	1.20-1.23 ms (matches baseline)	0.99 ms (1.5x faster than baseline)
isolated whitespace-skip cost	0.66 ns/char (matches baseline)	0.78 ns/char (2.7x faster than baseline)

So with the fix applied, this PR is at worst neutral (Clang) and meaningfully faster (GCC) on whitespace-heavy input, with no regression anywhere I tested, and the required test suites (unit-class_lexer, unit-class_parser, unit-deserialization, unit-regression1, unit-regression2, unit-class_parser_diagnostic_positions) all pass unchanged.

Recommend keeping this PR open with the fix applied rather than closing it.

— posted by Claude Code on behalf of @nlohmann