[nlohmann]

Summary

Parsing numbers from contiguous input scanned every digit twice. scan_number_bulk_contiguous() validated the token one byte at a time and copied it into token_buffer. The conversion then went through two out-of-line functions that read the digits again:

• parse_float_native() called parse_float_significand(), which returned its result through memory and called std::any_of twice per number (GCC does not inline it).
• parse_integer_signed() divided by ten on every digit for its overflow check.

This PR accumulates the value while the digits are validated:

• Digits are taken eight at a time where at least eight bytes remain: is_eight_digits() tests them and parse_eight_digits() converts them, as in fast_float.
• Floats with at most 19 digits go straight to decimal_to_float() (Clinger's fast path or Eisel–Lemire), so the float fast path has no calls left.
• Integers with at most 18 digits are stored without an overflow check. A narrower number_integer_t/number_unsigned_t still rejects values that do not round-trip, as before.
• Longer tokens, and float formats other than binary32/binary64 (e.g. x87 long double), take the existing path. It is unchanged, except that has_nonzero_digit() is now a plain loop and parse_float_significand() is always inlined.

The stream windows of #5828 are handled like the existing fast path: a number that reaches the end of a window falls back to scan_number(), and the terminator is read and ungotten.

This is the first of five stacked PRs from an assembly review of the number and string parsing; the stack is based on #5828.

Performance

Change of the best-of-runs time against the previous PR (the first: against #5828), negative is faster. accept is json::accept() (no allocations), parse is json::parse(). Apple M1 Max: Apple Clang 21 and Homebrew GCC 16; x86-64 (Haswell VM): GCC 13, and Clang 18 with -falign-loops=64 -falign-functions=64 on both sides (without it, code placement alone moves Clang's numbers by up to ±5 % between builds). Improvements of 3 % or more are bold.

file	M1 Clang accept	M1 Clang parse	M1 GCC 16 accept	M1 GCC 16 parse	x86 GCC accept	x86 Clang accept
canada.json	−18.8 %	−9.8 %	−17.2 %	−9.3 %	−11.5 %	−20.7 %
citm_catalog.json	−6.8 %	−3.0 %	−8.3 %	−1.9 %	−1.3 %	+3.1 %
twitter.json	−0.3 %	−0.3 %	−1.0 %	+0.1 %	+4.7 %	+0.8 %
numbers.json	−25.1 %	−22.7 %	−20.4 %	−20.1 %	−18.4 %	−24.7 %
long_strings.json	−0.2 %	+1.6 %	−1.9 %	−1.6 %	−1.4 %	+0.7 %
twitterescaped.json	−0.7 %	+0.1 %	−0.9 %	−0.3 %	+6.4 %	+1.0 %
mesh.json	−18.6 %	−13.5 %	−14.9 %	−12.4 %	−13.7 %	−16.1 %

This PR does not touch the string code; the x86-64 GCC rows for twitter.json and twitterescaped.json (+5 to +6 %) come from code placement and from the shared machine, which was under load during this run (the M1 rows for these files are within ±1 %).

Tests

• New section "digits accumulated while scanning" in unit-class_lexer.cpp:
  • covers every digit count from 1 to 25, in the integer and the fraction part, with and without sign and exponent, with leading zeros, and with exponents beyond every range;
  • compares each case with the byte path. The reference input is a std::list<char>: since Make parsing from std::istream and FILE* fast #5828, a std::stringstream takes the bulk path as well, so it is no longer an independent reference.
• New section for narrower number types: a basic_json with 32-bit integers checks the round-trip check.
• Mutation check: raising either digit limit by one makes these tests fail.
• Differential check against Make parsing from std::istream and FILE* fast #5828's head: identical DOM output, error messages, accept() results and SAX events (including the float text), for 737 files of json_test_data and 500k generated inputs. Each input is parsed from a string, an std::istream and a FILE*. Run on Apple M1 (Clang) and x86-64 (GCC 13, Clang 18).
• Full test suite: passes under ASan/UBSan on macOS (Clang) and Linux x86-64 (GCC).

Breaking changes

None. Only internal functions in nlohmann::detail change. Values, exceptions and error positions are the same, as the differential check above shows.

🤖 Generated with Claude Code

[gregmarr]

Is this repeated code that could use a helper?

[nlohmann]

Yes, though not quite this loop: the integer part and the fraction in parse_float_significand() ran the same eight-digits-then-one-digit loop. In 2dfda27 they share accumulate_significant_digits(), and clang -O2 emits the same code as before. accumulate_digits() stays separate because it checks each byte for a digit and has no 19-digit cap, so a shared helper would need both as parameters.

— posted by Claude Code on behalf of @nlohmann