Skip to content

Convert floats with Eisel-Lemire when std::from_chars is unavailable - #5617

Merged
nlohmann merged 2 commits into
json-view/01-float-chainfrom
json-view/02-eisel-lemire
Sep 30, 2026
Merged

nlohmann merged 2 commits into
json-view/01-float-chainfrom
json-view/02-eisel-lemire

Conversation

@nlohmann

@nlohmann nlohmann commented Sep 29, 2026 •

Copy link
Copy Markdown
Owner

Part of the stack for the zero-copy view (#5295). This change speeds up json::parse on its own; the view uses it later.

Summary

Some float tokens are too long for Clinger's fast path, e.g. the 17-digit coordinates of canada.json. Such tokens go to strtod unless std::from_chars is available, and it is not:

  • in C++11 and C++14;
  • with libc++, which has the float overloads but does not define __cpp_lib_to_chars, so macOS builds never used it.

These tokens are now converted with the Eisel-Lemire algorithm, adapted from fast_float's compute_float. It uses integer arithmetic only.

  • Up to 19 significant digits: the result is correctly rounded, and no fallback is needed (Mushtak and Lemire, 2023).
  • Longer tokens: the mantissa is truncated to 19 digits. The result is used when the truncated value w and w + 1 round to the same double; otherwise strtod decides, as before.
  • Overflow: still yields infinity, and with it out_of_range.406.

Changes

  • detail/input/number_parse.hpp: eisel_lemire() and parse_float_eisel_lemire(), for double; float and long double take the existing path.
  • detail/input/pow5_table.hpp: the 128-bit powers of five from fast_float, in a function-local static.
  • detail/bit_ops.hpp: count_leading_zeros(), a 64×64→128-bit multiplication, and read_eight_bytes(). Each has a portable fallback where there is no builtin or __int128, and none depends on the byte order.
  • README and license.md: credit for fast_float (MIT, Apache 2.0, or BSL; used under MIT).

Tests

New test case in unit-class_lexer.cpp (622,082 assertions):

  • every table entry is recomputed with big-integer arithmetic;
  • known values generated with Python, whose float() is correctly rounded;
  • 200,000 random doubles round-tripped through to_chars;
  • the 128-bit products and leading-zero counts are checked against references, with and without __int128.

Also checked outside the test suite: 6.5 million tokens compared with strtod, among them 60,000 exact halfway cases. There was no difference.

Benchmarks

tests/benchmarks: Apple M1 Max, Apple clang 21, -O3 -flto, C++11, median of 10 repetitions, before → after:

benchmark change
ParseString/canada −6.5%
ParseFile/canada −8.3%
ParseString/floats −8.1%
ParseFile/floats −2.9%
every other Parse* within ±2%

json::parse of canada.json in a separate process, best of 5: −8.6% (C++11), −7.6% (C++17).

Compile time (clang -O2): +0.7% for a translation unit that only includes json.hpp, +0.9% for one that also instantiates parse.

x86-64: not measured yet; the numbers will follow in a comment.

Public API

No change. Parsed values are the same, since both paths round correctly.


Written by Claude Code.

🤖 Generated with Claude Code

@nlohmann

Copy link
Copy Markdown
Owner Author

CI fixes for this PR (commit a4556e8):

  • ci_test_gcc, ci_test_standards_gcc (23): -Werror=unused-result for CHECK_THROWS_WITH_AS(json::parse("1.7976931348623159e308"), ...) in unit-class_lexer.cpp (GCC's warn_unused_result ignores doctest's cast to void). The result is now assigned to a dummy json _, as elsewhere in the tests.
  • ci_clang_tidy: abseil-string-find-str-contains on longer.find('.') == std::string::npos. The position is stored in a variable first.
  • clang (16.0.6) [Windows]: infra. The MinGW download from GitHub returned HTTP 500. Not caused by this PR.

Verified with Homebrew GCC 16 using the CI's GCC_CXXFLAGS (the test builds and passes). The clang-tidy change touches only the reported line.

The fix was merged upward through the whole stack (merge commits only: #5618 to #5635).

— posted by Claude Code on behalf of @nlohmann

@nlohmann nlohmann added the 🚀 ready to merge Ready to merge - just waiting for CI to complete. label Sep 30, 2026
@nlohmann nlohmann added this to the Release 3.13.0 milestone Sep 30, 2026
@nlohmann
nlohmann marked this pull request as ready for review September 30, 2026 05:22
@nlohmann
nlohmann removed this pull request from stack #5636 September 30, 2026 13:19
@nlohmann
nlohmann added this pull request to stack #5739 September 30, 2026 13:21
Float tokens that Clinger's fast path cannot convert (e.g. the 17-digit
coordinates of canada.json) went to strtod unless std::from_chars was
available. It is not used in C++11/14, and not with libc++, which does not
define __cpp_lib_to_chars. The Eisel-Lemire algorithm (after fast_float's
compute_float) now converts them with integer arithmetic, correctly rounded
for any token with at most 19 significant digits. Longer tokens are
truncated; the result is used if w and w + 1 round alike, else strtod
decides as before. Overflow still yields infinity (out_of_range.406).

The table of powers of five (fast_float's) lives in pow5_table.hpp; a unit
test recomputes every entry with big-integer arithmetic. Further tests:
known values generated with Python (whose float() is correctly rounded),
200,000 round trips through to_chars, and the 128-bit multiplication and
leading-zero count against big-integer references (both with and without a
128-bit type). Checked against strtod on 6.5 million tokens, among them
60,000 exact halfway cases: no difference.

json::parse on canada.json: -8.6% (C++11), -7.6% (C++17, Apple clang);
other files unchanged. Compile time of a TU including json.hpp: +0.7%.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
GCC (-Werror=unused-result) rejected CHECK_THROWS_WITH_AS(json::parse(...))
because parse() is [[nodiscard]]; assign the result to a dummy json as the
other tests do. clang-tidy flagged longer.find('.') == npos with
abseil-string-find-str-contains; store the position in a variable first.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
@nlohmann
nlohmann force-pushed the json-view/02-eisel-lemire branch from a4556e8 to 3ad6a88 Compare September 30, 2026 18:06
@nlohmann
nlohmann merged commit d268eaa into develop Sep 30, 2026
1 of 158 checks passed
@nlohmann
nlohmann deleted the json-view/02-eisel-lemire branch September 30, 2026 18:06
@nlohmann nlohmann mentioned this pull request Oct 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation L 🚀 ready to merge Ready to merge - just waiting for CI to complete. tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants