Write doubles with the shortest digits (Żmij), digits in registers by nlohmann · Pull Request #5634 · nlohmann/json

nlohmann · 2026-09-29T14:20:41Z

Part of the stack for the zero-copy view (#5295), but also a change of the core library: the output of dump() changes for about 0.1% of doubles.

Summary

Shortest round-trip double formatting. dump() now writes binary64 numbers with the conversion of Żmij by Victor Zverovich (MIT), ported to C++11 (detail/conversions/zmij.hpp). It finds the shortest decimal that reads back as the same double, and the closest one if there are several. Grisu2, used until now, is fast but not always shortest: it sometimes writes a 17th digit where 16 suffice, or a last digit that is not the closest.
- The layout is unchanged (1.5, 100.0, 1e+100, -0.0), but the digits can differ.
- Selection by a trait. dtoa_impl::is_binary64<T> (IEEE 754, 53 digits, max_exponent 1024, 8 bytes) selects Żmij. Every other floating-point type keeps Grisu2: float, and a double that is not binary64 (it still compiles). A long double that is binary64 (MSVC, Apple arm64) now uses Żmij as well, so its output changes the same way as double's.
- The powers of ten come from the table pow5_table.hpp that is already used for number parsing, and are computed from Żmij's compressed tables beyond it.
- The view writes floats through the library's to_chars(), so json_view::dump() changes the same way and stays identical to json::dump() (once the view exists, later in the stack).
Digits converted eight at a time. The BCD conversion of Xiang JunBo, as in Żmij, converts eight digits per step and stores them with one byte swap per eight digits; leading and trailing zeros are counted from those bytes. format_buffer()'s layouts are written with fixed-size moves instead of per-digit loops and buffer moves. to_chars() goes through a local buffer when the caller's is shorter than the 41 bytes this may write.
In-register digit writing. write_shortest() converts the 16 digits of a double in one vector register: SSE2 on x86-64, NEON on 64-bit little-endian Arm, both baseline, so there is no CPU check. Other targets convert eight digits at a time. The decimal point is inserted inside the register. Before, the digits were stored and read back right away to shift them by one byte, and that load waited for the stores to complete (store forwarding fails).
Writing straight into the output buffer. json::dump() writes floats and integers straight into its write buffer instead of copying them from the number_buffer member, and integers below 10^16 eight digits at a time.
Always-inline the digit kernels. read_eight_bytes() and parse_eight_digits() are always inlined. GCC called both as out-of-line functions in the lexer's number loops.
Docs and credits. dump.md, number_handling.md, and template_parameters.md describe the new output. The SPDX header of zmij.hpp/to_chars.hpp names Victor Zverovich; README and license name Żmij, Xiang JunBo (conversion of the digits to text), and Dougall Johnson (SIMD instruction sequence); zmij.hpp also credits Johnson for the compressed powers of ten. The new file is added to BUILD.bazel.

What changes in output

Of one million random doubles (random bits), 1,425 (0.14%) are written differently: 810 with fewer digits, e.g. 5.3165205877497296e+16 → 5.31652058774973e+16; 615 with the same digit count but the closest ones. None of one million short decimals (at most 8 significant digits, as typically written by people) changes. One existing test needed an update: 1.2345e+21 was written as 1.2344999999999999e+21, although the test's own comment lists 1.2345e21 as the shortest form.

Performance

Measured on Apple M1 Max (Apple clang, -O2) and an x86-64 KVM Haswell VPS pinned to one core (GCC 13 / Clang 18, -O2; the VPS is noisy, about ±5-10%). µs, best of 3 interleaved rounds of 15 runs; files from nativejson-benchmark. json::dump and json::parse, previous PR (#5738) → this PR:

	Apple M1	x86-64 GCC	x86-64 Clang
dump canada	6987 → 2405 (−66%)	16955 → 5124 (−70%)	18983 → 5384 (−72%)
dump citm_catalog	521 → 479 (−8%)	1352 → 1152 (−15%)	1144 → 1090 (−5%)
dump twitter	411 → 408 (−1%)	1007 → 923 (−8%)	875 → 897 (+3%)
parse citm_catalog	2974 → 2961 (0%)	7843 → 7646 (−3%)	6546 → 6722 (+3%)
parse canada	7529 → 7633 (+1%)	15553 → 14672 (−6%)	14950 → 15589 (+4%)

Documents with many floats gain the most (canada). Documents with few floats (twitter) are unchanged within the noise.

Tests

The 128-bit powers of ten, for every k from −342 to 341, recomputed with a small big-integer and compared.
Checked for 100,000 random doubles, all powers of two, all powers of ten and their neighbours, and boundary values (smallest subnormal, largest double, 2^53 + 1, ...): the output reads back as the same double, no decimal with one digit fewer does, and with C++17's std::to_chars the digits match (shortest, closest).
The choice of the conversion: is_binary64 for float, double, int, and long double; Żmij and Grisu2 differ on a known double and agree on a float; a long double with the format of a double dumps like the double.
unit-binary_formats.cpp size/ratio checks updated: the test JSON dumps slightly shorter now that some doubles get fewer digits.

Public API

No breaking changes to the interface. The output of dump() (and operator<<, to_string, json_view::dump) changes for about 0.1% of doubles, always to digits that read back as the same value. It changes the same way for a long double that is IEEE 754 binary64 (MSVC, Apple arm64). float and other types are unchanged. read_eight_bytes()/parse_eight_digits() are in detail and only change their inlining. The serializer (in detail) no longer formats floats and integers through its number_buffer member. Release note:

Doubles are now written with the shortest digits that read back as the same value (Żmij); about 0.1% of doubles are written differently than before, most with fewer digits.

Written by Claude Code.

🤖 Generated with Claude Code

nlohmann · 2026-09-29T22:38:18Z

CI fixes for this PR (commit fb6f874, plus the lower fixes merged in). CI had not run yet; these were found locally with gcc:16 (linux/amd64) and the CI flags:

ci_test_gcc (every test, since it is in json.hpp): -Werror=useless-cast on static_cast<std::size_t>(d.significand % 100) in the new digit writer in detail/conversions/to_chars.hpp. It is now cast through a named std::uint64_t variable.
ci_test_gcc: -Werror=format-truncation for snprintf(..., "%.*e", digits.size() - 2, v) in unit-to_chars.cpp (GCC cannot bound the precision). The neighboring decimal is now written with an std::ostringstream (classic locale, std::scientific, std::setprecision), which gives the same text.
single_include/nlohmann/json.hpp was regenerated.

Verified: unit-to_chars, unit-serialization and unit-json_view build and pass (C++11/C++20 with the CI flags, and a JSON_NOEXCEPTION build).

— posted by Claude Code on behalf of @nlohmann

github-actions · 2026-09-30T22:04:23Z

🔴 Amalgamation check failed! 🔴

The source code has not been amalgamated and/or formatted correctly, or BUILD.bazel is out of date.

@A

…5729) * Share scalar serialization between dump_internal and dump_value dump_value()'s cases for string, binary, boolean, number_integer, number_unsigned, number_float, discarded and null were a byte-for-byte copy of dump_internal()'s (added together in #5285 for the iterative fallback path). Any future change to scalar output had to be made in both places, or the recursive and depth-limited paths would silently start producing different bytes. Extract the shared cases into a private dump_scalar() and have both dump_internal() and dump_value() call it. Output is unchanged: dump(), dump(4), dump(-1,' ',true) and the replace/ignore error_handler_t variants are byte-identical over the json_test_data corpus before and after, and dump() throughput on a scalar-heavy document is unaffected. Part of #5709 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Drop serializer.hpp's dependency on binary_writer.hpp The only use of binary_writer in serializer.hpp was binary_writer<BasicJsonType, char>::to_char_type() to write the U+FFFD replacement character's three bytes. With CharType=char this is an identity conversion, so the include of binary_writer.hpp (and transitively binary_reader.hpp) pulled in a large, unrelated header for a no-op call. Write the three bytes directly instead. serializer.hpp compiles standalone with -Wall -Wextra -Werror, with and without -funsigned-char, and unit-serialization's error_handler_t::replace cases (with and without ensure_ascii) still pass. Moving binary_writer's to_char_type/to_msgpack_length to its private section is left as an optional follow-up. Part of #5709 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix stale #include lines in the output headers output_adapters.hpp included <algorithm> and <iterator> for std::copy and std::back_inserter, which have not been used there since #3569 (2022). serializer.hpp included <algorithm> for std::reverse (also unused), <cmath> for labs/isnan/signbit (only std::isfinite is used) and <utility> for std::move (nothing from <utility> is used there), while using std::next without including <iterator> at all, relying on getting it transitively through output_adapters.hpp's own stale <iterator>. Drop the unused includes, add <iterator> for std::next, and correct the remaining include comments. Both headers still compile standalone with -Wall -Wextra -Werror. Part of #5709 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Remove JSON_HEDLEY_NON_NULL(2) from write_characters() overrides output_vector_adapter, output_stream_adapter and output_string_adapter declared their write_characters(const CharType*, std::size_t) override JSON_HEDLEY_NON_NULL(2), but binary_writer legitimately calls it with a null pointer and length 0 for an empty string or binary value; the type-erased call path only stayed silent under UBSan because the static callee at those call sites is the unattributed virtual base. A nonnull attribute on a definition lets GCC and Clang assume the parameter is non-null inside the function body even when the call is virtual, so this was latent undefined behavior, not just style. Drop the attribute from the three overrides and document the (nullptr, 0) contract on output_adapter_protocol::write_characters. unit-cbor, unit-msgpack, unit-bson and unit-bon8 (which all exercise empty binary/string payloads through the stream and vector/string adapters) pass under -fsanitize=address,undefined,nonnull-attribute. Part of #5709 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Update stale serializer doc comments to match the current implementation dump_internal()'s doc block still described the pre-#5285/#5449 implementation: an escape_string() function that does not exist (the function is dump_escaped), integer conversion "implicitly via operator<<" (dump_integer actually uses a digit-pair lookup table), and floating-point conversion via "%g" (IEEE-754 types go through to_chars, others through snprintf). dump_value()'s comment said elements are pushed for dump_internal to walk, but it is dump_iteratively() that walks the stack. dump_escaped(), dump_integer() and dump_float() each said they write "to output stream @A o", which has not been true since the writer moved to write_buffer. Doc-only change; no behavior, API or ABI impact. Part of #5709 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Merge duplicate byte-to-hex helper and drop stale '| 0' promotions serializer::hex_bytes() and binary_writer::hex_byte() had identical bodies. Keep one, detail::hex_byte() in string_utils.hpp, and use it from both. Also drop the `| 0` at the two serializer call sites (hex_bytes(byte | 0) and hex_bytes(s.back() | 0)): #3088 (7440786) added it so that `ss << std::hex << (byte | 0)` printed a number rather than a char with the old stringstream writer; the int result just narrows back to uint8_t now, so it was a no-op. Behavior is unchanged: unit-serialization, unit-bon8 (whose type_error.316 messages exercise this code) and unit-diagnostics pass, and both headers still compile standalone with -Werror. Part of #5709 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Trim two stale lint suppressions in serializer.hpp dump_integer()'s `auto buffer_ptr = number_buffer.begin();` carried NOLINT entries for cppcoreguidelines-pro-type-vararg and hicpp-vararg, left over from the snprintf-based implementation (#3088); there is no variadic call on that line, so keep only the qualified-auto suppressions it actually needs. remove_sign()'s assert checked `x < 0 && x < (std::numeric_limits<number_integer_t>::max)()) `with a NOLINT(misc-redundant-expression) to hide it; the second conjunct is always true once x < 0, and has been since 6ce2f35 (2019), so reduce the assert to `x < 0` and drop the suppression instead of masking it. Both are documentation-only changes to assertions/suppressions, not behavior. The to_chars.hpp `#if 0` branch this item also flagged is left alone, next to draft PR #5634's pending hunk. Part of #5709 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Move the Hoehrmann SPDX copyright line to string_utils.hpp serializer.hpp carried the SPDX-FileCopyrightText line for Björn Hoehrmann's UTF-8 decoder, but the decoder (decode() and the utf8d table) has lived in string_utils.hpp since #5185 (d19f7f5); serializer.hpp now only calls decode(). Move the copyright line to where the code it covers actually is. Part of #5709 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Stop calling std::localeconv() on every dump() The serializer constructor snapshotted std::localeconv() into a locale_chars member on every dump(), even though the only reader is dump_float(number_float_t, std::false_type)'s snprintf path, taken only for a number_float_t that is neither IEEE single nor double. localeconv() is not required to be thread-safe with setlocale(), so every dump() paid for and raced on a lookup that almost never mattered. Remove locale_chars and the locale member. Right before the thousands-separator/decimal-point fixups in the snprintf path, read std::localeconv() into local thousands_sep/decimal_point variables (null-checked, first byte only, as before) - the same way lexer::get_decimal_point() already does since #5597. Output is unchanged unless the locale changes during a single dump(); in that case the fixups now match what snprintf just produced, instead of a value snapshotted before the call. Overlaps draft PR #5608, which touches the same constructor and dump_float() lines to move this code into a new dump_float_snprintf(); this lands the lookup change now as #5709 asks, and #5608 can do the lookup inside dump_float_snprintf() when it rebases. Verification: the full json_test_data corpus (742 files, dump(), dump(4) and dump(-1,' ',true)) is byte-identical to before the change under the C locale. Added a test pinning the new per-conversion lookup: it switches LC_NUMERIC mid-dump() (via a streambuf that switches on its first write, after the serializer's write buffer has been flushed once but before a later float is converted) and checks the decimal point is still normalized using the locale active at conversion time. On a platform where long double is IEEE-754 double (e.g. 64-bit Arm), dump_float() takes the locale-independent to_chars() path and the test is a no-op there; it is meaningful on a platform where long double is extended precision (most x86 targets). Part of #5709 item 3 Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>

gregmarr

Did not review the ported zmij file.

Write doubles with the conversion of Zmij by Victor Zverovich (MIT), ported to C++11 (detail/conversions/zmij.hpp). It finds the shortest decimal that reads back as the same double, and the closest one if there are several. Grisu2, used until now, is fast but not always shortest: it sometimes writes a 17th digit where 16 suffice, or a last digit that is not the closest. The layout is unchanged (1.5, 100.0, 1e+100, -0.0); float keeps Grisu2. Digits are converted eight at a time with the BCD conversion of Xiang JunBo, as in Zmij, and written with one byte swap per eight digits and fixed-size moves instead of per-digit loops. Leading and trailing zeros are counted from those bytes. to_chars() uses a local buffer when the caller's is shorter than the 41 bytes this may write. The powers of ten come from the number-parsing table, adjusted where it holds values rounded up, and extended with Zmij's compressed tables beyond 10^308. write_shortest() converts its 16 digits in one vector register (SSE2 on x86-64, NEON on 64-bit Arm, both baseline) and inserts the decimal point inside the register, avoiding a store-forwarding stall that cost about 25% of the time to write a double. dump() writes floats and integers straight into the serializer's write buffer instead of copying them from a member buffer, and small integers eight digits at a time. read_eight_bytes() and parse_eight_digits() are marked always-inline, which GCC had been calling out of line in the number-parsing loops. Of one million random doubles, about 0.14% are now written with different digits, always to a value that still reads back as the same double. Signed-off-by: Niels Lohmann <mail@nlohmann.me>

- bit_ops.hpp: include macro_scope.hpp (JSON_HEDLEY_ALWAYS_INLINE); fixes IWYU - to_chars.hpp: C4100 for the unused parameter in release builds, clang-tidy sign comparison, cpplint runtime/int, GCC -Wstrict-overflow (unsigned abs) - unit-to_chars.cpp: parse with the library instead of strtod (MinGW's strtod rounds some 16/17 digit inputs wrongly); no floating-point std::to_chars with icpc Signed-off-by: Niels Lohmann <mail@nlohmann.me>

The non-template overloads for double asserted binary64 doubles wherever json.hpp was included, so the library no longer compiled where double is not IEEE 754 binary64 (AVR, -fshort-double). A trait now picks Zmij for any binary64 type, including a long double of that format (MSVC, Apple Arm), and Grisu2 for the others. Remove the unused write_short_decimal, powers_of_ten_16, zmij::decimal, zmij::to_decimal, and shortest_digits(double). Signed-off-by: Niels Lohmann <mail@nlohmann.me>

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

nlohmann added this pull request to stack #5636 September 29, 2026 14:20

nlohmann changed the title ~~Write doubles with the shortest digits (Zmij)~~ Write doubles with the shortest digits (Żmij) Sep 29, 2026

github-actions Bot added documentation aspect: binary formats BSON, CBOR, MessagePack, UBJSON L tests labels Sep 29, 2026

nlohmann removed this pull request from stack #5636 September 30, 2026 13:19

nlohmann force-pushed the json-view/23-zmij branch from 7f34f5f to 3336423 Compare September 30, 2026 13:20

nlohmann added this pull request to stack #5739 September 30, 2026 13:21

nlohmann force-pushed the json-view/23-zmij branch 3 times, most recently from 5242356 to 165bdff Compare September 30, 2026 18:06

nlohmann added the review needed It would be great if someone could review the proposed changes. label Sep 30, 2026

nlohmann marked this pull request as ready for review September 30, 2026 18:16

nlohmann force-pushed the json-view/23-zmij branch from 165bdff to 0a97d94 Compare September 30, 2026 19:19

github-advanced-security Bot found potential problems Oct 2, 2026

View reviewed changes

nlohmann removed this pull request from stack #5739 October 6, 2026 09:26

nlohmann changed the base branch from json-view/22-view-dump-fast to json-view/02b-float-parser October 6, 2026 09:26

nlohmann mentioned this pull request Oct 6, 2026

Speed up json_view on x86-64 Linux and make the comparison fair #5765

Closed

nlohmann force-pushed the json-view/23-zmij branch from 173fbc0 to 1784269 Compare October 6, 2026 09:28

nlohmann changed the title ~~Write doubles with the shortest digits (Żmij)~~ Write doubles with the shortest digits (Żmij), digits in registers Oct 6, 2026

nlohmann added this pull request to stack #5768 October 6, 2026 09:29

github-advanced-security Bot found potential problems Oct 6, 2026

View reviewed changes

nlohmann mentioned this pull request Oct 7, 2026

Use MSVC intrinsics for the 64×64-bit multiplication in full_multiplication() #5780

Closed

nlohmann force-pushed the json-view/23-zmij branch from 1784269 to bfa4886 Compare October 7, 2026 14:44

This comment was marked as resolved.

Sign in to view

github-advanced-security Bot found potential problems Oct 10, 2026

View reviewed changes

Comment thread include/nlohmann/detail/conversions/to_chars.hpp Dismissed

Comment thread single_include/nlohmann/json.hpp Dismissed

nlohmann force-pushed the json-view/23-zmij branch 2 times, most recently from 2ad81e9 to 05e11ce Compare October 11, 2026 09:33

github-actions Bot added the CMake label Oct 11, 2026

nlohmann force-pushed the json-view/23-zmij branch from 05e11ce to e59da8f Compare October 13, 2026 15:43

gregmarr approved these changes Oct 13, 2026

View reviewed changes

nlohmann removed the review needed It would be great if someone could review the proposed changes. label Oct 13, 2026

nlohmann added this to the Release 3.13.0 milestone Oct 13, 2026

nlohmann force-pushed the json-view/23-zmij branch from e59da8f to 70cb33b Compare October 13, 2026 17:51

nlohmann added 6 commits October 13, 2026 22:22

Credit Zmij's authors in to_chars.hpp, the README, and the license page

a403eb4

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

Silence MSVC C4127 for a platform-constant check in the to_chars test

ecd9ae2

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

Regenerate the amalgamated headers

b177f48

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

nlohmann force-pushed the json-view/23-zmij branch from 70cb33b to b177f48 Compare October 13, 2026 20:22

nlohmann merged commit 91a7a94 into develop Oct 14, 2026
169 checks passed

nlohmann deleted the json-view/23-zmij branch October 14, 2026 15:38

nlohmann mentioned this pull request Oct 15, 2026

Fix AppVeyor failures from the Zmij double writer #5843

Merged

Conversation

nlohmann commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What changes in output

Performance

Tests

Public API

Uh oh!

nlohmann commented Sep 29, 2026

Uh oh!

github-actions Bot commented Sep 30, 2026

🔴 Amalgamation check failed! 🔴

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

Uh oh!

gregmarr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nlohmann commented Sep 29, 2026 •

edited

Loading