Skip to content

Write the floats of json_view from their digits - #5635

Closed
nlohmann wants to merge 6 commits into
json-view/23-zmijfrom
json-view/24-view-token-digits
Closed

nlohmann wants to merge 6 commits into
json-view/23-zmijfrom
json-view/24-view-token-digits

Conversation

@nlohmann

@nlohmann nlohmann commented Sep 29, 2026 •

Copy link
Copy Markdown
Owner

Part of the stack for the zero-copy view (#5295). Depends on the previous PR (shortest doubles with Żmij).

Summary

json_view::dump() writes a float token of at most 15 significant digits from its digits, without converting it to a double and back. The output is unchanged: it is what json::dump() writes for the same value.

  • Why this is exact. Two decimals of at most 15 significant digits are farther apart than the rounding interval of a normal double; this is the argument behind DBL_DIG. So the token's digits, without trailing zeros, are the shortest digits of its double. The library now writes exactly those (Żmij); with Grisu2 this would not hold.
  • When it applies. The exponent must keep the value away from subnormals and overflow (leading digit between 10^−290 and 10^304).
  • Other tokens (16 or more digits, or edited values) are converted from the digits already read, with the library's decimal_to_float() (Speed up the lexer: own float parser, string scan, and \u table #5738), so the token is not read twice.
  • Doubles are written into the output directly, not through a local buffer.
  • With NEON, the fixed layouts (12.5, 0.001, 100.0) are put together in vector registers by a table lookup of the digit bytes, as in the prototype. The library's portable layout copies the digits through a buffer at another offset, and a load that spans several recent stores waits until they reach the cache. x86 and JSON_VIEW_NO_SIMD use the library's layout.

Benchmarks

json_view::dump(), before vs. after this PR, on Apple M1 (best of 7 alternating processes):

file before after ratio
numbers 450 µs 140 µs 0.31
marine_ik 5.19 ms 1.96 ms 0.38
mesh.pretty 1.66 ms 1.09 ms 0.66
canada (mostly 16–17 digits) 5.32 ms 4.57 ms 0.86
citm_catalog 144 µs 145 µs 1.00
twitter 73.2 µs 73.2 µs 1.00

canada's tokens mostly have 16 or 17 digits, so they are converted; its gain comes from writing in place and from the NEON layout.

Tests

  • 20,000 float tokens with 1 to 17 significant digits, in every spelling: with and without a point, exponents with e/E and signs, leading zeros (0.000123), trailing zeros, and negative values, from about 1e−320 to 1e300 (subnormals included). json_view::dump() must equal json::dump() of the same text.
  • On AArch64, the same tests check the NEON layout; on x86 and with JSON_VIEW_NO_SIMD, the library's layout.

Public API

No change.


Written by Claude Code.

🤖 Generated with Claude Code

@nlohmann

Copy link
Copy Markdown
Owner Author

CI fix for this PR (commit 977bb1b, plus all lower fixes merged in). CI had not run yet; this was found locally with gcc:16 (linux/amd64) and the CI flags:

  • ci_test_gcc: -Werror=useless-cast on static_cast<std::size_t>(1 + (tokens() % 17)) and static_cast<std::size_t>(tokens() % (digits.size() + 1)) in the float-token test of unit-json_view.cpp. std::mt19937_64 yields std::uint_fast64_t, which is std::size_t on Linux. The numbers are now drawn through a lambda that casts a named std::uint64_t. That lambda also makes the std::string(count, '0') conversions explicit where std::size_t is 32 bits (MSVC Win32). The sequence of draws is unchanged.

Verified: unit-json_view builds and passes (C++11/C++20 with the CI flags, and a JSON_NOEXCEPTION build run as --no-throw). All view tests also pass a silkeh/clang:4 syntax check on this branch.

— posted by Claude Code on behalf of @nlohmann

@nlohmann
nlohmann removed this pull request from stack #5636 September 30, 2026 13:19
@nlohmann
nlohmann force-pushed the json-view/24-view-token-digits branch from f4a72e4 to ba55e27 Compare September 30, 2026 13:20
@nlohmann
nlohmann added this pull request to stack #5739 September 30, 2026 13:21
@nlohmann
nlohmann force-pushed the json-view/24-view-token-digits branch from ba55e27 to 8ce7529 Compare September 30, 2026 15:34
@nlohmann
nlohmann force-pushed the json-view/24-view-token-digits branch 2 times, most recently from 85e1a37 to badc20d Compare September 30, 2026 18:06
@nlohmann nlohmann added the review needed It would be great if someone could review the proposed changes. label Sep 30, 2026
@nlohmann
nlohmann marked this pull request as ready for review September 30, 2026 18:16
dump() writes a float token of at most 15 significant digits from its
digits, without converting it to a double and back: two decimals of at
most 15 digits are farther apart than the rounding interval of a
normal double (the argument behind DBL_DIG), so the token's digits are
the shortest ones of its double, which the library's conversion (Zmij)
writes. The exponent must keep the value away from subnormals and
overflow. Longer tokens are converted from the digits already read.

Doubles are written into the output directly instead of through a
local buffer. With NEON, the fixed layouts ("12.5", "0.001", "100.0")
are put together in vector registers by a table lookup of the digit
bytes: the portable layout copies the digits through a buffer at
another offset, and a load that spans several recent stores waits
until they reach the cache.

dump() of float-heavy documents: numbers -69%, marine_ik -62%,
mesh.pretty -34%, canada (mostly 16 or 17 digits) -14%.

Tests: 20,000 float tokens of 1 to 17 significant digits in every
spelling (point, exponent, leading and trailing zeros, sign), from about
1e-320 to 1e300, written as json::dump() writes them. On AArch64 they
check the NEON layout; x86 and JSON_VIEW_NO_SIMD use the library's.
Other float types, now the only ones on the general path, are tested
with non-finite values set by edits (written as null).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
GCC -Werror=useless-cast on Linux x86-64 rejected
static_cast<std::size_t>(tokens() % n): std::mt19937_64 yields
std::uint_fast64_t, which is std::size_t there. Draw the numbers through
a lambda that casts a named std::uint64_t, which also makes the
conversions for std::string's count explicit where std::size_t is
32 bits wide. The sequence of draws is unchanged.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
@github-actions

Copy link
Copy Markdown

🔴 Amalgamation check failed! 🔴

The source code has not been amalgamated and/or formatted correctly, or BUILD.bazel is out of date.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Comment thread include/nlohmann/detail/view/serializer.hpp Dismissed
Comment thread single_include/nlohmann/json_view.hpp Dismissed
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
@nlohmann
nlohmann removed this pull request from stack #5739 October 6, 2026 09:26
@nlohmann

nlohmann commented Oct 6, 2026

Copy link
Copy Markdown
Owner Author

The zero-copy view stack (#5295) was rebased onto develop and regrouped from 20 PRs into 10, one squashed commit each. The changes of this PR are now in #5633; its review comments stay here for reference.


Written by Claude Code.

@nlohmann nlohmann closed this Oct 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

L review needed It would be great if someone could review the proposed changes. tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Idea: borrowed / zero-copy read-only view over a source buffer

2 participants