Convert floats independently of the C locale's decimal point

[nlohmann]

Summary

Under a locale whose decimal point is longer than one byte (U+066B in fa_IR.UTF-8, ar_EG.UTF-8, ...), every float that reached the strtod fallback was silently truncated at the decimal point, and out-of-range values such as 1.5e400 became 1.0 instead of throwing. With libc++ (all standards) and in C++11/14, that fallback handled most floats, because libc++ does not define __cpp_lib_to_chars. This PR implements the four steps proposed in #5660 (comment), so that the locale-dependent strtod is only the last resort, and fixes that last resort for multi-byte decimal points.

Changes

The lexer now converts a float token in this order (lexer::convert_number()):

1. std::from_chars: as before if __cpp_lib_to_chars is defined (float, double, long double), and now also for float/double with libc++ 20 or later in C++17 or later (_LIBCPP_VERSION >= 200000), but only where the C library has no strtod_l (see step 3). libc++'s implementation is 1.3x to 2.8x slower per number than Apple's strtod_l, and it would be tried before Clinger's fast path (see "Performance" below). On Apple platforms, floating-point from_chars also lives in the system dylib and is marked available from macOS/iOS 26, so the check also requires _LIBCPP_AVAILABILITY_HAS_FROM_CHARS_FLOATING_POINT. long double is not supported by libc++ and skips this step.
2. Out of range without strtod: if from_chars reports result_out_of_range, the value is derived from the token (parse_float_out_of_range()): the sign from a leading -, the direction from the decimal exponent of the first nonzero digit. ±infinity then throws out_of_range.406 in the parser as before, and an underflow gives ±0. The reason is in a comment: implementations disagree on the stored value (P4168). One deviation from the proposal: a value that may be subnormal is left to the next step, because libstdc++'s from_chars before GCC 13 reports some subnormal results as out of range (it uses strtod and its ERANGE: for all types in GCC 11, for long double in GCC 12; checked in the libstdc++ sources). ±0 is only returned when the token is below half the smallest subnormal number regardless of its digits (a conservative bound computed from numeric_limits).
3. Clinger's fast path (double only), unchanged.
4. strtof_l/strtod_l/strtold_l with a "C" locale (parse_float_c_locale()), revived from Switch number parsing to std::from_chars / extended locale strto*_l if available #5237. The locale is created once in a function-local static (newlocale(LC_NUMERIC_MASK, "C", nullptr), or _create_locale(LC_NUMERIC, "C") with _strto*_l on MSVC) and never freed. It is used only on an allowlist: MSVC/UCRT (_MSC_VER >= 1900, not MinGW), Apple (<xlocale.h>, included after <cstdlib> because it only declares the strto*_l functions if <stdlib.h> came first), and glibc (__GLIBC__ and __USE_GNU, not uClibc). g++ and clang++ define _GNU_SOURCE for C++ on glibc, so users do not need to define anything. If _GNU_SOURCE is not in effect, the code falls back cleanly.
5. Last resort (parse_float_locale_aware(), previously lexer::convert_float_locale_aware()): strtod with the decimal point of the current locale, still looked up at conversion time with the retry from Look up the locale decimal point at conversion time, not lexer construction #5597. A decimal point longer than one byte is now put into a copy of the token, as in the patch from the issue.

Other changes:

• The conversion helpers (strtof overloads, decimal point lookup, last-resort conversion) moved from lexer.hpp to number_parse.hpp, next to the other conversion helpers, so the last resort can be tested directly on platforms that no longer use it.
• New internal macros JSON_HAS_FLOAT_FROM_CHARS, JSON_HAS_LONG_DOUBLE_FROM_CHARS, and JSON_HAS_C_LOCALE_STRTOD, which are undefined at the end of json.hpp unless JSON_TEST_KEEP_MACROS is defined.
• Docs: features/types/number_handling.md describes how floats are converted and that underflows become ±0; api/basic_json/parse.md has a 3.13.0 version-history entry.

Which platform uses which path after this change:

platform	float / double	long double
libstdc++ ≥ 11, MSVC STL (C++17+)	from_chars	from_chars
libc++ ≥ 20, C++17+, C library without strtod_l (e.g. Android, FreeBSD)	from_chars	last resort
Apple (all standards and deployment targets)	Clinger (double) → strtod_l	strtold_l
glibc, C++11/14 or libstdc++ < 11 / any libc++	Clinger (double) → strtod_l	strtold_l
MSVC, C++11/14	Clinger (double) → _strtod_l	_strtold_l
MinGW, musl, Android/bionic, Cygwin, FreeBSD, uClibc, others (without from_chars)	Clinger (double) → last resort (now multi-byte safe)	last resort

Overlap with draft #5617 (Eisel-Lemire for double when from_chars is unavailable): both PRs touch number_parse.hpp. The conflicts should be textual only, and this PR does not depend on it. After this PR, #5617 would only speed up the double path and avoid strtod_l/strtod for it on platforms without from_chars. It is no longer needed for correctness under exotic locales.

Tests

• unit-locale-cpp.cpp, "locale with a multi-byte decimal point": now checks values under the first usable locale: 3.141592653589793238462643383279 → 3.141592653589793, 1.7976931348623157e308 → DBL_MAX, -2.5e-320, 1.5e400 → out_of_range.406 (exact message; the exception has no JSON context, so diagnostic positions do not change it), 1.5e-400 → 0.0, and 1.5 with float and long double as number_float_t. It is still skipped with a message if no such locale is installed.
• unit-locale-cpp.cpp, new "conversion with the decimal point of the current locale": calls parse_float_locale_aware() directly under C, de_DE, and the multi-byte locale (all three float types, token without a dot, an invalid token that stops early, token restored to .).
• unit-class_lexer.cpp: parse_float_out_of_range() directly (overflow, underflow, saturated exponents, possibly subnormal → declines, float/long double), and locale-free parsing of out-of-range values for double/float/long double number types, positive and negative, incl. signed zeros and values next to the smallest subnormal.
• Fails before, passes after: the new locale checks fail on develop (633de8e) under fa_IR.UTF-8 on macOS: 7 failed assertions with Apple clang (C++11/17/20) and GCC 16 C++11, 2 with GCC 16 C++17 (1.5e400, 1.5e-400). The locale-free out-of-range tests already pass on develop; they guard the new token-derived results.
• Configurations run (unit-locale-cpp, unit-class_lexer): Apple clang 21 with C++11/17/20 and ASan/UBSan; GCC 16 (Homebrew, macOS SDK) with C++11/17. Also unit-testsuites, unit-class_parser, unit-regression1, and unit-deserialization with Apple clang C++11/17. Compile checks: -mmacosx-version-min=10.15 and 11.0 with C++17/20. After 42e3489, unit-locale-cpp and unit-class_lexer pass again with Apple clang C++11/17/20 (ASan/UBSan) and GCC 16 C++17; the Apple clang C++17 binaries no longer reference from_chars. No warnings with -Weverything (CI's clang flags, Apple clang and clang 22 on Linux) and with CI's gcc_flags.cmake (GCC 16). In Docker/glibc: GCC 16 C++11/17, GCC 4.9 C++11, and clang 4 C++11/14 run both tests. clang 22 with -stdlib=libc++ compiles; that image's libc++ is 14, so it correctly does not use from_chars.
• Issue example (extended), LC_NUMERIC=fa_IR.UTF-8, macOS 27:

input	develop: Apple clang C++11/17/20, GCC 16 C++11	develop: GCC 16 C++17	this PR: all five configurations
3.141592653589793238462643383279	3.0	3.141592653589793	3.141592653589793
1.7976931348623157e308	1.0	1.7976931348623157e+308	1.7976931348623157e+308
-2.5e-320	-2.0	-2.5e-320	-2.5e-320
1.5e400	1.0	1.0	out_of_range.406
1.5e-400	1.0	1.0	0.0
1.5 (float)	1.0	1.5	1.5
1.5 (long double)	1.0	1.5	1.5
1.5 (double, Clinger)	1.5	1.5	1.5

Performance

Parse time of this PR relative to develop (1.00 = unchanged, lower is faster; median of 21 parses, best of 2–3 rounds, -O2, arm64; glibc natively in Docker). Parsed values are bit-identical to develop in every configuration (checksum over all floats).

input	Apple clang C++11	Apple clang C++17	GCC 16 macOS C++11	GCC 16 macOS C++17	glibc GCC 14 C++11	glibc GCC 14 C++17
canada.json	0.94	0.91	0.95	0.99	0.97	1.02
mesh.json	1.01	1.01	0.99	1.01	1.00	1.01
100k random doubles, %.17g	0.94	0.95	0.91	1.01	0.85	1.00
100k doubles in [0,1), %.17g	0.92	0.91	0.91	1.00	0.81	1.02
100k short doubles (123.45, Clinger)	1.02	1.01	0.99	1.00	0.99	1.00
100k floats (float type)	0.90	0.90	0.90	1.00	0.60	1.01
canada.json (long double type)	0.94	0.90	0.93	1.01	0.79	0.97
integers, poet.json (controls)	≈1.00	≈1.00	≈1.00	≈1.00	≈1.00	≈1.00

• C++11/14: faster, because strtod_l needs neither localeconv() nor a copy of the token per number (strtod_l itself is as fast as strtod: 203 vs 206 ns per random double on glibc).
• libstdc++ in C++17: unchanged (from_chars before and after).
• Apple clang in C++17: the first version of this PR used libc++'s from_chars there and was slower (random doubles 1.75x, short numbers 1.3x, mesh.json 1.2x, canada.json 1.03–1.08x). Measured directly, libc++ 22's from_chars takes 104 ns per random double, 41 ns per 17-digit value in [0,1), and 19 ns per short number, versus 37, 27, and 14 ns for Apple's strtod_l. 42e3489 therefore uses libc++'s from_chars only where there is no strtod_l; the Apple clang C++17 column shows the result.
• MSVC was not measured.

Public API

No breaking changes. User-visible differences:

• Floats convert correctly under locales with a multi-byte decimal point, and out-of-range values throw out_of_range.406 or become ±0 there too.
• Where std::from_chars reports out of range, the result now comes from the token (±infinity → out_of_range.406, ±0) instead of strtod. The values are the same as strtod produced in the "C" locale.
• libc++ 20+ users (C++17+) on C libraries without strtod_l now get from_chars for float/double: correctly rounded and locale-independent.
• New platform-dependent includes: <xlocale.h> on Apple platforms; number_parse.hpp now includes <clocale>, <cstdlib>, <string>, and <utility> (lexer.hpp no longer includes <clocale>/<cstdlib> directly; they are still included through number_parse.hpp).
• On glibc, Apple, and MSVC, the first float conversion that does not use from_chars allocates a "C" locale object that is intentionally never freed (still reachable, not a leak for LSan/Valgrind's default settings).
• New internal macros (see above), undefined at the end of json.hpp.

Fixes #5660

---

This PR was written by Claude Code on behalf of @nlohmann.

🤖 Generated with Claude Code

[nlohmann]

I measured how this PR affects parsing speed and pushed one change as a result (42e3489).

What I found: step 1 of the proposal, using std::from_chars with libc++ 20 or later, made Apple clang C++17 slower than develop:

input	slowdown
random doubles	1.75x
short numbers such as 123.45	1.3x
mesh.json	1.2x
canada.json	1.03–1.08x

Timed directly, libc++ 22's from_chars is 1.3x to 2.8x slower per number than Apple's strtod_l: 104 vs 37 ns for a random double, 19 vs 14 ns for a short number. Also, the PR called it before Clinger's fast path.

The change: libc++'s from_chars is now only used where the C library has no strtod_l. Apple platforms use Clinger's fast path and then strtod_l, as they do in C++11. The docs and the platform table in the description are updated.

Result: compared with develop, this PR is now equally fast or up to 10% faster on Apple clang (C++11 and C++17). With GCC on macOS, C++11 is also up to 10% faster. On glibc with C++11, it is 15–40% faster on float-heavy input, because strtod_l doesn't need localeconv() or a copy of each number. libstdc++ in C++17 is unchanged. The parsed values are bit-identical to develop in every configuration. The full table is in the new "Performance" section of the description. MSVC was not measured.

_{This comment was written by Claude Code on behalf of @nlohmann.}

[nlohmann]

Closing in favor of #5738 (in the json_view stack #5739, directly after #5617). It fixes #5660 more completely, and with less platform-specific code:

• float, double, and long double where it is binary64 (MSVC, Apple arm64) are converted by the library itself: correctly rounded and independent of the locale. std::from_chars, strtod_l, and strtod are no longer used for them.
  • This is faster than this PR everywhere. canada.json takes 0.49–0.85x the time of this PR, depending on the platform.
  • The parsed values are bit-identical in every configuration measured.
• The remaining strtold fallback (x87 and binary128 long double) gets the multi-byte decimal point fix from this PR: the token is copied with the whole decimal point. Speed up the lexer: own float parser, string scan, and \u table #5738's locale test compares those values with the "C" locale; with x87 long double it fails without the fix.

Parts of this PR that are deliberately dropped:

• std::from_chars on libc++ 20+: not needed; the library's own converter is faster.
• strtold_l with a "C" locale (glibc, Apple, MSVC allowlist, <xlocale.h>, a never-freed locale object): after the fix above, it would only have made x87/binary128 long double about 20% faster in C++11. That doesn't justify the platform code.
• Resolving result_out_of_range from the token:
  • float and double handle out-of-range values themselves now.
  • For long double, strtold produces ±inf or ±0 correctly once the decimal point is substituted correctly.

#5660 will be closed when #5738 lands.

_{This comment was written by Claude Code on behalf of @nlohmann.}