Validate UTF-8 in CBOR/MessagePack/BSON text strings at decode time

[nlohmann]

Summary

from_cbor()/from_msgpack() (and, incidentally, from_bson()/from_ubjson()/from_bjdata(), all of which share the same string-reading code path) decoded text strings by copying raw bytes with no UTF-8 validation. RFC 8949 §3.1 requires CBOR text strings to be valid UTF-8 (MessagePack and BSON have the equivalent requirement for their string types), but this library only ever caught invalid UTF-8 later, if/when the resulting value was dump()'d — so from_cbor(bytes, /*strict*/true, /*allow_exceptions*/false), the pattern used specifically to get a discarded sentinel instead of an exception on malformed input, did not discard this category of malformed input, unlike every other kind of malformed binary input this library already rejects.

auto got = json::from_cbor(std::vector<uint8_t>{0x62, 0xc0, 0xae}, true, false);
got.is_discarded(); // -> false (before this PR)
got.dump();          // -> throws type_error.316, the *only* place this was ever caught

Fixes #5529.

Root cause and fix

binary_reader::get_string() (include/nlohmann/detail/input/binary_reader.hpp) was a thin wrapper around get_bytes() — a raw, chunked byte copy with only an EOF check, also used (unchanged here) by get_binary() for binary-blob reads, which must stay unvalidated. get_string() is the single choke point shared by BSON/CBOR/MessagePack/UBJSON text string reads, so it's the correct place to add the check once.

There was no existing shared UTF-8 validator: the lexer's own scanner is tightly coupled to its streaming state machine, and the only DFA-based, buffer-friendly validator was private to serializer<BasicJsonType> in detail/output/serializer.hpp (used by dump_escaped() for type_error.316). This PR extracts that DFA (decode(), the utf8d table, UTF8_ACCEPT/UTF8_REJECT) into include/nlohmann/detail/string_utils.hpp as free functions/constants in detail::, adds a small is_valid_utf8(...) built on top, and has both serializer.hpp and binary_reader.hpp use the one shared implementation — so there's now exactly one UTF-8 DFA in the codebase instead of a second one waiting to be written.

get_string() now validates the bytes it just read and, on failure, reports it exactly the way every other binary_reader error is reported:

return sax->parse_error(chars_read, get_token_string(),
    parse_error::create(113, chars_read,
        exception_message(format, "invalid string: ill-formed UTF-8 byte", "string"), nullptr));

parse_error.113 already covers other "invalid string" cases for these formats, so this reuses the existing code rather than adding a new one. Going through sax->parse_error() means allow_exceptions is respected automatically (throws when true, propagates to a discarded value when false) — no new plumbing needed.

Base branch

This PR targets bson-sizes (#5185) rather than develop: that PR already touches binary_reader.hpp's string-decoding path in the same area, so stacking avoids an unnecessary later rebase.

Test plan

• tests/src/unit-cbor.cpp / tests/src/unit-msgpack.cpp: new "invalid UTF-8 in string" sections — the exact parse_error.113 message with allow_exceptions=true, is_discarded() with allow_exceptions=false, a check that the identical bytes as a binary blob (CBOR byte string / MessagePack bin8) are still accepted unvalidated, and a valid-UTF-8 (including multibyte) round-trip.
• tests/src/unit-serialization.cpp: regression guard confirming dump()'s existing type_error.316 message/behavior is byte-for-byte unchanged after the decode() extraction.
• Full existing suites for CBOR, MessagePack, BSON, UBJSON, BJData, Unicode, serialization, regression, diagnostics, and deserialization all pass unchanged (no new failures introduced by get_string() now validating BSON/UBJSON/BJData strings too — their existing test data was already valid UTF-8).
• Compiled include/ and single_include/ independently with g++ -std=c++17 -Wall -Wextra -Wshadow -Wpedantic and -std=c++11 — zero warnings.
• make amalgamate run; single_include/nlohmann/json.hpp confirmed in sync with include/.

Breaking change?

No breaking changes to the public API. allow_exceptions=true (default) behavior changes only for previously-malformed input that used to silently decode and only fail later on dump() — it now fails immediately with parse_error.113, consistent with every other kind of malformed binary input. allow_exceptions=false now correctly discards this input instead of returning a value that would later throw on dump(). Valid UTF-8 input and binary-blob reads are unaffected.

🤖 Generated with Claude Code

https://claude.ai/code/session_01N4RQ1Ahan5YAGbnAQGjZTY

---

Generated by Claude Code