from_cbor()/from_msgpack() do not validate UTF-8 in text strings at decode time (only dump() does)

Description

from_cbor() (and, by the same code path, from_msgpack()) decodes a CBOR/MessagePack text string by copying its raw bytes into a string_t without validating that they are well-formed UTF-8. RFC 8949 §3.1 requires CBOR major type 3 (text string) to contain "a UTF-8 string", and rejecting ill-formed UTF-8 there is expected decoder behavior — but this library defers that check to dump(), so from_cbor() itself neither throws nor returns a discarded value for invalid UTF-8 payloads. The invalid bytes only surface as an error later, if and when the resulting value is serialized back to JSON text.

Reproduction

#include <nlohmann/json.hpp>
#include <iostream>
using json = nlohmann::json;

int main() {
    // CBOR: 0x62 = text string, length 2; payload 0xC0 0xAE is not valid UTF-8
    std::vector<uint8_t> raw = {0x62, 0xc0, 0xae};

    auto got = json::from_cbor(raw, /*strict*/true, /*allow_exceptions*/false);
    std::cout << "is_discarded: " << got.is_discarded() << "\n";   // false
    std::cout << "type: "        << got.type_name()     << "\n";   // string

    got.dump(); // throws json.exception.type_error.316:
                // "invalid UTF-8 byte at index 0: 0xC0"
}

So from_cbor() reports success (a normal, non-discarded string value) for input the CBOR spec requires to reject; the error only appears later and only if the caller happens to call dump() (or another UTF-8-sensitive operation) on the resulting value.

Traced to binary_reader::get_string(), which forwards straight to get_bytes() — a raw byte copy with no UTF-8 check. Contrast with the text/JSON lexer, which does validate UTF-8 during scanning (scan_string() in lexer.hpp), and with dump()'s own validation (type_error.316).

Why this seems worth a look

docs/features/binary_formats/bson.md documents comparable BSON leniency explicitly, with its own "Lenient BSON input handling" admonition and an escape hatch ("validate it separately before passing it to from_bson()"). I could not find an equivalent note for CBOR/MessagePack on the CBOR docs page or the from_cbor API docs — so this specific deferred-validation behavior currently isn't documented anywhere I could find.
It means from_cbor(..., /*allow_exceptions=*/false) — the pattern used specifically to avoid exceptions and get a discarded sentinel on bad input instead — does not actually catch this category of malformed input; the exception can still surface later, from an unrelated call (dump()), if the caller isn't also validating UTF-8 there.

Suggested fix / alternatives

Either (a) validate UTF-8 in get_string()/get_bytes() at CBOR/MessagePack decode time, consistent with the text-JSON lexer, or (b) document the deferred-validation behavior the way BSON's leniency is documented, so it's a known, intentional tradeoff rather than a surprise.

Context

Found while investigating a third-party compliance report (#5525) — most of that report's findings turned out to be a bug in the reporting tool itself, but this particular case survived verification against the current develop header (commit 3bfe2b6da7393af5cfd68c44f58a1058e60499e2).

🐛 reject ill-formed UTF-8 in CBOR/MessagePack/BSON text strings at de…

Reproduced on develop (c72f37a), and it is not limited to CBOR and MessagePack — UBJSON has the same gap, and it is undocumented there too.

Feeding the ill-formed payload C0 AE as a text string to each reader, with strict=true, allow_exceptions=false:

format	input	`is_discarded()`	`dump()`
CBOR	`62 C0 AE`	false	throws `type_error.316`
MessagePack	`A2 C0 AE`	false	throws `type_error.316`
UBJSON	`53 69 02 C0 AE`	false	throws `type_error.316`
BSON	`0F 00 00 00 02 6B 00 03 00 00 00 C0 AE 00 00`	false	throws `type_error.316`

BSON behaving that way is expected given the "Lenient BSON input handling" note, so the finding is that UBJSON belongs in the same bucket as CBOR/MessagePack — accepted silently, nothing in the UBJSON docs mentioning it.

One correction to my own numbers, in case anyone repeats this: my first BSON attempt reported is_discarded() == true, which I nearly wrote up as "BSON validates". It does not — I had hand-written the document with a wrong length prefix (0x10 instead of 0x0F) and it was rejected as malformed for that reason. The row above uses a document built by to_bson() with only the two payload bytes swapped, and it round-trips cleanly before the swap.

On your (a)/(b): I am happy to send either, with tests, but they point in quite different directions for callers who currently rely on the lenient behaviour, so I would rather not guess. If you want (a), is validating in get_string() for all four readers the intent, or only the three that are currently undocumented? If you want (b), I can add the CBOR/MessagePack/UBJSON equivalents of the BSON admonition plus a note on the from_* API pages. Just say which and I will put it together.

Thanks for the thorough follow-up — good catch on UBJSON/BJData, and on the underlying reason the bug still reproduces on develop: #5531 was merged onto the still-open bson-sizes branch (#5185) rather than develop, so none of the fix had actually landed on develop yet.

Went with (a): ported the same get_string() fix directly onto develop on branch claude/issue-5529-comment-9f1lck. Since get_string() is the single choke point shared by all five binary readers, this covers CBOR, MessagePack, BSON, UBJSON, and BJData in one place — no per-format special-casing needed. Added decode-time tests for all five formats and the UBJSON/BJData docs admonitions that were missing from #5531's docs updates. No need to send a separate PR for this — happy to fold in anything else you spot, though.

Generated by Claude Code

One correction to the framing used in this issue and in #5531/the develop port above: none of the affected specifications actually require a decoder to reject ill-formed UTF-8.

Checked the primary sources directly:

CBOR (RFC 8949 §3.1): text strings are defined as UTF-8, but the RFC explicitly says "a decoder might or might not want to verify that the sequence of bytes in a UTF-8 string is actually valid UTF-8 and react appropriately." It even draws a formal distinction between "well-formed" (structurally valid CBOR) and "valid" (also has proper UTF-8 text) — an item with ill-formed UTF-8 in a text string is well-formed CBOR, just not "valid" under the full data model. Decoder-side validation is explicitly optional.
MessagePack: the type system section calls str "a UTF-8 string", but the spec's own Limitations section says: "String objects may contain invalid byte sequence and the behavior of a deserializer depends on the actual implementation when it received invalid byte sequence." Leniency is explicitly sanctioned here too.
BSON (bsonspec.org): defines the string grammar as "the (byte) is zero or more UTF-8 encoded characters"*, but says nothing anywhere about what a decoder should do with malformed UTF-8. It's silent, not permissive-by-name like CBOR/MessagePack, but also not a rejection requirement.
UBJSON / BJData: same as BSON — both spec texts say the string value "requires UTF-8 encoding" as a type definition, but neither says anything about decoder-side error handling.

So the validation added on claude/issue-5529-comment-9f1lck is a deliberate policy choice by this library (matching what dump() already enforces, and closing a real gap: none of the five to_* writers validate UTF-8 either, so a value with ill-formed UTF-8 could previously be silently decoded from one format and silently re-encoded into another, e.g. CBOR → BSON, with the invalid bytes surviving undetected) — not something any of these specs mandates. Worth being precise about that distinction in the docs/comments rather than saying the specs "require" it, since only the write side and non-dump() consumers were actually silent about it before; the specs themselves never obligated rejection.

Generated by Claude Code

So let's please discuss: Should we change the behavior of the binary readers to reject non-UTF-8 strings for all formats?

@gregmarr

I know we just merged #5531, but I would be happy to revert if we agree that these changes were overly strict. /cc @gregmarr

Sounds to me that we were following "be strict in what you generate and lenient in what you accept" before. Since this is a non-required validation, I think this would need to be opt-in.

Validate UTF-8 in CBOR and MessagePack text strings at decode time

🐛 reject ill-formed UTF-8 in CBOR/MessagePack/BSON text strings at de…

Fix BSON conformance issue (#5185)

Accept ill-formed UTF-8 in all binary readers again

@nlohmann

Validating UTF-8 at decode time is now possible, so I'm closing this.

Following the discussion here, the binary readers stay lenient by default, and validation is opt-in per call:

Follow each binary format's UTF-8 rule: strict writers (CBOR/UBJSON/BJData/BSON), lenient readers #5741 restored the lenient default for the readers ("be strict in what you generate, lenient in what you accept"). The writers can reject ill-formed UTF-8 with JSON_STRICT_BINARY_UTF8.
Add an error_handler parameter for UTF-8 to the binary readers and writers #5746 added an error_handler parameter to from_cbor, from_msgpack, from_ubjson, from_bjdata, and from_bson. It reuses error_handler_t from dump(). With error_handler_t::strict, ill-formed UTF-8 in a text string or an object key throws parse_error.113 (or yields a discarded value with allow_exceptions = false). replace and ignore behave as in dump().

Checked on develop (ef570827e) with a text string containing the byte 0xFF:

cbor strict:    [json.exception.parse_error.113] parse error at byte 2: syntax error while parsing CBOR string: invalid string: ill-formed UTF-8 byte
msgpack strict: [json.exception.parse_error.113] parse error at byte 2: syntax error while parsing MessagePack string: invalid string: ill-formed UTF-8 byte
ubjson strict:  [json.exception.parse_error.113] parse error at byte 4: syntax error while parsing UBJSON string: invalid string: ill-formed UTF-8 byte
bson strict:    [json.exception.parse_error.113] parse error at byte 13: syntax error while parsing BSON string: invalid string: ill-formed UTF-8 byte
cbor key strict: [json.exception.parse_error.113] parse error at byte 3: syntax error while parsing CBOR key: invalid string: ill-formed UTF-8 byte

Without the argument, the same input is still accepted, as in 3.12.0.

Written by Claude Code on behalf of @nlohmann.

Navigation Menu

Repository navigation

from_cbor()/from_msgpack() do not validate UTF-8 in text strings at decode time (only dump() does) #5529

Description

Reproduction

Why this seems worth a look

Suggested fix / alternatives

Context

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

from_cbor()/from_msgpack() do not validate UTF-8 in text strings at decode time (only dump() does) #5529

Description

Description

Reproduction

Why this seems worth a look

Suggested fix / alternatives

Context

Activity

xhon-pelushi commented on Sep 16, 2026

nlohmann commented on Sep 16, 2026

nlohmann commented on Sep 16, 2026

nlohmann commented on Sep 16, 2026

nlohmann commented on Sep 16, 2026

gregmarr commented on Sep 17, 2026

nlohmann commented on Oct 7, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions