Add BON8 support

[nlohmann]

This PR adds support for BON8 (Binary Object Notation 8), see #2980. BON8 uses the byte values that cannot begin a UTF-8 character as type markers, so strings are stored as plain UTF-8 without a length prefix. Small integers, true/false/null, and -1.0/0.0/1.0 take one byte, and containers with up to four elements need no terminator.

The branch was rebuilt from scratch on top of develop; the previous commits (2021/2022) could not read back their own output.

Size

BON8 is the most compact of the supported formats on the benchmark files (sizes relative to minified JSON):

Format	canada.json	twitter.json	citm_catalog.json	jeopardy.json
CBOR	50.5 %	86.3 %	68.4 %	88.0 %
MessagePack	50.5 %	86.0 %	68.5 %	87.9 %
BON8	50.5 %	83.8 %	63.5 %	87.5 %

Changes

• to_bon8 (vector, uint8_t and char output adapters) and from_bon8 (input, iterator/sentinel), plus input_format_t::bon8 for sax_parse.
• Reader: non-recursive, built on container_stack like the other binary readers. A string ends at the first byte that cannot continue it, which is the first byte (or, for integers that start with a UTF-8 lead byte, the first two bytes) of the next value; the reader hands these bytes back to the next value through a two-byte pushback buffer. Strings are fully validated as UTF-8. Non-canonical input (longer encodings than necessary, 0x85/0x8B containers with few elements, unsorted keys, unneeded 0xFF) is accepted.
• Writer: produces the specification's canonical representation (shortest encodings, binary32 whenever lossless, -0.0/infinity/NaN as binary32, NaN as 0x7F800001, sorted keys for json), except for NFC normalization. Unsigned integers above int64 throw out_of_range.407, strings that are not valid UTF-8 throw type_error.316, and binary values are written as arrays of integers (like UBJSON/BJData).
• Tests: unit-bon8.cpp with the integer test vectors of the reference implementation (HikoGUI, Boost Software License), the examples of the specification and of BON8 support #2980, every integer from -300000 to 600000, string-termination cases, error messages, SAX aborts, deep nesting, and byte-for-byte round trips over 140 files of the test data. BON8 was also added to the other cross-format tests (sizes, alternative string/object/binary types, ordered_json, output sinks, deque arrays), the benchmarks, and a new fuzzer fuzzer-parse_bon8.cpp (picked up by OSS-Fuzz's build.sh automatically).
• Test data: the .bon8 files were generated with HikoGUI's reference encoder and published in json_test_data v3.2.0 (Add BON8 test files json_test_data#6). Before that, the reference encoder and to_bon8 were checked to produce identical bytes on every file, and each decoder to read the other's output back. JSON_TEST_DATA_VERSION is bumped to 3.2.0.
• Docs: a feature page with the type mappings, API pages for to_bon8/from_bon8, examples, and BON8 in the size table, README, and format lists.

Note that the specification's example {"a":["b","c"],"d":1} omits the 0xFF bytes after "b" and "c" that its own rules require; the tests use the corrected encoding 88 61 82 62 FF 63 FF 64 91.

Public API

No breaking changes. The change only adds API: basic_json::to_bon8, basic_json::from_bon8, and the enumerator input_format_t::bon8, which is appended at the end so the values of the existing enumerators are unchanged.

---

This description and the implementation were written by Claude Code.

🤖 Generated with Claude Code

[coveralls]

Coverage decreased (-0.2%) to 99.842% when pulling f9003d1 on bon8 into 8fcdbf2 on develop.

[gregmarr]

@nlohmann @takev Is this still in progress?

[nlohmann]

It is currently on hold from my side. It's definitely a nice idea for the format, but I currently don't find the time to push this further.

[nlohmann]

I revived this. I still thing BON8 is a nice format, and from all implemented, it's the most compact one.

[gregmarr]

Why is this bon8 specific?

[nlohmann]

Good point: it's a property of the input adapter, not of BON8. Renamed it to bulk_scan, like the lexer's equivalent, in 53ab406. To show it's not BON8-specific, BSON now uses it too (a1381a6): its keys and array indices are null-terminated and were read one byte at a time. They're now read up to the 0x00 in one step, which reads twitter.json 27% faster and citm_catalog.json and jeopardy.json 12% faster. The length-prefixed formats don't need it; they already memcpy contiguous input via get_elements().

(Written by Claude Code.)

[gregmarr]

Is this different than the other utf8 bulk functions?

[nlohmann]

Yes, in which bytes stop it. string_bulk_run() / scalar_string_bulk_run() scan the inside of a JSON string, so they also stop at ", \ and control characters. In a BON8 string those are ordinary characters: only an invalid or incomplete UTF-8 sequence ends the run, and that is also where the string ends. That's also why the simdutf path doesn't carry over. It validates the run up to the next delimiter in one call, but a BON8 string has no delimiter to find in advance; its end only shows up as the first byte that isn't valid UTF-8. Apart from the stop set, the loop is the same: skip ASCII 8 bytes at a time, then validate_one_utf8() for anything else.

(Written by Claude Code.)