Self-Describing Binary Log Format (v3) by masaori335 · Pull Request #13231 · apache/trafficserver

masaori335 · 2026-06-03T06:31:07Z

Motivation

In version 2, a segment header carries the field symbols (fmt_fieldlist,
e.g. "chi cqu pssc") and a printf-style template (fmt_printf) but
not the field types. To decode an entry a reader had to already know the
type of each symbol, because the value encodings are only self-delimiting once
the type is known (IP is variable length, for example). That coupled every
out-of-tree parser to the exact ATS build that wrote the log.

Version 3 adds one thing: a per-segment field-type schema that lists the
wire type of every field, in field order. Decoding then needs only the symbols
(as keys) and the schema (for types).

Depends on

Copilot

Pull request overview

This PR introduces binary log format v3 for Apache Traffic Server logging, making .blog segments self-describing by embedding a per-segment field-type schema. This decouples binary log decoding from the exact ATS build that produced the log and adds tooling/tests/docs around the new format.

Changes:

Add a v3 per-segment field-type schema to the binary log segment header and make segment header reads version-sized (v2/v3 compatibility).
Extend traffic_logcat with a schema-driven JSON output mode (-j/--json) and harden iteration/decoding against malformed segments.
Add unit + gold tests and documentation for the v3 on-disk format and configuration (binary_log_version).

Reviewed changes

Copilot reviewed 31 out of 31 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
tests/gold_tests/logging/gold/binary_log_v3_json.gold	Gold output for `traffic_logcat -j` JSON decoding of v3 logs.
tests/gold_tests/logging/gold/binary_log_v3_ascii.gold	Gold output for ASCII decoding of v2/v3 logs.
tests/gold_tests/logging/binary_log_v3.test.py	End-to-end gold test exercising v2/v3 binary logs and v3 JSON output.
src/traffic_logstats/logstats.cc	Read segment headers using on-disk header size; add corruption guards.
src/traffic_logcat/unit-tests/test_LogEntryJson.cc	Unit tests for schema-driven v3 JSON reference decoder.
src/traffic_logcat/LogEntryJson.h	Public interface/contract for v3 entry-to-JSON reference decoder.
src/traffic_logcat/LogEntryJson.cc	Implementation of schema-only v3 JSON decoder with bounds checks.
src/traffic_logcat/logcat.cc	Add `-j/--json` option and version-sized header reads; emit JSON lines.
src/traffic_logcat/CMakeLists.txt	Build `LogEntryJson` into `traffic_logcat` and add Catch2 unit test target.
src/proxy/logging/YamlLogConfig.cc	Add `binary_log_version` logging.yaml key parsing and propagation to LogObject.
src/proxy/logging/unit-tests/test_LogBuffer.cc	Unit tests covering v3 schema/type alignment and version-sized header sizing/iteration.
src/proxy/logging/LogObject.cc	Extend LogObject ctor to store per-object `binary_log_version`.
src/proxy/logging/LogFormat.cc	Update type enum usage for aggregation checks.
src/proxy/logging/LogFilter.cc	Update type enum usage and improve unknown-type error reporting.
src/proxy/logging/LogFile.cc	Accept v2/v3 segment versions for ASCII conversion paths.
src/proxy/logging/LogField.cc	Convert type to scoped enum and update assertions/display formatting.
src/proxy/logging/LogBuffer.cc	Write v3 type schema, set per-object segment version, and harden iterator.
src/proxy/logging/LogAccess.cc	Fix pointer advancement bug in `unmarshal_http_version`.
src/proxy/logging/Log.cc	Align field declarations to new `LogField::Type` and fix several field types (incl. ppv).
src/proxy/logging/CMakeLists.txt	Add Catch2 unit test target for LogBuffer v3.
src/api/InkAPI.cc	Pin TSLogType-to-LogField::Type relationship and validate plugin-provided types.
include/ts/apidefs.h.in	Update TSLogType enum values/comments to mirror v3 wire-type codes.
include/proxy/logging/LogObject.h	Add `binary_log_version` ctor arg/default and accessor/storage.
include/proxy/logging/LogFormat.h	Expose `field_list()` for writer-side schema emission in v3.
include/proxy/logging/LogField.h	Redefine `LogField::Type` as a stable, append-only wire-code enum class.
include/proxy/logging/LogBuffer.h	Bump `LOG_SEGMENT_VERSION` to 3; add schema struct and helper sizing/version utilities.
doc/developer-guide/logging-architecture/index.en.rst	Add v3 format page to logging architecture docs index.
doc/developer-guide/logging-architecture/binary-log-v3-format.en.rst	New specification document for v3 on-disk format and decoding rules.
doc/appendices/command-line/traffic_logstats.en.rst	Document v2/v3 support and reference v3 format spec.
doc/appendices/command-line/traffic_logcat.en.rst	Document v2/v3 support and new `-j/--json` option behavior.
doc/admin-guide/files/logging.yaml.en.rst	Document `binary_log_version` key and provide v2/v3 configuration examples.

bneradt

Looks like a good improvement.

Copilot

Pull request overview

Copilot reviewed 31 out of 31 changed files in this pull request and generated 10 comments.

Copilot

Pull request overview

Copilot reviewed 31 out of 31 changed files in this pull request and generated 11 comments.

masaori335 · 2026-06-22T00:16:14Z

+  field = new LogField("proxy_protocol_dst_ip", "ppd", LogField::Type::IP, &LogAccess::marshal_proxy_protocol_dst_ip,
                       &LogAccess::unmarshal_ip_to_str);
  global_field_list.add(field, false);
  field_symbol_hash.emplace("ppdip", field);


Pre-exiting issue prior to this change. I'll fix it in other PR.

Publish each field's type in a per-segment schema so a generic reader can decode a .blog from the file alone, without an embedded ATS symbol-to-type table that must track the writer in lockstep. The per-field code is LogField::Type serialized directly (now an enum class : uint8_t with INVALID=0 reserved and sINT..IP = 1..4 as the frozen wire codes); a static_assert pins the values. This relies on each field's declared type matching its marshalled framing, which the parent commit ("Fix mismatched sINT/dINT log field types") establishes. Readers (LogBufferIterator, logcat, logstats, the ASCII output paths) accept both v2 and v3 segments, sizing the header read to the on-disk version, so a v3 build keeps decoding logs written by an older one. Integer values stay in host byte order, as in v2 (no endianness change). The public TSLogType enum is given the same values as LogField::Type so TSLogFieldRegister can static_cast between them; static_asserts in InkAPI.cc (the only TU that sees both) pin the alignment so a future reorder fails to compile. The writer version is per-LogObject: logging.yaml "binary_log_version: 2" pins a binary log to the pre-v3 layout (no schema, shorter header) so a not-yet-upgraded downstream parser keeps working during a migration; the default is v3. Decoding untrusted .blog input is bounded: LogBufferIterator validates data_offset and each entry against the segment, and the JSON decoder validates the schema offset alignment and cross-checks field_count against the symbol list.

Copilot

Pull request overview

Copilot reviewed 31 out of 31 changed files in this pull request and generated 3 comments.

fmt_fieldlist() and fmt_fieldtypes() form a pointer from a header offset read off disk; an out-of-range value makes the pointer arithmetic undefined behavior even if never dereferenced. Guard both against byte_count.

* Self-describing binary log format (LogBuffer v3) Publish each field's type in a per-segment schema so a generic reader can decode a .blog from the file alone, without an embedded ATS symbol-to-type table that must track the writer in lockstep. The per-field code is LogField::Type serialized directly (now an enum class : uint8_t with INVALID=0 reserved and sINT..IP = 1..4 as the frozen wire codes); a static_assert pins the values. This relies on each field's declared type matching its marshalled framing, which the parent commit ("Fix mismatched sINT/dINT log field types") establishes. Readers (LogBufferIterator, logcat, logstats, the ASCII output paths) accept both v2 and v3 segments, sizing the header read to the on-disk version, so a v3 build keeps decoding logs written by an older one. Integer values stay in host byte order, as in v2 (no endianness change). The public TSLogType enum is given the same values as LogField::Type so TSLogFieldRegister can static_cast between them; static_asserts in InkAPI.cc (the only TU that sees both) pin the alignment so a future reorder fails to compile. The writer version is per-LogObject: logging.yaml "binary_log_version: 2" pins a binary log to the pre-v3 layout (no schema, shorter header) so a not-yet-upgraded downstream parser keeps working during a migration; the default is v3. Decoding untrusted .blog input is bounded: LogBufferIterator validates data_offset and each entry against the segment, and the JSON decoder validates the schema offset alignment and cross-checks field_count against the symbol list. * Address Copilot's comment * Address Copilot's comment * Cleanup * Fix logcat for AuTest * Range-check untrusted .blog offsets before pointer arithmetic fmt_fieldlist() and fmt_fieldtypes() form a pointer from a header offset read off disk; an out-of-range value makes the pointer arithmetic undefined behavior even if never dereferenced. Guard both against byte_count. (cherry picked from commit f31bfc5)

* Self-Describing Binary Log Format (v3) (#13231) * Self-describing binary log format (LogBuffer v3) Publish each field's type in a per-segment schema so a generic reader can decode a .blog from the file alone, without an embedded ATS symbol-to-type table that must track the writer in lockstep. The per-field code is LogField::Type serialized directly (now an enum class : uint8_t with INVALID=0 reserved and sINT..IP = 1..4 as the frozen wire codes); a static_assert pins the values. This relies on each field's declared type matching its marshalled framing, which the parent commit ("Fix mismatched sINT/dINT log field types") establishes. Readers (LogBufferIterator, logcat, logstats, the ASCII output paths) accept both v2 and v3 segments, sizing the header read to the on-disk version, so a v3 build keeps decoding logs written by an older one. Integer values stay in host byte order, as in v2 (no endianness change). The public TSLogType enum is given the same values as LogField::Type so TSLogFieldRegister can static_cast between them; static_asserts in InkAPI.cc (the only TU that sees both) pin the alignment so a future reorder fails to compile. The writer version is per-LogObject: logging.yaml "binary_log_version: 2" pins a binary log to the pre-v3 layout (no schema, shorter header) so a not-yet-upgraded downstream parser keeps working during a migration; the default is v3. Decoding untrusted .blog input is bounded: LogBufferIterator validates data_offset and each entry against the segment, and the JSON decoder validates the schema offset alignment and cross-checks field_count against the symbol list. * Address Copilot's comment * Address Copilot's comment * Cleanup * Fix logcat for AuTest * Range-check untrusted .blog offsets before pointer arithmetic fmt_fieldlist() and fmt_fieldtypes() form a pointer from a header offset read off disk; an out-of-range value makes the pointer arithmetic undefined behavior even if never dereferenced. Guard both against byte_count. (cherry picked from commit f31bfc5) * Default binary log format to v2 for 10.2.x compatibility ATS 10.2.x downstream log parsers predate the v3 self-describing format, so binary logs must default to the v2 layout. Split the single LOG_SEGMENT_VERSION macro (which served as both "latest" and "default writer") into LOG_SEGMENT_VERSION (3, still the max written/read and the opt-in target) and LOG_SEGMENT_VERSION_DEFAULT (2, the writer default). v3 stays fully readable and selectable per-object via logging.yaml binary_log_version: 3. * Fmt

cmcfarlen · 2026-07-13T14:39:25Z

Backported to 10.2.x by #13339

* Self-describing binary log format (LogBuffer v3) Publish each field's type in a per-segment schema so a generic reader can decode a .blog from the file alone, without an embedded ATS symbol-to-type table that must track the writer in lockstep. The per-field code is LogField::Type serialized directly (now an enum class : uint8_t with INVALID=0 reserved and sINT..IP = 1..4 as the frozen wire codes); a static_assert pins the values. This relies on each field's declared type matching its marshalled framing, which the parent commit ("Fix mismatched sINT/dINT log field types") establishes. Readers (LogBufferIterator, logcat, logstats, the ASCII output paths) accept both v2 and v3 segments, sizing the header read to the on-disk version, so a v3 build keeps decoding logs written by an older one. Integer values stay in host byte order, as in v2 (no endianness change). The public TSLogType enum is given the same values as LogField::Type so TSLogFieldRegister can static_cast between them; static_asserts in InkAPI.cc (the only TU that sees both) pin the alignment so a future reorder fails to compile. The writer version is per-LogObject: logging.yaml "binary_log_version: 2" pins a binary log to the pre-v3 layout (no schema, shorter header) so a not-yet-upgraded downstream parser keeps working during a migration; the default is v3. Decoding untrusted .blog input is bounded: LogBufferIterator validates data_offset and each entry against the segment, and the JSON decoder validates the schema offset alignment and cross-checks field_count against the symbol list. * Address Copilot's comment * Address Copilot's comment * Cleanup * Fix logcat for AuTest * Range-check untrusted .blog offsets before pointer arithmetic fmt_fieldlist() and fmt_fieldtypes() form a pointer from a header offset read off disk; an out-of-range value makes the pointer arithmetic undefined behavior even if never dereferenced. Guard both against byte_count.

masaori335 added this to the 11.0.0 milestone Jun 3, 2026

masaori335 self-assigned this Jun 3, 2026

Copilot AI review requested due to automatic review settings June 3, 2026 06:31

masaori335 added the Logging label Jun 3, 2026

masaori335 added this to ATS v10.2.x Jun 3, 2026

Copilot started reviewing on behalf of masaori335 June 3, 2026 06:31 View session

Copilot AI reviewed Jun 3, 2026

View reviewed changes

masaori335 marked this pull request as draft June 3, 2026 23:44

bneradt reviewed Jun 8, 2026

View reviewed changes

Comment thread src/proxy/logging/LogAccess.cc

masaori335 force-pushed the binary-format-v3 branch from 8358d71 to 5810e78 Compare June 9, 2026 08:35

masaori335 marked this pull request as ready for review June 9, 2026 08:36

Copilot AI review requested due to automatic review settings June 9, 2026 08:36

Copilot started reviewing on behalf of masaori335 June 9, 2026 08:36 View session

Copilot AI reviewed Jun 9, 2026

View reviewed changes

masaori335 marked this pull request as draft June 9, 2026 09:05

masaori335 force-pushed the binary-format-v3 branch from 5810e78 to 1a828f4 Compare June 21, 2026 23:30

masaori335 marked this pull request as ready for review June 21, 2026 23:30

Copilot AI review requested due to automatic review settings June 21, 2026 23:30

Copilot started reviewing on behalf of masaori335 June 21, 2026 23:30 View session

Copilot AI reviewed Jun 21, 2026

View reviewed changes

bneradt previously approved these changes Jun 24, 2026

View reviewed changes

masaori335 added 5 commits June 25, 2026 10:01

Address Copilot's comment

dc359a5

Address Copilot's comment

bca679b

Cleanup

75d014c

Fix logcat for AuTest

d1ad070

Copilot AI review requested due to automatic review settings June 25, 2026 01:39

masaori335 dismissed bneradt’s stale review via d1ad070 June 25, 2026 01:39

masaori335 force-pushed the binary-format-v3 branch from b4a4a51 to d1ad070 Compare June 25, 2026 01:39

Copilot started reviewing on behalf of masaori335 June 25, 2026 01:39 View session

Copilot AI reviewed Jun 25, 2026

View reviewed changes

Comment thread src/traffic_logcat/LogEntryJson.cc

Comment thread src/traffic_logcat/logcat.cc

Comment thread src/traffic_logcat/logcat.cc

Range-check untrusted .blog offsets before pointer arithmetic

9f92eb8

fmt_fieldlist() and fmt_fieldtypes() form a pointer from a header offset read off disk; an out-of-range value makes the pointer arithmetic undefined behavior even if never dereferenced. Guard both against byte_count.

bneradt approved these changes Jun 25, 2026

View reviewed changes

masaori335 merged commit f31bfc5 into apache:master Jun 25, 2026
15 checks passed

github-project-automation Bot moved this to For v10.2.0 in ATS v10.2.x Jun 25, 2026

masaori335 mentioned this pull request Jun 26, 2026

[10.2.x] Self-Describing Binary Log Format (v3) (#13231) #13339

Merged

masaori335 added the Incompatible label Jun 26, 2026

cmcfarlen modified the milestones: 11.0.0, Backported Jul 13, 2026

cmcfarlen removed this from ATS v10.2.x Jul 13, 2026

Uh oh!

Conversation

masaori335 commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bneradt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

masaori335 Jun 22, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cmcfarlen commented Jul 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

masaori335 commented Jun 3, 2026 •

edited

Loading