Repository navigation
fix: keep default values of proto3 fields without presence in protobuf ingestion - #20485
Merged
amaechler merged 4 commits intoOct 6, 2026
Merged
Conversation
…estion ProtobufConverter built rows from Message.getAllFields(), which omits singular fields without presence tracking (proto3 fields not marked optional) when they hold their default value. Those columns were ingested as null instead of 0, "", false or the first enum value. Add such fields explicitly.
FrankChen021
reviewed
Oct 6, 2026
FrankChen021
left a comment
FrankChen021
left a comment
Member
There was a problem hiding this comment.
🟢 Approval recommended
The converter now backfills only singular fields without presence from the message descriptor, while preserving existing handling for present, optional, oneof, message, repeated, and map fields. The added tests cover implicit proto3 scalar/enum defaults and defaults inside a present nested message. No actionable correctness issue was found in the merge-base diff or the relevant surrounding protobuf ingestion code.
Reviewed 3 of 3 changed files.
Validation: git diff --check 1265b47c9a51252295a4aed42d95c459b00bf2a3 1893e6228d585f8e1ce6018c0f5a0d6b9dfd74fe passed.
This is an automated review by Codex GPT-5.6-Luna(max)
Ingests proto3 records through Kafka and Kinesis supervisors and queries the default-valued fields. Also fixes the ProbufData typo in MoreResources and adds the missing timeout on test_protobufDataFormat.
abhishekrb19
approved these changes
Oct 6, 2026
This was referenced Oct 8, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #18792.
Description
Protobuf ingestion turned proto3 fields that are not marked
optionalintonullwhenever they used their default value (0,"",false, or the first enum value). Users had to wrap such fields innvl()in transforms to get the actual value back.ProtobufConverter.convertMessagebuilt each row fromMessage.getAllFields(). That method only returns fields that are "set", and for proto3 fields without explicit presence a default value is indistinguishable from unset when serialized, so those fields were dropped from the row.convertMessagenow also walks the message descriptor and adds every singular field whereFieldDescriptor.hasPresence()is false and whichgetAllFields()did not already return, usingmsg.getField()to read the default. This mirrorsJsonFormat.printer().alwaysPrintFieldsWithNoPresence(), with one deliberate difference: empty repeated and map fields are still omitted, as before, so they keep ingesting asnullrather than an empty array.Release note
Protobuf ingestion now keeps the default value of proto3 fields that are not marked
optional. Previously anint32field holding0(or astringholding"", aboolholdingfalse, an enum holding its first value) was ingested asnull. Such fields now ingest as their actual value.This PR has: