Repository navigation
Core: copy ByteBuffers when copying GenericRecord (cherry-pick apache/iceberg#12855) - #18
Merged
abrarsher23 merged 8 commits intoOct 6, 2026
Conversation
(cherry picked from commit a7f3dc7)
Adds version.txt (1.8.1-PATCH.4) and an AFFIRM_VERSIONS.md changelog for the 1.8.1 line, following the convention now used on 1.11.0-affirm-patch. PATCH.1 and PATCH.3 were built from an uncommitted version.txt; their contents are recorded from branch history. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rayhondo
approved these changes
Sep 29, 2026
rayhondo
left a comment
There was a problem hiding this comment.
LGTM. Reviewed the diff and PR description only. I did not check out the branch or run the tests.
- Cherry-pick of apache#12855:
GenericRecord.copy()now deep-copiesByteBufferand nestedGenericRecordvalues. This fixes equality deletes on binary keys collapsing to the last record of each delete file. - The new and updated tests (
DeleteReadTests,TestGenericReaderDeletes,TestGenericRecord) cover the fix. I found no correctness problems. - The
PATCH.4version bump,AFFIRM_VERSIONS.mdentry anddev/.rat-excludeschange are consistent. - Non-blocking:
ListandMapvalues are still shared bycopy(). Equality-delete fields can't be lists or maps, and upstream has the same limit.copy()also allocates per binary or struct field, which matches upstream.
Review authored by Claude Code (Claude Sonnet 5.5) at Ray Kim's request.
darthburrito
approved these changes
Oct 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Fixes a bug in Iceberg 1.8.1 where some equality deletes on binary columns are silently ignored when reading a table.
This is a straight cherry-pick of the upstream fix, apache/iceberg#12855
(for issue #11239), with no changes. It also sets this fork's version
to
1.8.1-PATCH.4.The bug
When Iceberg reads an equality-delete file, it reuses a single buffer for binary values and relies on copying each
record to keep them separate. In 1.8.1 that copy is shallow, so every delete record loaded from the file ends up
pointing at the same buffer, which by the end holds the last record's value. For a binary key, only the last delete
in each delete file actually takes effect.
What that looks like:
writes those duplicates out permanently.
deleted, so live rows can disappear from reads.
The fix
GenericRecord.copy()now makes a real copy of binary (ByteBuffer) and nested-record values instead of sharing them.One production file, about 16 lines. The upstream regression tests come with it.
Testing
iceberg-datasuite passes: 602 tests, 0 failures (JDK 17).tests (binary and struct keys, across Avro, ORC and Parquet) plus
testDeepCopyValues.first with
1.8.1-PATCH.3and then with this patch. Before: duplicate keys. After: the duplicates are gone, and theresults match an independent check computed from the same files.
What's in this PR
core/.../data/GenericRecord.javadata/.../DeleteReadTests.java,TestGenericReaderDeletes.java,TestGenericRecord.javaversion.txt1.8.1-PATCH.4. The build reads this file; without it the jar would be versioned1.9.0-SNAPSHOT.AFFIRM_VERSIONS.md1.11.0-affirm-patch)dev/.rat-excludesAfter merge I'll tag
1.8.1-PATCH.4at the merge commit.Worth knowing
Failing CI checks are pre-existing and unrelated to this change. They fail the same way on other PRs against
1.8.1-affirm-patch(e.g. [LAKE-6732] Spark 3.5: refuse to rewrite a table with duplicate data file registrations #6), and this PR doesn't touch any of the code involved:build-checks:iceberg-core:spotlessJavaCheckExponentialHttpRequestRetryStrategy.java,TestExponentialHttpRequestRetryStrategy.javaandTestRESTCatalog.java, which came in with Prevent Retries 5xx Errors for Non-Idempotent Requests #1. Not touched here, and deliberately not reformatted in this PR.core-tests:iceberg-aws:test— 6 S3 test classes fail at startupminio/minio), and that pull is now denied.kafka-connect-testsThe tests for this fix live in
iceberg-data;:iceberg-data:testruns insidecore-testsand passes. The Spark,Flink and Hive checks pass too. The MinIO failures were fixed on the 1.11 line by Build: Pull MinIO images from Quay (cherry-pick apache/iceberg#18071) #11 (pull the image from Quay instead);
porting that and the formatting fix to this line are separate, small PRs.
1.10.0 and later already contain this fix, so this patch goes away when we move to the 1.11 line.
Change management
CMT approvals happen on this PR. The downstream version-bump PR points here instead of carrying its own.
bigger risk is the manual release process (build the jar, then the Spark package), and a dev build of that went through
a test deployment first.
1.8.1-PATCH.3build. No data or metadata formats change, so the old reader just picks upwhere it left off.