What is the problem the feature request solves?
For high-cardinality columns, the native Iceberg writer produces Parquet files with a large dictionary page where iceberg-java produces none, so selective reads of native-written files fetch several MB more than they need to.
parquet-mr, which iceberg-java writes through, drops dictionary encoding for a column chunk when the dictionary is not paying for itself: the first time it checks, if the dictionary-encoded size plus the dictionary is not smaller than the raw size, it switches the whole column chunk to plain and writes no dictionary page. parquet-rs has no such check. It keeps dictionary encoding until the dictionary reaches dictionary_page_size_limit (Iceberg's write.parquet.dict-size-bytes, 2 MB by default), then falls back to plain for the rest of the chunk and still writes the full dictionary page.
Measured on the fixture of CometIcebergNativeSuite "native scan skips pages within a single row group": 1M rows of id BIGINT (sorted, unique) and payload STRING (16 hex characters, unique), one 512 MB row group, uncompressed, 2000-row pages.
|
iceberg-java (parquet-mr 1.17.1) |
native (parquet-rs) |
id encodings |
PLAIN, BIT_PACKED, no dictionary page |
RLE_DICTIONARY, PLAIN, RLE, 2.10 MB dictionary page |
payload encodings |
PLAIN, RLE, BIT_PACKED, no dictionary page |
RLE_DICTIONARY, PLAIN, RLE, 2.10 MB dictionary page |
bytes scanned for WHERE id BETWEEN 1000 AND 1100 |
0.6 MB |
4.7 MB (of a 20.7 MB file) |
Every read that touches a column chunk has to fetch its dictionary page, so page-index pruning can't bring a selective read below roughly 2 MB per such column per row group. The full-scan cost is small (the dictionary is mostly redundant bytes), but the selective-read cost is what page skipping and row-group pruning exist to avoid. Results are correct either way. This is a file layout and read-performance divergence.
It showed up when #5677 turned the native writer on by default and the test above started failing its byte bound. The test now writes its fixture on the JVM, so nothing in CI pins this today.
Describe the potential solution
Match parquet-mr's decision closely enough that high-cardinality columns written natively carry no dictionary page. Options, roughly in order of preference:
- Upstream a compression-ratio fallback to parquet-rs (an opt-in
WriterProperties setting that abandons the dictionary, and discards it, when the first page shows no saving), then enable it from build_writer_properties in native/core/src/execution/operators/iceberg_write.rs.
- Decide per column in Comet, for example by disabling the dictionary for a column once a sample of the first batch shows near-unique values. Cheaper to land, but it's a heuristic that won't match parquet-mr's cut-off exactly.
- Leave the dictionary on and document the divergence under the accepted differences in
iceberg-writes.md.
Whichever is chosen, add a test that writes a high-cardinality column natively and asserts there is no dictionary page (or a bounded one), so the layout is pinned.
Additional context
Part of #5649 (Phase 5, performance). It should be settled or explicitly accepted before the native writer is on by default (#5644), since it affects every native-written table with unique-ish columns (ids, UUIDs, timestamps).
What is the problem the feature request solves?
For high-cardinality columns, the native Iceberg writer produces Parquet files with a large dictionary page where iceberg-java produces none, so selective reads of native-written files fetch several MB more than they need to.
parquet-mr, which iceberg-java writes through, drops dictionary encoding for a column chunk when the dictionary is not paying for itself: the first time it checks, if the dictionary-encoded size plus the dictionary is not smaller than the raw size, it switches the whole column chunk to plain and writes no dictionary page. parquet-rs has no such check. It keeps dictionary encoding until the dictionary reaches
dictionary_page_size_limit(Iceberg'swrite.parquet.dict-size-bytes, 2 MB by default), then falls back to plain for the rest of the chunk and still writes the full dictionary page.Measured on the fixture of
CometIcebergNativeSuite"native scan skips pages within a single row group": 1M rows ofid BIGINT(sorted, unique) andpayload STRING(16 hex characters, unique), one 512 MB row group, uncompressed, 2000-row pages.idencodingsPLAIN, BIT_PACKED, no dictionary pageRLE_DICTIONARY, PLAIN, RLE, 2.10 MB dictionary pagepayloadencodingsPLAIN, RLE, BIT_PACKED, no dictionary pageRLE_DICTIONARY, PLAIN, RLE, 2.10 MB dictionary pageWHERE id BETWEEN 1000 AND 1100Every read that touches a column chunk has to fetch its dictionary page, so page-index pruning can't bring a selective read below roughly 2 MB per such column per row group. The full-scan cost is small (the dictionary is mostly redundant bytes), but the selective-read cost is what page skipping and row-group pruning exist to avoid. Results are correct either way. This is a file layout and read-performance divergence.
It showed up when #5677 turned the native writer on by default and the test above started failing its byte bound. The test now writes its fixture on the JVM, so nothing in CI pins this today.
Describe the potential solution
Match parquet-mr's decision closely enough that high-cardinality columns written natively carry no dictionary page. Options, roughly in order of preference:
WriterPropertiessetting that abandons the dictionary, and discards it, when the first page shows no saving), then enable it frombuild_writer_propertiesinnative/core/src/execution/operators/iceberg_write.rs.iceberg-writes.md.Whichever is chosen, add a test that writes a high-cardinality column natively and asserts there is no dictionary page (or a bounded one), so the layout is pinned.
Additional context
Part of #5649 (Phase 5, performance). It should be settled or explicitly accepted before the native writer is on by default (#5644), since it affects every native-written table with unique-ish columns (ids, UUIDs, timestamps).