Skip to content

feat: support bloom filters in native Iceberg writes #5724

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
NikitaMatskevich wants to merge 7 commits into
base: main
Choose a base branch
from
Open

feat: support bloom filters in native Iceberg writes #5724

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
d60c19f
feat: support bloom filters in native Iceberg writes
NikitaMatskevich Sep 5, 2026
b544d41
feat: support Iceberg bloom filter write settings
NikitaMatskevich Sep 6, 2026
b185224
fix: fall back for sanitized Iceberg bloom paths
NikitaMatskevich Sep 8, 2026
4a5ec47
test: gate Iceberg bloom shape properties independently
NikitaMatskevich Sep 8, 2026
fcef2c4
fix: match Iceberg-version bloom property support
NikitaMatskevich Sep 8, 2026
709a360
fix: use Parquet 59 bloom filter APIs
NikitaMatskevich Sep 8, 2026
298cdc7
fix: satisfy formatting checks
NikitaMatskevich Sep 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 40 additions & 2 deletions docs/source/user-guide/latest/iceberg-writes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,9 @@ A write is eligible only when ALL of the following hold:
| `write.parquet.page-version` | unset or `v1` |
| `write.parquet.shred-variants` | unset or `false` (Spark 4.x / Iceberg 1.11 resolve this into every parquet write) |
| `write.parquet.variant-inference-buffer-size` | any value (only meaningful when shredding, which is gated) |
| `write.parquet.bloom-filter-enabled.column.<col>` | unset or `false` |
| `write.parquet.bloom-filter-enabled.column.<col>` | `true` or `false`; an explicit NDV enables the column even when this value is `false`, matching Iceberg's property application order |
| `write.parquet.bloom-filter-fpp.column.<col>` / `write.parquet.bloom-filter-ndv.column.<col>` | For every column named by an `enabled` property, FPP must be a finite double strictly between 0 and 1 and NDV must be a positive Java long no greater than `Long.MAX_VALUE / 8`; the Iceberg FPP default is `0.01` |

@andygrove andygrove Sep 9, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This row states the FPP and NDV rules unconditionally, but interpretedBloomFilterProperties makes them depend on the Iceberg runtime. On 1.5.2 both prefixes are dropped before validation, on 1.8.1 and 1.10.0 only NDV is, and only 1.11 interprets both. Someone on Iceberg 1.10 who sets bloom-filter-ndv.column.x gets neither the requested sizing nor a fallback, and the table as written does not predict that. Could the row name the versions that interpret each property, the way the write.parquet.shred-variants row already calls out Spark 4.x and Iceberg 1.11?

| `write.parquet.bloom-filter-max-bytes` | unset (Iceberg default: 1 MiB), or a power of two from 32 bytes through 128 MiB inclusive; every other value falls back to iceberg-java. A 32-byte value also falls back when explicit NDV/FPP would request a larger filter, because Parquet Java ignores that boundary as a maximum. |
| `write.metadata.metrics.*` | any value (manifest metrics are re-derived on the JVM with Iceberg's own logic) |
| `write.spark.fanout.enabled` | any value (the native writer implements both clustered and fanout modes) |
| `write.target-file-size-bytes` | any value (file rolling cadence differs; see accepted divergences) |
Expand All @@ -164,7 +166,7 @@ A write is eligible only when ALL of the following hold:

Within the namespaces that shape data-file bytes — `write.parquet.*` and `parquet.*` —
everything not listed above must be absent: unvetted `write.parquet.*` keys (e.g.
`bloom-filter-max-bytes`, `stats-enabled.column.*`, keys added by future Iceberg versions),
`stats-enabled.column.*`, keys added by future Iceberg versions),
any `parquet.*` table property (including `parquet.enable.dictionary`), and any `parquet.*`
key in the session Hadoop configuration (with `HadoopFileIO`-backed output those reach
iceberg-java's writer but not the native one). Also gated explicitly: any `encryption.*` key,
Expand Down Expand Up @@ -231,6 +233,42 @@ no reader decision is based on them), differences visible in manifest metadata (
the write and feed later readers' pruning decisions, so each one is analyzed individually
below), and one operational path-layout caveat.

### Parquet Bloom-filter sizing

[Iceberg's documented write properties](https://iceberg.apache.org/docs/latest/configuration/#write-properties)
describe three related inputs. FPP is the requested false-positive probability (default `0.01`),
NDV is the expected number of distinct values when explicitly set, and `max-bytes` is an upper
bound (default 1 MiB).

Apache Parquet Java permits arbitrary integer caps. When such a cap binds, it serializes exactly
that many bytes, although only complete 32-byte SBBF blocks are used and any trailing partial
block remains zero. The Apache Arrow Rust `parquet` crate requires a power-of-two block count so
its post-write folding remains valid. A non-power-of-two cap can therefore change the
hash-to-block mapping, making a filter that may have worse reader pruning than Parquet Java's
filter.

Comet uses its native Iceberg writer only when the effective `max-bytes` value is a power of two
from 32 bytes through 128 MiB inclusive. If an explicit value is not a power of two or is outside
that range, `CometIcebergWriteExec` is not used for the write; Spark's default Iceberg Java writer
writes the table instead. The same fallback applies when `max-bytes=32` would bind an explicit
NDV/FPP request, because Parquet Java ignores exactly 32 bytes as a maximum, and when NDV is above
`Long.MAX_VALUE / 8`, where Parquet Java's sizing multiplication can overflow.

For supported values, Apache Parquet Java applies the sizing properties as follows:

- with no NDV, allocate the full `max-bytes` value;
- with an NDV, calculate a requested size from NDV and FPP, then cap it at `max-bytes`;
- when the cap binds, it takes precedence, so the requested FPP is not guaranteed;
- a large maximum never enlarges the allocation selected by an explicit, smaller NDV.

For every write that is eligible for the native path, Comet applies exactly the same allocation
decision algorithm.

After values are inserted, the Apache Arrow Rust `parquet` crate may fold a sparsely populated
filter to a smaller power-of-two filter while preserving the requested FPP. Parquet Java's
non-adaptive Iceberg path keeps its initial allocation. The native result can consequently use
less file space while remaining safe for every Parquet reader.

### Physical file layout only (cosmetic)

No Iceberg reader bases a planning or correctness decision on these; they change the bytes of
Expand Down
Loading