[EPIC] Fast manifest reads for PyIceberg through pyiceberg-core #3262

What's the feature are you trying to implement?

PyIceberg wants to read manifests and manifest lists through pyiceberg-core instead of its Cython Avro decoder. Today the pyiceberg-core path is 4x to 5x slower than Cython end to end. This issue tracks the iceberg-rust work needed to make it faster. The overall effort, including the PyIceberg tasks, is tracked in apache/iceberg-python#4007. This issue falls under the PyIceberg integration epic #1694 and revisits #1280.

The full benchmark writeup from @kevinjqliu, including an in-crate profile of Manifest::parse_avro and the script, is in kevinjqliu/iceberg-python#45.

Where the time goes

These numbers come from manifests written by PyIceberg (V2, deflate, 12 columns with full column stats, identity partition). pyiceberg-core is built locally from v0.10.1, once as released (opt-level = "z") and once with opt-level = 3. Each number is the best of 7 runs in ms, single threaded, on Apple Silicon. v0.10.1 predates #3028, which its PR measured as making Manifest::parse_avro about 5% faster.

"End to end" includes building PyIceberg ManifestEntry and DataFile objects from the binding's getters, which is what the Cython path produces. Two separate costs have to go for pyiceberg-core to beat Cython:

• The Rust parse costs 21.7 us per entry at opt-level = 3, against 10.3 us per entry for the whole Cython path. The linked profile puts about 82% of the parse inside apache-avro, in decoding into apache_avro::Value and then resolving against the reader schema. Resolution always runs, because the reader schema iceberg-rust generates never equals a PyIceberg or Java writer schema. Task 1 removed most of this cost. At 10,000 entries and opt-level = 3, Manifest::parse_avro on main now takes 6.7 us per entry on these manifests, and 3.7 us per entry on the same entries written with the 64 KB blocks of perf(avro): buffer records into size-bounded blocks iceberg-python#4008. arrow-avro, which Task 3 uses, decodes the same entries at 4.0 and 1.1 us per entry.
• Converting the binding's output into PyIceberg objects costs another 10.3 us per entry, as much as Cython. That cost doesn't depend on how fast the Rust parser is.

Task list

• Task 1: apache-avro 0.22 and a byte-level manifest reader. perf(manifest): upgrade to apache-avro 0.22 and read manifest entries without Value #3354 upgraded to 0.22 and reads manifest entries with its schema-aware deserializer (feat!: Support enums and tuples in SchemaAwareSerializer and implement SchemaAwareDeserializer avro-rs#512) instead of through Value and schema resolution. It closes Upgrade to using apache-avro 0.22 #3063. In its benchmark at opt-level = 3, parsing a 10,000-entry PyIceberg manifest went from 211.5 ms to 36.8 ms. The writer now defines each named Avro type once, and the reader still reads manifests that iceberg-rust wrote with repeated decimal definitions. equality_ids written as array<long> still read, and a value outside the int range is now an error instead of wrapping.
• Task 2: Build the pyiceberg-core wheel with opt-level = 3. The release profile is set in bindings/python/pyproject.toml#L57-L62, where pyiceberg-core: create smaller artifacts #1841 chose "z" for wheel size. It could also use opt-level = 2, or "z" everywhere except the hot crates. Building v0.10.1 with opt-level = 3 showed three problems. I haven't checked them on main.
  • iceberg-storage-opendal fails to compile with "queries overflow the depth limit" while computing the layout of an opendal_core async fn body. Raising the crate's recursion limit to 256 fixes it.
  • With strip = true, the resulting .so fails to load on macOS with "mis-aligned LINKEDIT string pool". It loads with strip = false.
  • The macOS arm64 wheel grows from 12.6 MB to 21.6 MB.
• Task 3: Read manifests and manifest lists into Arrow with arrow-avro. Decode a manifest or manifest list from its bytes into a RecordBatch with the spec's manifest_entry or manifest_file schema, using arrow-avro with a reader schema for the format version. Expose both through the binding in place of per-entry #[pyclass] objects. The binding already depends on arrow with the pyarrow feature. arrow-avro 59.3 matches the workspace's arrow version, and DataFusion's Avro data source reads through it. On the 10,000-entry PyIceberg manifest above, arrow-avro decodes the file in 11.0 ms with 64 KB blocks, against 42.2 ms for Manifest::parse_avro plus a conversion to Arrow and 84.1 ms for Cython. With one block per entry it takes 40.3 ms, against 71.0 ms and 113.5 ms. These numbers stop at the RecordBatch and don't include the handoff to pyarrow. arrow-avro also reads the manifests that iceberg-rust wrote with repeated decimal definitions. The reader schema comes from the spec, not from any one writer. It matches fields by field ID, as projection requires, and it reads v1 files with the spec's v2+ read behavior, including the 0 defaults for content and sequence numbers in v1 manifests. The reader goes in the iceberg crate, and the binding follows in its own PR.
  • Reader. arrow-avro keeps only the Avro name and element-id in the Arrow field metadata, so the reader restores the field IDs. Avro stores the int-keyed metric maps as arrays of key-value records, which the reader turns into Arrow maps. The partition struct is typed by the manifest's partition spec, including uuid partitions written as string or fixed[16]. The reader also applies inheritance from the manifest list, which neither arrow-avro nor Manifest::parse_avro does, and which PyIceberg's Cython reader does today. It takes the manifest's ManifestFile and fills null snapshot_id, sequence_number, and file_sequence_number values as the manifest_entry fields specify, and null first_row_id values as first row ID inheritance specifies. feat(arrow): read manifest entries into Arrow with arrow-avro #3388 implements it for manifests, and manifest lists follow in a separate PR. Until Bump to Arrow 60 #3257 moves to arrow 60, it works around an arrow-avro 59 bug with records that have no fields (fix(arrow-avro): decode records with no fields arrow-rs#10771), which every unpartitioned manifest has.
  • Binding. Return pyarrow tables from new binding functions. read_manifest_list still hard-codes FormatVersion::V2 (bindings/python/src/manifest.rs#L223-L228), which this PR should fix.
• Task 4: Filter manifest entries over Arrow. Scan planning in PyIceberg filters every entry with partition and metrics evaluators, so returning Arrow alone doesn't help it. iceberg-rust's ManifestEvaluator, ExpressionEvaluator, and InclusiveMetricsEvaluator (expr/visitors) take a parsed DataFile, so filtering with them would bring back the slower Manifest::parse_avro decode. Filtering over the Task 3 output keeps the arrow-avro decode. Whether that runs vectorized in PyIceberg or in an Arrow-based evaluator in Rust is open.
• Raise Python errors instead of panicking in the binding. fix(python): raise instead of panicking in the manifest bindings #3241 replaced the binding's unwrap() calls with Python errors, which fixed Reduce panics in pyiceberg_core #3240 and Python ManifestFile.partitions panics when partition summaries are absent #2883.
• Task 5: Release pyiceberg-core with Tasks 1 to 3 and fix(python): raise instead of panicking in the manifest bindings #3241. PyIceberg pins pyiceberg-core>=0.10.1,<0.11.0.

These items improve iceberg-rust's own readers, Manifest::parse_avro and ManifestList::parse_with_version. PyIceberg won't read through them once Task 3 lands, so they're off this epic's critical path. All but the first are follow-ups to Task 1 that were built during that work but held back to limit Task 1 to the behavior changes that come with the upgrade.

• Read manifest lists without Value. The manifest list reader decodes the whole list into Values and converts it with from_value. perf: parse manifest list entries without buffering Avro values #3224 reads each entry through the Resolved<T> adapter from perf(manifest): upgrade to apache-avro 0.22 and read manifest entries without Value #3354 instead, so it builds no Value for the list or for any record. On a 10,000-entry V2 list, its description measures peak live allocations going from 31,194,672 to 4,355,177 bytes. It also reads V1 lists against the writer schema, which fix(manifest-list): read V1 file counts written with spec field names #3447 needs to fix V1 manifest lists written by Java lose added/existing/deleted file counts #3445.
• Match manifest partition fields by field-id before name. The spec makes the partition struct's field IDs the partition field IDs and stores them in each Avro field's field-id property. Java sanitizes partition field names that aren't valid Avro names and keeps the original in iceberg-field-name (TypeToSchema.java#L116-L128). The reader matches by name, so today it drops those values without an error. This fixes the read side of avro schema reader ignores iceberg-field-name property — returns sanitized names from java manifests #2536. avro schema writer does not sanitize field names that violate avro naming rules #2535 covers the matching fix on the write side.
• Match manifest_entry and data_file fields by field-id as well. A manifest must use valid Iceberg column projection, and projection selects columns by field ID. The reader matches these fields by name through serde. The spec fixes their names, so a mismatch is less likely than for partition fields. A writer that uses other names would have its optional fields read as null and its required fields rejected.
• Return an error for an unknown manifest-list format-version instead of reading it as V2.
• Size partition structs to the partition spec on the JSON DataFile path, as the Avro path already does.
• Write uuid partition values as fixed[16] with logical type uuid, which is how the spec maps uuid to Avro. This changes the file format, so it needs a compatibility check. It overlaps with Partitions with type UUID fail to create transactions #2913 and fix: table UUID partitions not working #2916.
• Regenerate DEPENDENCIES.rust.tsv with cargo-deny 0.19.9.

Willingness to contribute

I can contribute to this feature independently