Upgrade to using apache-avro 0.22 #3063

Activity

1. JosephLenton

   added

   enhancementNew feature or requestNew feature or request

   on Aug 25, 2026
2. JosephLenton commented on Aug 25, 2026

   

   JosephLenton

   on Aug 25, 2026

   Last edited by JosephLenton

   ContributorAuthor

   More actions

   I'm just going to dump code changes I've found here to capture them.

   1. RecordField::order is gone

   Before: In 0.21.0 the Avro docs order does nothing and is ignored. See: https://docs.rs/apache-avro/0.21.0/apache_avro/schema/struct.RecordFieldBuilder.html#method.order

   Now: the order field is just gone.

   2. The first_row_id needs to be serialised as i64 not u64

   We are serialising a u64, however Avro now expects an i64 for a long, and the spec says it should be signed as well. https://iceberg.apache.org/spec/#manifest-lists
3. kevinjqliu commented on Sep 5, 2026

   

   kevinjqliu

   on Sep 5, 2026

   Contributor

   More actions

   @JosephLenton thanks for opening this issue! I'm keen to upgrade to 0.22. From my benchmark, utilizing the new SchemaAwareDeserializer (apache/avro-rs#512) drastically speed up avro file parsing 🚀

   Please let me know if there's anything I can help with here!

   🎉React with 🎉1Reacted by Joseph Lenton
4. JosephLenton commented on Sep 5, 2026

   

   JosephLenton

   on Sep 5, 2026

   ContributorAuthor

   More actions

   Great to hear @kevinjqliu. I have the upgrade done on a draft PR, however Avro 0.22 has a regression which I think blocks the upgrade.
5. kevinjqliu commented on Sep 5, 2026

   

   kevinjqliu

   on Sep 5, 2026

   Contributor

   More actions

   I looked into apache/avro-rs#654 and I don't think it actually blocks us. It only drops logicalType when parsing schema JSON. The only code we have that reads that attribute is avro_schema_to_schema, and nothing calls it outside its own tests. Writing is unaffected, so manifests come out the same as before.

   For the failing test, I made avro/schema.rs also recognize the map-as-array shape, i.e. an array whose item is a record named k<keyId>_v<valueId> with key and value fields. Java, PyIceberg and iceberg-go all write it that way so there's no ambiguity. The attribute check still runs first, so once 0.23 is out with the fix we can just drop the shape check.

   I've got the upgrade working locally with this plus the API changes. Want me to open a PR, or would it be easier if I send you the diff for your draft?
6. laskoviymishka

   closed this as completedin #3354on Oct 9, 2026