fix(arrow): strip metadata from list elements and map entries

[NoahKusaba]

Which issue does this PR close?

• Closes strip_metadata_from_schema fails on list and map columns #3297.

What changes are included in this PR?

strip_metadata_from_schema failed with Field stack underflow in list on any schema with a list or map column. MetadataStripVisitor pushes each field's name and nullability onto a stack in before_field and pops it when rebuilding the field. For list elements and map keys and values, the traversal calls before_list_element, before_map_key and before_map_value instead of before_field, so rebuilding them popped from an empty stack. The visitor now pushes in those three hooks too.

While here:

• The stack holds each field's name and nullability instead of a Field with a DataType::Null placeholder, and one pop_field helper rebuilds fields for struct, list, map and primitive.
• The doc comment states two normalizations the function already made, which are now reachable for maps: a map's entries field is renamed to DEFAULT_MAP_FIELD_NAME, as schema_to_arrow_schema names it, and a dictionary-encoded field becomes its value type. Callers comparing stripped schemas rely on both. For example, datafusion-iceberg compares an INSERT's input, whose map entries field DataFusion names entries, against the table's schema.

Are these changes tested?

Three unit tests in arrow::schema, each comparing the whole stripped schema with the expected one:

• test_strip_metadata_from_nested_schema: lists, large and fixed-size lists, sorted and unsorted maps, and a list of structs holding a map of lists, with metadata on the schema and every field. Fails without the fix with the error from the issue.
• test_strip_metadata_renames_map_entries
• test_strip_metadata_unwraps_dictionaries

AI Disclosure

• Used Claude Code to review the fix, simplify the visitor, write the tests and draft this description. I reviewed the change.

[NoahKusaba]

cc @mbutrovich, @comphead

[comphead]

Two small docs points.

• ArrowSchemaVisitor::before_field and after_field are documented as called around a "struct/list/map field". The traversal calls them for every schema and struct field, primitives included, and never for list elements or map keys and values, which have their own hooks. A visitor that pushes in before_field only, like MetadataStripVisitor did, underflows on lists and maps. Consider rewording to "Called before each field of the schema or of a struct, whatever its type. List elements and map keys and values use before_list_element, before_map_key and before_map_value instead." and mirroring it on after_field.
• The comment on test_strip_metadata_renames_map_entries says a map from arrow-rs or DataFusion compares equal to a table's once stripped. That holds for DataFusion's map() (entries, key, value), but not for an arrow-rs MapBuilder default (entries, keys, values), as the test's own keys and values fixture shows. Rewording it to say only the entries field is renamed would match what the test asserts.

[comphead]

This says map key and value names are kept, but it is silent on list element names. Are they kept on purpose? arrow-rs names list elements item (Field::new_list_field) and schema_to_arrow_schema names them element, so a stripped List(item) still differs from a stripped table List(element).

If keeping them is the intent, one more sentence here would complete the contract. If callers need them normalized, LIST_FIELD_NAME is the name to rebuild with. That may not matter for datafusion-iceberg if DataFusion already casts the input to the table's nested names, which is unverified.

[NoahKusaba]

Keeping them is intentional: renaming list elements would change what callers see, and the map rename only exists to match schema_to_arrow_schema. I added a paragraph in 1980676 that says all other names are kept, including list elements, and calls out the item vs element case.

[comphead]

Thanks @NoahKusaba it looks good to me, some nits

[NoahKusaba]

@comphead Thanks! Both doc points are addressed in 1980676: I used your wording for before_field/after_field, and the test comment now says only the entries field is renamed.