Skip to content

[Variant] Export complete Arrow Fields through the native FFI boundary #5547

Description

@peterxcli

Problem

The native Arrow C Data exporter constructs FFI_ArrowSchema from ArrayData.data_type(). That preserves the physical datatype but drops the top-level Field name, nullability, and metadata. Logical types represented by Field metadata cannot survive that boundary reliably.

Parent: #5546

Solution

  • Pass the corresponding RecordBatch Field to every native array-export call.
  • Construct FFI_ArrowSchema from that Field.
  • Keep FFI_ArrowArray export unchanged.
  • Ensure offset-normalized arrays still use the original output Field.
  • Substitute embedded NUL only in the exported C-string name while preserving datatype, nullability, and metadata.

This task must not add Variant scan admission or Variant-specific schema inference.

Tests

  • Top-level Field name, nullability, and metadata survive schema export.
  • Offset-normalized arrays retain the original Field.
  • A top-level name containing NUL does not panic or lose metadata.
  • Existing non-Variant array exports remain unchanged.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:ffiArrow FFI / JNI boundaryenhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions