Skip to content

[Feature](lance) Support additional Lance and Arrow data types #66496

Open

Description

@Gabriel39

Parent

Tracked by #66340. Core Arrow and Lance scalar, nested, and vector types were introduced by #65730.

Scope

  • Lance Blob v2.
  • Arrow JSON Extension.
  • Lance BFloat16 Extension.
  • Arrow null.
  • Arrow duration.
  • Preserve extension metadata and define Doris type mappings and unsupported-column behavior.

Completion criteria

  • FE schema conversion and BE Arrow deserialization are implemented safely.
  • Unit and regression tests cover schema discovery, projection, nullability, and malformed input.
  • User documentation describes mappings, precision, and limitations.
  • Representative datasets for every listed type are verified.

Partial completion — 2026-09-30

#67325 merged FE/BE mappings and tests for Arrow/Lance JSON, BFloat16, Null, and Duration into branch-4.1; #67921 added nested nullable Arrow Null support and regression coverage. Both PRs have successful recorded External Regression statuses. English and Chinese mapping/precision/limitation documentation is synchronized by merged apache/doris-website#4182.

These four type items are complete. JSON requires compatible extension storage, BFloat16 widens to FLOAT, Duration preserves the signed count in its declared unit, and unsupported extensions/storage combinations remain rejected. The current catalog documentation still explicitly lists Lance Blob v2 as unsupported.

Keep this issue open for Blob v2. The general extension-metadata and all-types completion criteria remain unchecked because they include that outstanding type. Completion on branch-4.1 satisfies the parent tracker's branch criterion; this audit does not claim a fresh test run.

Activity

  1. added
    kind/featureCategorizes issue or PR as related to a new feature.
    area/catalogIssues or PRs related to catalog management
    on Aug 5, 2026
  2. wangzhaobo957-cloud commented on Aug 24, 2026

    @wangzhaobo957-cloud

    I'd like to work on this issue. Please assign it to me.

  3. 28 remaining items

  4. added 3 commits that reference this issue on Sep 14, 2026
    5445bdf
    be96287
    42bf158
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

area/catalogIssues or PRs related to catalog managementkind/featureCategorizes issue or PR as related to a new feature.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions