Skip to content

[EPIC] Complex Type Support #1043

Description

@andygrove

What is the problem the feature request solves?

We would like Comet to fully support complex types (arrays, structs, and maps). This issue is for tracking all of the individual issues.

Google doc: https://docs.google.com/document/d/1eiDFEScPjxBMahJW6lmBI8JjVlI6CwhiJgkTSsTvPVY/edit?usp=sharing

Implement new native scans based on DataFusion's DataSourceExec

We now have new native_datafusion and native_iceberg_compat scans that use DataFusion's DataSourceExec, which already supports complex types.

We need to fix the remaining Spark SQL test failures:

Known issues:

Other scan-related work

These items may not be relevant to all users, but for some environments, there is more work required to allow the new ParquetExec scans to be used. Comet's current default native_comet scan is JVM-based and leverages Hadoop data source functionality that is not available in DataFusion.

Supporting expressions that operate on complex types

Performance

  • Create benchmarks for complex types

Testing

Older / related issues:

Describe the potential solution

No response

Additional context

No response

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions