Skip to content

Agree on the scope and repositories for Apache Comet #6789

Open

Description

@andygrove

What is the problem the feature request solves?

Part of #6793.

The board resolution defines what the new project is responsible for, and the proposal lists the repositories that move to it. We need to agree on both before the DataFusion PMC votes on the proposal.

Describe the potential solution

Scope statement

DataFusion's resolution says "software related to an extensible query engine". Options for Comet:

  1. "software related to accelerating Apache Spark using native, columnar execution"
  2. "software related to Spark-compatible native query execution", which would also cover Spark-compatible components that are not part of the Spark plugin

Repositories and content

  • Move apache/datafusion-comet in full and rename it to apache/comet. GitHub redirects the old URLs, as it did when the DataFusion repositories were renamed.
  • Decide whether any other code moves. In [DISCUSS] Is Comet ready to move to top-level Apache Comet project? #5184, @comphead suggested moving the datafusion-spark crate from apache/datafusion to Apache Comet. That crate is released with DataFusion and Comet depends on it, so a move needs agreement from the DataFusion community. Code can move between ASF projects after the new project exists, so this decision does not need to block the resolution.
  • Decide what happens to the Comet content that DataFusion hosts, such as the documentation at https://datafusion.apache.org/comet and the Comet posts on the DataFusion blog.

Additional context

Discussion: #5184

When DataFusion graduated from Arrow, the repositories were discussed in a separate thread.

Activity

  1. comphead commented on Oct 8, 2026

    @comphead
    Contributor

    when we say Spark-compatible does that mean LakeSail?

  2. self-assigned this
    on Oct 8, 2026
  3. andygrove commented on Oct 8, 2026

    @andygrove
    MemberAuthor

    4. when we say Spark-compatible does that mean LakeSail?

    Yes, and there may be others. Personally, I don't think that data fusion-spark should move out of the core project into Comet, but it is good to have the discussion. There is a danger that we would make changes that are good for Comet but not so good for other users, for example.

  4. comphead commented on Oct 8, 2026

    @comphead
    Contributor

    There is a danger that we would make changes that are good for Comet but not so good for other users, for example.

    It is true, and slightly biased into Comet. Maybe we can do the same thing we did for datafusion-iceberg and move datafusion-spark into separate repo with its own release cadence

  5. andygrove commented on Oct 8, 2026

    @andygrove
    MemberAuthor

    7. > There is a danger that we would make changes that are good for Comet but not so good for other users, for example.

    It is true, and slightly biased into Comet. Maybe we can do the same thing we did for `datafusion-iceberg` and move `datafusion-spark` into separate repo with its own release cadence
    

    That makes sense, but also makes more work for whoever would volunteer to be release manager.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions