Agree on the scope and repositories for Apache Comet #6789
Copy link
Copy link
Open
Labels
enhancementNew feature or requestNew feature or request
Description
What is the problem the feature request solves?
Part of #6793.
The board resolution defines what the new project is responsible for, and the proposal lists the repositories that move to it. We need to agree on both before the DataFusion PMC votes on the proposal.
Describe the potential solution
Scope statement
DataFusion's resolution says "software related to an extensible query engine". Options for Comet:
- "software related to accelerating Apache Spark using native, columnar execution"
- "software related to Spark-compatible native query execution", which would also cover Spark-compatible components that are not part of the Spark plugin
Repositories and content
- Move apache/datafusion-comet in full and rename it to apache/comet. GitHub redirects the old URLs, as it did when the DataFusion repositories were renamed.
- Decide whether any other code moves. In [DISCUSS] Is Comet ready to move to top-level Apache Comet project? #5184, @comphead suggested moving the
datafusion-sparkcrate from apache/datafusion to Apache Comet. That crate is released with DataFusion and Comet depends on it, so a move needs agreement from the DataFusion community. Code can move between ASF projects after the new project exists, so this decision does not need to block the resolution. - Decide what happens to the Comet content that DataFusion hosts, such as the documentation at https://datafusion.apache.org/comet and the Comet posts on the DataFusion blog.
Additional context
Discussion: #5184
When DataFusion graduated from Arrow, the repositories were discussed in a separate thread.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request
Activity
comphead commentedon Oct 8, 2026
when we say Spark-compatible does that mean LakeSail?
andygrove commentedon Oct 8, 2026
Yes, and there may be others. Personally, I don't think that
data fusion-sparkshould move out of the core project into Comet, but it is good to have the discussion. There is a danger that we would make changes that are good for Comet but not so good for other users, for example.comphead commentedon Oct 8, 2026
It is true, and slightly biased into Comet. Maybe we can do the same thing we did for
datafusion-icebergand movedatafusion-sparkinto separate repo with its own release cadenceandygrove commentedon Oct 8, 2026
That makes sense, but also makes more work for whoever would volunteer to be release manager.