Skip to content

Support OneRowRelation as a native Comet source #6555

Open

Description

@pingzh

What is the problem the feature request solves?

Comet does not yet have a dedicated native source for Spark's OneRowRelation: exactly one row with zero columns. Scalar queries and constant branches in larger plans should be able to start native execution without depending on Spark to produce that row and convert it to Arrow.

Checked Apache Comet main at 0fdfaad708d5c82195269134e0621dd959e2c0c9:

The main motivation is preserving native execution through larger plans. For example, #4949 documents a Spark 4.2 TPC-DS q77a plan where unsupported one-row branches cause unions and downstream aggregates to fall back.

Describe the potential solution

Introduce a native one-row source and connect it to Spark's OneRowRelation planning paths across supported Spark versions. It should produce one zero-column row in one partition, allowing eligible consumers to run in Comet with spark.comet.sparkToColumnar.enabled=false.

Acceptance criteria:

  • Recognize the actual Spark one-row relation; do not reinterpret arbitrary RDDScanExec inputs as single-row sources.
  • Preserve exactly one row, an empty output schema, and single-partition execution. Keep this distinct from an empty relation, which produces zero rows.
  • Assert both native source selection and Spark-compatible results for scalar projections, UNION ALL with a native input, and aggregate consumers. Check row counts and duplicate preservation.
  • Include cases such as SELECT 42 and SELECT count(*), with the Spark-to-columnar bridge disabled. Ensure the tested Spark plan actually retains a one-row source.
  • Cover the Spark 4.2 plan shape from Spark 4.2: OneRowRelation in Union branches forces Union and downstream aggregates off Comet (TPC-DS q77a) #4949, including the intended native union/aggregate behavior, and exercise AQE on and off.
  • Preserve expression evaluation semantics and existing fallbacks for unsupported consumers.

Additional context

Activity

  1. pingzh commented on Oct 2, 2026

    @pingzh
    ContributorAuthor

    take

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions