Repository navigation
Support OneRowRelation as a native Comet source #6555
Description
What is the problem the feature request solves?
Comet does not yet have a dedicated native source for Spark's OneRowRelation: exactly one row with zero columns. Scalar queries and constant branches in larger plans should be able to start native execution without depending on Spark to produce that row and convert it to Arrow.
Checked Apache Comet main at 0fdfaad708d5c82195269134e0621dd959e2c0c9:
- The native operator registry and native plan protocol have no dedicated one-row source.
- The existing Spark-to-columnar row path calls
child.execute()and converts the resulting Spark rows. It can enable native consumers, but Spark still produces the input row.
The main motivation is preserving native execution through larger plans. For example, #4949 documents a Spark 4.2 TPC-DS q77a plan where unsupported one-row branches cause unions and downstream aggregates to fall back.
Describe the potential solution
Introduce a native one-row source and connect it to Spark's OneRowRelation planning paths across supported Spark versions. It should produce one zero-column row in one partition, allowing eligible consumers to run in Comet with spark.comet.sparkToColumnar.enabled=false.
Acceptance criteria:
- Recognize the actual Spark one-row relation; do not reinterpret arbitrary
RDDScanExecinputs as single-row sources. - Preserve exactly one row, an empty output schema, and single-partition execution. Keep this distinct from an empty relation, which produces zero rows.
- Assert both native source selection and Spark-compatible results for scalar projections,
UNION ALLwith a native input, and aggregate consumers. Check row counts and duplicate preservation. - Include cases such as
SELECT 42andSELECT count(*), with the Spark-to-columnar bridge disabled. Ensure the tested Spark plan actually retains a one-row source. - Cover the Spark 4.2 plan shape from Spark 4.2:
OneRowRelationin Union branches forces Union and downstream aggregates off Comet (TPC-DS q77a) #4949, including the intended native union/aggregate behavior, and exercise AQE on and off. - Preserve expression evaluation semantics and existing fallbacks for unsupported consumers.
Additional context
- Support OneRowRelation #516 was closed after scalar queries could use the Spark-to-columnar bridge, following minor: Add RDDScan to default value of sparkToColumnar.supportedOperatorList #2422. This request tracks native row generation itself.
- Spark 4.2:
OneRowRelationin Union branches forces Union and downstream aggregates off Comet (TPC-DS q77a) #4949 tracks the specific Spark 4.2 union/aggregate regression. A bridge fix may address that regression independently; this feature should establish a native source that does not require the bridge. - This request is based on source inspection and the existing upstream reports. The illustrative SQL cases above have not been newly executed as reproductions.
Activity
pingzh commentedon Oct 2, 2026
take