Skip to content

test: Spark SQL tests that inspect the physical plan fail when LocalTableScanExec is native #6795

Open

Description

@andygrove

Found while enabling spark.comet.exec.localTableScan.enabled by default in draft PR #6780 (Spark 4.1 SQL tests and [scans] job). Part of #4347.

With the default flipped (#6780, Spark 4.1), about 40 upstream tests fail only because the plan contains CometLocalTableScan / CometColumnarToRow where the test expects a Spark node. Examples: ReplaceNullWithFalseInPredicateEndToEndSuite SPARK-25860/33847, LocalScanSuite "full scan", JoinHintSuite "join strategy hint" (4), BroadcastJoinSuite (3), DataFrameSetOperationsSuite SPARK-52921 / SPARK-37371, CTEInlineSuite SPARK-36447 / "CTE Predicate push-down", DataFrameAggregateSuite SPARK-19471 / SPARK-31620, BatchEvalPythonExecSuite Python UDF push down (4), SQLAppStatusListenerSuite (SQLMetrics, codegen stage id), SPARK-46170, DatasetSuite SPARK-43124 / SPARK-47270, "Support ExplainMode". Decide per test: set localTableScan.enabled=false in the dev/diffs, or adapt the assertion.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpriority:lowMinor issues, test failures, tooling, cosmeticspark sql testsSpark SQL test failurestestTesting related

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions