Skip to content

Native CASE WHEN names its struct result's fields after the ELSE branch instead of the first THEN branch #6482

Description

@andygrove

Describe the bug

Spark treats two struct types whose field names differ only in case as the same type when spark.sql.caseSensitive is false, the default. So it accepts CASE WHEN branches like named_struct('x', i) and named_struct('X', i) without a cast. CaseWhen.dataType merges the branch types left to right, starting from the first THEN branch, so the result's fields carry the first THEN branch's names.

Since #6350, native CASE WHEN takes its common type from DataFusion's get_coerce_type_for_case_expression, which folds starting from the ELSE branch. It then casts every branch to that type, so the native result carries the ELSE branch's field names on every row.

The values are right, and checkSparkAnswer passes, because row comparison ignores struct field names. The names show up wherever native code reads them from the Arrow type. One example is native to_json, which is opt-in (spark.comet.expression.StructsToJson.allowIncompatible=true). With the default configs, to_json runs through the codegen dispatcher, which evaluates the CASE WHEN in the JVM, so the output is right.

This is on main only, because #6350 isn't in 1.1.0.

Steps to reproduce

// In a suite extending CometTestBase, on main (9c7fcc5aa4)
withSQLConf("spark.comet.expression.StructsToJson.allowIncompatible" -> "true") {
  sql("CREATE TABLE t(q boolean, i int) USING parquet")
  sql("INSERT INTO t SELECT id % 2 = 0, CAST(id AS INT) FROM range(0, 4, 1, 1)")
  checkSparkAnswer(
    "SELECT to_json(CASE WHEN q THEN named_struct('x', i) ELSE named_struct('X', i) END) FROM t")
}

Expected behavior

Spark returns {"x":0}, {"x":1}, {"x":2} and {"x":3}. Comet, with the CASE WHEN and to_json in a CometProject, returns {"X":0}, {"X":1}, {"X":2} and {"X":3}.

Additional context

#6458 builds native IF through create_if_expr, which runs the same coercion starting from the THEN branch for this reason. Folding from the first THEN branch in create_case_when too, with the ELSE branch last, would match CaseWhen.dataType.

Found while addressing review on #6458.

Activity

  1. pingzh commented on Oct 1, 2026

    @pingzh
    Contributor

    take

  2. pingzh commented on Oct 1, 2026

    @pingzh
    Contributor

    @andygrove i can work on it

  3. added
    priority:mediumFunctional bugs, performance regressions, broken features
    and removed on Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:expressionsExpression evaluationbugSomething isn't workingcorrectnesspriority:mediumFunctional bugs, performance regressions, broken features

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions