Skip to content

named_struct with duplicate field names falls back to Spark #5586

Description

@andygrove

Part of #5572.

named_struct with duplicate field names falls the whole projection back to Spark:

if (expr.names.length != expr.names.distinct.length) {
  Unsupported(Some(duplicateNamesReason))
}

spark/src/main/scala/org/apache/comet/serde/structs.scala:42-43

docs/source/user-guide/latest/expressions.md records it as "Duplicate field names fall back".

Spark permits duplicate field names in a struct and CreateNamedStruct.doGenCode builds the row positionally, so the dispatcher reproduces it exactly. StructType is supported by CometBatchKernelCodegen.isSupportedDataType recursively, and duplicate names are not a problem on the Arrow side — the kernel's output field construction is positional too. Worth confirming with a test that the resulting Arrow schema round-trips, since duplicate field names in an Arrow struct are legal but unusual.

Fix: mix CodegenDispatchFallback into CometCreateNamedStruct.

CometCreateNamedStruct also declines inside convert at :66 when a child fails to serialize, which the dispatcher cannot see; see the prerequisite issue on convert-side declines.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions