Part of #5572.
to_csv never runs inside Comet by default. Every path through CometStructsToCsv.getSupportLevel returns Unsupported or Incompatible:
if (containsComplexType) {
return Unsupported(...)
}
if (containsIncompatibleDataTypes) {
return Incompatible(...)
}
// https://github.com/apache/datafusion-comet/issues/3232
Incompatible()
spark/src/main/scala/org/apache/comet/serde/structs.scala:271-288
The serde has no CodegenDispatchFallback mixin, so with the default configuration the enclosing operator falls back to Spark for every to_csv call. The native path is reachable only with spark.comet.expression.StructsToCsv.allowIncompatible=true.
The rest of the structured-text family does not behave this way. to_json is Hybrid, and from_csv and schema_of_csv are both plain CometCodegenDispatch (serde/csv.scala:24, :26). to_csv is the odd one out.
There is also a docs bug here: docs/source/user-guide/latest/expressions.md lists to_csv as "✅ Native" with no note, which reads as "runs natively by default". It does not. Adding the mixin fixes the docs too, since GenerateDocs derives the Implementation column from the trait mixins and would then report it as Hybrid.
Fix: mix CodegenDispatchFallback into CometStructsToCsv. The complex-type arm dispatches fine — arrays, structs and maps are all in CometBatchKernelCodegen.isSupportedDataType as long as their leaves are.
Part of #5572.
to_csvnever runs inside Comet by default. Every path throughCometStructsToCsv.getSupportLevelreturnsUnsupportedorIncompatible:spark/src/main/scala/org/apache/comet/serde/structs.scala:271-288The serde has no
CodegenDispatchFallbackmixin, so with the default configuration the enclosing operator falls back to Spark for everyto_csvcall. The native path is reachable only withspark.comet.expression.StructsToCsv.allowIncompatible=true.The rest of the structured-text family does not behave this way.
to_jsonis Hybrid, andfrom_csvandschema_of_csvare both plainCometCodegenDispatch(serde/csv.scala:24,:26).to_csvis the odd one out.There is also a docs bug here:
docs/source/user-guide/latest/expressions.mdliststo_csvas "✅ Native" with no note, which reads as "runs natively by default". It does not. Adding the mixin fixes the docs too, sinceGenerateDocsderives the Implementation column from the trait mixins and would then report it as Hybrid.Fix: mix
CodegenDispatchFallbackintoCometStructsToCsv. The complex-type arm dispatches fine — arrays, structs and maps are all inCometBatchKernelCodegen.isSupportedDataTypeas long as their leaves are.