What is the problem the feature request solves?
Comet now has two experimental write features under two different config namespaces:
| Config |
Added by |
spark.comet.write.iceberg.splitOperator.enabled |
#4658 |
spark.comet.parquet.write.enabled |
native Parquet writes |
One is spark.comet.write.<format>.<feature>, the other spark.comet.<format>.write.<feature>. Both are CATEGORY_TESTING and default to false, so nothing user-facing is broken yet — which makes this the cheap moment to pick one shape, before either graduates and the key has to be kept as a deprecated alias.
Worth settling alongside it: the operator opt-in for native Parquet writes is spark.comet.operator.WriteFilesExec.allowIncompatible (with spark.comet.operator.DataWritingCommandExec.allowIncompatible retained as an alternative), so a user enabling native Parquet writes currently sets two configs while the Iceberg feature needs one.
Describe the potential solution
Pick one namespace and move the other with withAlternative, per the rename checklist in docs/source/contributor-guide/config_conventions.md. spark.comet.write.* reads better as writes grow more formats and features, but either is fine as long as it is one.
What is the problem the feature request solves?
Comet now has two experimental write features under two different config namespaces:
spark.comet.write.iceberg.splitOperator.enabledspark.comet.parquet.write.enabledOne is
spark.comet.write.<format>.<feature>, the otherspark.comet.<format>.write.<feature>. Both areCATEGORY_TESTINGand default to false, so nothing user-facing is broken yet — which makes this the cheap moment to pick one shape, before either graduates and the key has to be kept as a deprecated alias.Worth settling alongside it: the operator opt-in for native Parquet writes is
spark.comet.operator.WriteFilesExec.allowIncompatible(withspark.comet.operator.DataWritingCommandExec.allowIncompatibleretained as an alternative), so a user enabling native Parquet writes currently sets two configs while the Iceberg feature needs one.Describe the potential solution
Pick one namespace and move the other with
withAlternative, per the rename checklist indocs/source/contributor-guide/config_conventions.md.spark.comet.write.*reads better as writes grow more formats and features, but either is fine as long as it is one.