What is the problem the feature request solves?
After #4658 is merged, Comet users will have the ability to "split" the OSS iceberg spark writer operator into separate writing and committing operations. Right now, it does not clean up any orphaned files on an aborted operation when a commit fails. Executor task level failures still clean themselves up.
Describe the potential solution
No response
Additional context
No response
What is the problem the feature request solves?
After #4658 is merged, Comet users will have the ability to "split" the OSS iceberg spark writer operator into separate writing and committing operations. Right now, it does not clean up any orphaned files on an aborted operation when a commit fails. Executor task level failures still clean themselves up.
Describe the potential solution
No response
Additional context
No response