Preserve catalog transactions for split Iceberg writes on Spark 4.2

[unikdahal]

What is the problem?

While working on native Iceberg merge-on-read write support in #6240, I noticed a Spark 4.2 transaction-handling gap in Comet's split native Iceberg write path.

Spark 4.2 injects catalog transactions into physical write nodes implementing TransactionalExec.

Comet's split Iceberg write path replaces Spark's normal V2 commit node with IcebergCommitExec. Since this node does not currently implement the Spark 4.2 transaction contract, a catalog transaction attached by Spark is not propagated to the Comet commit node or committed after the Iceberg write.

This is not specific to MOR. It affects the general native Iceberg write path being productionized under #5649, and was surfaced while integrating the row-level write work tracked in #6240.

The expected commit sequence on Spark 4.2 is:

1. commit the Iceberg BatchWrite
2. post driver metrics
3. commit the attached catalog transaction
4. refresh the table cache

Earlier Spark versions do not use this transaction mechanism and should remain unchanged.

Proposed fix

Add a Spark 4.2-specific transaction-aware wrapper around IcebergCommitExec that implements TransactionalExec.

The wrapper should:

• receive the catalog transaction injected by Spark
• preserve it across physical-plan copies/clones
• commit it after the Iceberg write succeeds
• preserve Spark's write/transaction/cache-refresh ordering

Spark 3.x, 4.0, and 4.1 should continue using the existing commit node unchanged through version-specific shims.

Testing

Add Spark 4.2 coverage verifying that:

• split Iceberg writes expose a transaction-aware physical commit node
• the write completes successfully through the Comet split-write path

Related

• [EPIC] Production-quality native Iceberg writes #5649 — production-quality native Iceberg writes
• Support Native Iceberg MOR write #6240 — native Iceberg merge-on-read write support

[unikdahal]

take