Spark 3.5: scan-based dangling delete removal (backport of apache/iceberg#15727 / port of #4)

[davkim]

Summary

Ports #4 onto 1.11.0-affirm-patch. Second of two backports queued for this base (#1, REST retry semantics, is covered separately -- see below, it drops out entirely).

Same motivation as #4: RemoveDanglingDeletesSparkAction's old partition-min-sequence-number heuristic has structural blind spots (monotonically-increasing keys, deletes committed standalone via CDC newRowDelta() writes) that let dangling delete files accumulate without bound on affected tables. This replaces it with apache#15727's scan-based approach -- ask the query planner which delete files a reader would actually use (LIVE), diff against every delete file in the manifests (ALL), and remove the complement. Still not merged upstream, so it has to travel with us onto every base we cut.

Changes

Same file scope as #4:

File	Change
RemoveDanglingDeletesSparkAction.java	Replaced with the scan-based ALL - LIVE implementation. This also subsumes the separate findDanglingDvs() DV-orphan check that already existed natively on 1.11 -- a DV pointing to a data file with no live FileScanTask is, by construction, in ALL but not LIVE, so a dedicated DV pass is redundant once the general scan-based check is in place.
RewriteDataFilesSparkAction.java	Dangling-delete cleanup now also runs on the "nothing to rewrite" early-return path (plan.totalGroupCount() == 0), not just after a successful rewrite. This is the Affirm-only fix: a table stuck with zero rewritable data files (e.g. every remaining file already correctly sized) could never shed its dangling deletes before this, since cleanup only ran after a rewrite that would never happen.
DeleteFileIndex.java	A DV whose sequence number is below its data file's now returns "does not apply" instead of throwing ValidationException. Detection is itself a scan-planning call (RemoveDanglingDeletesSparkAction#findDanglingDeletes calls scan.planFiles()), so the throw would abort cleanup on exactly the stale DVs it exists to remove. Unreachable pre-format-v3.

API drift from the 1.8.1 base

This needed real adaptation, not a clean cherry-pick -- git cherry-pick -m1 on the merge commit landed 3 conflicts, all from 1.11-era refactors:

• RewriteDataFilesSparkAction moved off planFileGroups() onto the newer FileRewritePlan/planner.plan() planning API sometime between 1.8.1 and 1.11. The extracted removeDanglingDeletesAndBuild helper had to change from taking a raw Builder to taking the already-built Result: on 1.11, a rewrite can already report a non-zero removedDeleteFilesCount() on its own (e.g. orphaned DVs dropped as part of the rewrite commit itself), and that's a lazily-computed sum over rewriteResults(), not a plain stored field. Setting it via a fresh Builder silently discarded that baseline instead of adding to it -- caught by testRemoveDangledPositionDeletesPartitionEvolution's v3 case (expected removedDeleteFilesCount = 1, got 0, because the DV was already gone by the time the dangling-delete pass ran and the helper overwrote the baseline with its own "nothing left to find" count instead of adding to it).
• Dropped a 1.11-native "skip if single-spec unpartitioned" fast path in execute() (comment: "ManifestFilterManager already performs this table-wide delete on each commit"). That assumption doesn't hold for deletes committed standalone via newRowDelta() -- the same write pattern that produces dangling deletes in prod in the first place -- and testUnpartitionedTable (native to 1.11, not part of this backport) exercises exactly that gap. Keeping the fast path would have silently reintroduced a blind spot on unpartitioned tables and failed that pre-existing test.
• Two copies of testPartitionedDeletesWithDanglingDvs (plus its liveEntries/allEntries helpers) ended up in the test file post-merge -- one already native to 1.11 (upstream added DV-orphan detection + this exact test independently between 1.8.1 and 1.11), one carried in by this backport. Byte-identical bodies; kept one, deleted the duplicate (was a compile error otherwise).
• Lists.newArrayList() instead of new ArrayList<>() to satisfy this repo's checkstyle RegexpSingleline rule, which isn't present in the 1.8.1-affirm-patch checkstyle config.

On #1 (REST retry semantics)

Confirmed by diff, not just version number: 1.11.0-affirm-patch already contains apache#12818 and apache#13352 verbatim -- ErrorHandlers throws CommitStateUnknownException on 503/500/502/504, ExponentialHttpRequestRetryStrategy excludes 503 from the non-idempotent retry set while keeping it for idempotent requests, and HTTPClient threads an HttpClientContext through execute(). This PR is what #1 was cherry-picking from mid-2025 upstream, and it landed in the 1.11.0 release itself. Nothing to port.

Testing

• iceberg-core DeleteFileIndex tests: pass
• TestRemoveDanglingDeleteAction: 15 tests, 1 skipped, 0 failures
• TestRewriteDataFilesAction (full suite): 171 tests, 6 skipped, 0 failures
• spotlessCheck + checkstyleMain/checkstyleTest clean on iceberg-core and iceberg-spark-3.5_2.12

Test plan for reviewers

• Confirm the removeDanglingDeletesAndBuild signature change (Builder → Result) reads correctly against ImmutableRewriteDataFiles.Result's lazy-sum semantics
• Confirm dropping the unpartitioned fast path is the right call rather than special-casing it further
• CI green (spotless/checkstyle covered locally above, but full CI including flink-scala-2-12-tests-style flakiness not yet observed)