Compaction treats Parquet scan ranges as whole files and can lose rows during partial progress #2287

Apache Iceberg version

main (development), a4fb4d0dafaeaf455990c1c3142462107662e5ec; reproduced locally with Go 1.26.3 on macOS arm64.

Please describe the bug 🐞

compaction.Analyze passes Scan.PlanFiles results directly to Config.PlanCompaction. Parquet planning can return several byte-range tasks for one physical file, but the compaction planner counts and bin-packs each task as a separate file, using the full file size for every range.

ExecuteCompactionGroup reads only the assigned range and then schedules the whole source file for replacement. When partial progress puts ranges in separate commit batches, the first batch can commit only part of the file's rows. A later batch fails because the source file has already been removed from the current snapshot. Returning that error does not undo the first durable commit.

Reproduction

This needs no concurrent writer or external service:

1. Create a local, unpartitioned v2 table with one required id: long column and property read.split.target-size=1.

2. Add one valid Parquet file containing IDs 1–12 in twelve one-row row groups. The tiny split size keeps the reproduction small; PlanFiles returns twelve ranges with the same data-file path.

3. Let size be that file's size and call compaction.Analyze with:

   cfg := compaction.Config{
       TargetFileSizeBytes: size + 1,
       MinFileSizeBytes:    size + 1,
       MaxFileSizeBytes:    size * 2,
       MinInputFiles:       1,
       DeleteFileThreshold: 1,
       PackingLookback:     compaction.DefaultPackingLookback,
   }

4. Convert each returned compaction.Group to table.CompactionTaskGroup, preserving its tasks and size. The planner returns twelve one-task groups, although there is only one physical file.

5. Call txn.RewriteDataFiles(ctx, groups, table.RewriteDataFilesOptions{PartialProgress: true, MaxCommits: 12}) without injecting cancellation or a catalog failure.

6. Scan result.Table, the latest committed table returned with the error.

Observed twice, and repeated independently:

rewritten groups=1
error=stage partial rewrite batch: cannot delete files that do not belong to the table
rows after failed later group: 1 of 12

With the same rows and compaction settings but an unsplit file task, compaction succeeds and retains all twelve rows. With the five-input threshold, repeated source paths can instead fail duplicate-removal validation before committing; this is not a claim that the default configuration always loses data.

Expected behavior

Compaction must plan and replace complete physical files. Each committed group must preserve all live source rows, including when later partial-progress batches fail. Counts, sizes, and the minimum-input threshold should count a source file once. Incomplete explicit worker groups should fail before replacing a whole file with partial rows.

Relevant code

• Analyze passes scan tasks to the planner
• Per-task packing with whole-file sizes
• Range reads followed by whole-file removal

The reproduction shows rows missing from the committed active snapshot, not immediate physical deletion of the original file. An older retained snapshot may permit recovery.

I checked existing issues and PRs, including #1939, #2041 and #2276; those do not address this file/range-grain mismatch. I plan to work on a focused fix with regression coverage.