[DrakeLin]

Description

This PR updates the Delta Protocol specification to clarify that JSON commit files at checkpoint versions must be preserved during metadata cleanup. This ensures that commitInfo actions remain accessible for table features such as In-Commit Timestamps (ICT).

The current Delta Protocol's metadata cleanup section could be interpreted to allow deletion of JSON commit files when a checkpoint exists at the same version. However, this creates a critical problem for ICT-enabled tables:

1. Checkpoints don't preserve commitInfo actions - The protocol explicitly states that checkpoints must not preserve commit provenance information
2. ICT writers need previous commit timestamps - ICT-enabled tables need to read the previous commit's inCommitTimestamp to ensure monotonicity

Thus, if a checkpoint exists without its corresponding JSON commit file, the commitInfo is lost, causing writes to the table to fail.

How was this patch tested?

Does this PR introduce any user-facing changes?

[scottsand-db]

nit: move this line up to the one above it

[felipepessoto]

@DrakeLin do you know if current Spark implementation follows the new spec?

[DrakeLin]

@DrakeLin do you know if current Spark implementation follows the new spec?

Both Delta Spark and Kernel Java implicitly require this:

• Delta Spark
• Kernel Java

[scovich]

... and AFAIK, both have always required it

[DrakeLin]

@scovich Anything missing before we can merge?

[scovich]

LGTM. Both delta-spark and delta-kernel have always required this, so it's really just clarifying a constraint that always existed.

[felipepessoto]

@DrakeLin, @scottsand-db @scovich

Although Spark Delta preserve the JSON commit at checkpoint version, it doesn't validate it exists, like Kernel does:

#4090

delta/kernel/kernel-api/src/main/java/io/delta/kernel/internal/snapshot/SnapshotManager.java

Line 419 in 900fc55

String.format("Missing delta file for version %s", latestCompleteCheckpointVersion));

The Spark PR was never merged: #3080