Skip to content

Lift the metrics-mode counts/none eligibility restriction for native Iceberg writes #5642

Description

@andygrove

What is the problem the feature request solves?

Native Iceberg writes fall back when write.metadata.metrics.default or a per-column write.metadata.metrics.column.* is counts or none (CometIcebergNativeWrite trigger rules, pinned by CometIcebergWriteDetectionSuite). The restriction was introduced in #5298 because iceberg-rust could not produce those metrics modes, and iceberg-writes.md says it "remains only until that assembly lands".

#5361 moved manifest metrics assembly to the JVM: DataFile metrics are rebuilt at commit time from the written file's footer using Iceberg's own MetricsConfig / ParquetUtil code, with NaN counts supplied by the native writer. The parity tests in CometIcebergWriteActionSuite already cover mode none with a per-column full override and mode counts with WRITE ORDERED BY. That is the machinery the restriction was waiting for, so the restriction may now be liftable with no change to the writer.

Describe the potential solution

  1. Confirm what the JVM rebuild produces for counts (counts only, no bounds) and none (no metrics) against iceberg-java output for the same data, including the sorted-column promotion that MetricsConfig applies under counts.
  2. If identical, delete the counts / none trigger rules, update the allowlist table in iceberg-writes.md, and turn the existing detection-suite fall-back tests into parity tests.
  3. If not identical, document the specific divergence and keep the rule.

Additional context

Part of the native Iceberg writes epic, #5649. Related: #5298 (detection), #5361 (native writer and JVM metrics assembly).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions