Skip to content

Iceberg data tables whose names end with a metadata-table name fall back to Spark #6719

Open

Description

@unikdahal

Describe the bug

CometScanRule.isIcebergMetadataTable matches the table name with endsWith against the metadata table names (files, snapshots, history, entries, manifests, partitions, ...) with no separator. Any ordinary Iceberg table whose name ends with one of those words is treated as a metadata table and falls back to Spark with Iceberg Metadata tables are not supported, for example user_files, order_history, daily_partitions, or a table named snapshots.

val name = scanExec.table.name().toLowerCase(Locale.ROOT)
metadataTableSuffix.exists(name.endsWith)

Steps to reproduce

CREATE TABLE cat.db.user_files (id INT, name STRING) USING iceberg;
INSERT INTO cat.db.user_files VALUES (1, 'a'), (2, 'b');
SELECT * FROM cat.db.user_files;

With spark.comet.scan.icebergNative.enabled=true and spark.comet.explain.fallback.enabled=true the scan is reported as:

BatchScan cat.db.user_files [COMET: Iceberg Metadata tables are not supported]

The same table named users gets CometIcebergNativeScan.

Expected behavior

Only real metadata tables (cat.db.user_files.files, ...metadata.json#files) should fall back.

Additional context

Proposed fix: recognize a metadata table by its Iceberg type. Every metadata table extends org.apache.iceberg.BaseMetadataTable, and the table is reachable through the existing IcebergReflection.getTable(scan). Fall back to a whole-segment name match (.files, #files) only when the table cannot be reached. Happy to open a PR with a test covering user_files, snapshots and order_history plus a real .files read.

Activity

  1. aryan9948 commented on Oct 6, 2026

    @aryan9948

    Hi! I’d like to work on this issue and submit a PR with the regression tests and fix described above.

    Would it be possible to assign this issue to me?

  2. added
    bugSomething isn't working
    priority:mediumFunctional bugs, performance regressions, broken features
    area:scanParquet scan / data reading
    and removed on Oct 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:scanParquet scan / data readingbugSomething isn't workingpriority:mediumFunctional bugs, performance regressions, broken features

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions