CometNativeException: "arrays of different length" when using to_date on Iceberg Timestamp column #3255

Activity

1. boudica-dev-eng

   added

   bugSomething isn't workingSomething isn't working

   on Jan 23, 2026
2. vigneshsiva11 commented on Jan 26, 2026

   

   vigneshsiva11

   on Jan 26, 2026

   More actions

   Hi @boudica-dev-eng,

   Thanks for the detailed report.

   To help narrow this down: do you know whether the ts column is dictionary-encoded in the underlying Parquet files (e.g., parquet.enable.dictionary=true)?

   Also, does the error disappear if dictionary encoding is disabled when writing the Iceberg table?
3. boudica-dev-eng commented on Jan 26, 2026

   

   boudica-dev-eng

   on Jan 26, 2026

   Author

   More actions

   I've not explicitly enabled dictionary encoding via that config flag anywhere in the global or table configs.
4. vigneshsiva11 commented on Jan 26, 2026

   

   vigneshsiva11

   on Jan 26, 2026

   More actions

   Thanks for confirming @boudica-dev-eng
   For reference, this behavior is addressed in PR #3278, which ensures datediff (and similar date functions) handle dictionary-backed inputs safely by aligning array lengths before evaluation.

   Please let me know if you’d prefer this logic to live in a shared utility or be handled at a different layer.
5. andygrove

   added

   crashNative engine crash/panic/segfaultNative engine crash/panic/segfault

   on Feb 9, 2026
6. andygrove

   added

   priority:highCrashes, panics, segfaults, major functional breakageCrashes, panics, segfaults, major functional breakage

   on Mar 27, 2026
7. mbutrovich

   self-assigned this

   on Apr 2, 2026
8. mbutrovich commented on Jun 10, 2026

   

   mbutrovich

   on Jun 10, 2026

   Contributor

   More actions

   I wasn't able to reproduce this with the current Iceberg code path. I suspect this performed the scan in Iceberg Java, and then somehow they made their way over to Comet without unpacking the dictionaries. I wrote a reproducer and couldn't get it to fail. Please reopen if you re-encounter the issue. Thanks for the report!

     // https://github.com/apache/datafusion-comet/issues/3255
     // Reported as CometNativeException "Cannot perform binary operation on arrays of
     // different length" when applying to_date / datediff to a timestamp column read from
     // an Iceberg table. The suspected trigger is dictionary-encoded timestamp values, so
     // we insert many rows with few distinct timestamps to encourage dictionary encoding.
     test("to_date and datediff on Iceberg timestamp column - reproduces #3255") {
       assume(icebergAvailable, "Iceberg not available in classpath")
   
       withTempIcebergDir { warehouseDir =>
         withSQLConf(
           "spark.sql.catalog.test_cat" -> "org.apache.iceberg.spark.SparkCatalog",
           "spark.sql.catalog.test_cat.type" -> "hadoop",
           "spark.sql.catalog.test_cat.warehouse" -> warehouseDir.getAbsolutePath,
           CometConf.COMET_ENABLED.key -> "true",
           CometConf.COMET_EXEC_ENABLED.key -> "true",
           CometConf.COMET_ICEBERG_NATIVE_ENABLED.key -> "true") {
   
           // Spark TIMESTAMP maps to Iceberg timestamptz (TIMESTAMPTZ), matching the report.
           spark.sql("""
             CREATE TABLE test_cat.db.ts_date_test (
               id STRING,
               ts TIMESTAMP
             ) USING iceberg
           """)
   
           // 1000 rows over only 5 distinct timestamps -> highly compressible / dictionary-friendly.
           spark.sql("""
             INSERT INTO test_cat.db.ts_date_test
             SELECT
               CAST(id AS STRING) as id,
               CAST(CONCAT('2024-01-0', CAST(id % 5 + 1 AS STRING), ' 12:00:00') AS TIMESTAMP) as ts
             FROM range(1000)
           """)
   
           // Scalar (current_date / literal) vs dictionary-backed column: the reported crash.
           checkIcebergNativeScan("SELECT id, to_date(ts) FROM test_cat.db.ts_date_test ORDER BY id")
           checkIcebergNativeScan(
             "SELECT id, datediff(DATE '2025-01-01', ts) FROM test_cat.db.ts_date_test ORDER BY id")
           checkIcebergNativeScan(
             "SELECT COUNT(*) FROM test_cat.db.ts_date_test WHERE datediff(DATE '2025-01-01', ts) > 0")
   
           spark.sql("DROP TABLE test_cat.db.ts_date_test")
         }
       }
     }
9. mbutrovich

   closed this as completedon Jun 10, 2026
10. andygrove

    added a commit that references this issue on Oct 17, 2026

    feat: bump iceberg-rust past apache#3255 and narrow the nested schema…

    ...

    1c51487