[native_datafusion] Enable native_datafusion in auto scan mode — test failure tracking #3321

Activity

1. andygrove commented on Mar 13, 2026

   

   andygrove

   on Mar 13, 2026

   MemberAuthor

   More actions

   Once the above issues are all resolved, we can make this code change to CometScanRule:

   COMET_NATIVE_SCAN_IMPL.get() match {
             case SCAN_AUTO =>
               nativeDataFusionScan(plan, session, scanExec, r, hadoopConf)
                 .orElse {
                   // Clear explain info tags from the failed nativeDataFusionScan
                   // attempt so they don't leak into the fallback path
                   scanExec.unsetTagValue(CometExplainInfo.EXTENSION_INFO)
                   nativeIcebergCompatScan(session, scanExec, r, hadoopConf)
                 }
                 .getOrElse(scanExec)
2. andygrove

   self-assigned this

   on Mar 14, 2026
3. andygrove

   added this to the 0.15.0 milestone on Mar 14, 2026
4. andygrove commented on Mar 17, 2026

   

   andygrove

   on Mar 17, 2026

   MemberAuthor

   More actions

   Current status:

   #3311 — Schema/type mismatch errors (7 tests)

   • Spark native readers should respect spark.sql.caseSensitive (FileBasedDataSourceSuite)
   • SPARK-35640: read binary as timestamp should throw schema incompatible error (ParquetSchemaSuite)
   • SPARK-35640: int as long should throw schema incompatible error (ParquetSchemaSuite)
   • SPARK-36182: can't read TimestampLTZ as TimestampNTZ (ParquetSchemaSuite)
   • SPARK-34212 Parquet should read decimals correctly (ParquetSchemaSuite)
   • row group skipping doesn't overflow when reading into larger type (ParquetSchemaSuite)
   • schema mismatch failure error message for parquet vectorized reader (ParquetSchemaSuite)
   • SPARK-45604: schema mismatch failure error on timestamp_ntz to array<timestamp_ntz> (ParquetSchemaSuite)

   #3320 — Parquet filter pushdown behavior (2 tests)

   • SPARK-31026: Parquet predicate pushdown for fields having dots in the names (ParquetFilterSuite)
   • Filters should be pushed down for Parquet readers at row group level (ParquetFilterSuite)

   #3401 — Self-union streaming reads (2 tests)

   • self-union, DSv1, read via DataStreamReader API (StreamSuite)
   • self-union, DSv1, read via table API (StreamSuite)
5. andygrove commented on Mar 27, 2026

   

   andygrove

   on Mar 27, 2026

   MemberAuthor

   More actions

   also need to look at:

   -  test("filter pushdown - StringPredicate") {
   +  test("filter pushdown - StringPredicate",
   +    IgnoreCometNativeDataFusion("cannot be pushed down")) {
   

   need to file separate issue
6. andygrove commented on Mar 30, 2026

   

   andygrove

   on Mar 30, 2026

   MemberAuthor

   More actions

   also need to look at:

   -  test("filter pushdown - StringPredicate") {
   +  test("filter pushdown - StringPredicate",
   +    IgnoreCometNativeDataFusion("cannot be pushed down")) {
   

   need to file separate issue

   PR to fix this issue - #3830
7. andygrove

   added a commit that references this issue on Apr 1, 2026

   fix: add tracking issue link to IgnoreCometNativeScan for StringPredi…

   ...

   209885c
8. andygrove commented on Apr 23, 2026

   

   andygrove

   on Apr 23, 2026

   MemberAuthor

   More actions

   There are no more references to this epic from Spark SQL tests - more specific issues are now tagged instead, so closing this.
9. andygrove

   closed this as completedon Apr 23, 2026