Bug triage results: 2026-10-12 #6892

Triage pass over the open requires-triage queue, per the project Bug Triage Guide.

• Date: 2026-10-12
• Total issues processed: 89 (84 triaged, 5 skipped, 0 failed)
• Type counts: 49 bugs, 35 enhancements
• Priority counts applied: priority:critical 18, priority:high 0, priority:medium 25, priority:low 6
• Guide: docs/source/contributor-guide/bug_triage.md

Labels have already been applied. A reviewer should spot-check the calls below and close this issue when satisfied. Corrections should be made directly on the affected issue.

Notes on this pass:

• Reviewer corrections since the last pass. No type or priority label from the 2026-10-05 pass has changed since. correctness was added to five of its critical bugs (Native sort orders null elements of array and struct keys by the key's null order, unlike Spark #6476, RANGE window frames over an array or struct key with a null element span the whole partition #6477, percentile_approx returns different percentiles from Spark when its input holds a NaN with the sign bit set #6519, Codegen dispatcher initializes kernels with an incorrect partition index under UNION ALL and coalesce #6570 and input_file_name() returns empty values above a converted Spark Parquet scan #6573). The label isn't in the guide, so this pass doesn't add it. The critical bugs below that don't carry it are Native string to float/double and decimal casts accept strings Spark rejects #6839, from_csv: wrong results in permissive modes and failure with a variant schema #6800, Native hash joins lose the streamed side's order when it arrives sorted from outside the native plan #6787, Broadcast hash join produces incorrect results with sliced Arrow string/binary batches #6779, arrays_overlap keeps -0.0 and 0.0 apart on Spark 4.0.5, 4.1.4 and 4.2.1, which merge them (SPARK-59602) #6768, IN treats a column-dependent CASE WHEN / IF candidate as the constant NULL #6762, least orders a null inside a struct or array last, where Spark orders it first #6759, greatest/least compare struct arguments by field name instead of position when names differ only in case #6758, Native CSV scan reads every partition through the first partition's object store #6747, Native Parquet scan reads files from another bucket when one partition spans buckets #6746, arrays_overlap returns wrong answers or panics with a constant array beside a column #6742, Broadcast hash join on string keys drops rows when the build side is a large DISTINCT #6718 and Identical nondeterministic expressions share one codegen dispatcher kernel and continue each other's state #6711.
• Two type reclassifications. [followup] Broadcast build-side auto-bridge only accelerates the innermost join when a fact has multiple unsupported-format dimensions #6832 arrived as bug and is now enhancement: its outer joins run on Spark, as they would without the opt-in feature, and the issue says correctness is unaffected. Native fanout Iceberg writes fail when their open partitions outgrow the memory pool #6771 arrived as enhancement and is now a priority:medium bug, because the native fanout write fails its task where iceberg-java's writer succeeds. Either can be flipped back with one edit. The 29 issues that arrived without a type label were classified from their bodies.
• One author priority was changed. Native timezone conversions stop applying DST after 2099, while Spark keeps applying the zone's rules #6816 went from priority:low to priority:medium (see the escalations). The author's priority:low was confirmed on SQLAppStatusListenerMemoryLeakSuite "no memory leak" is flaky in the Spark SQL tests #6890. Three enhancements arrived with an author's priority: Add a GroupsAccumulator for the native regr_* aggregates #6886 (priority:medium), Check the timezone label of every native expression that returns a timestamp #6849 and Vectorize the lower-bound check in native date_trunc for fine units #6717 (priority:low). As in earlier passes, these were left in place, and the reviewer may want to remove them.
• Spark SQL test findings from draft feat: enable LocalTableScanExec acceleration by default #6780. test: Spark SQL tests that inspect the physical plan fail when LocalTableScanExec is native #6795 to CometIcebergWriteDetectionSuite assumes a split Iceberg write plan #6803 come from turning on spark.comet.exec.localTableScan.enabled, which is off by default. The three whose failures are only in the tests (test: Spark SQL tests that inspect the physical plan fail when LocalTableScanExec is native #6795, collect_set / collect_list return elements in a different order than Spark #6796 and CometIcebergWriteDetectionSuite assumes a split Iceberg write plan #6803) are low. The others are labeled by the product behavior they report: percentile of -0.0 returns 0.0 #6799 and from_csv: wrong results in permissive modes and failure with a variant schema #6800 critical, and Math functions differ from Spark in the last digits #6797, Native errors surface as SparkException where Spark raises a specific error class #6798, GROUP BY on a struct containing a map fails with CometNativeException #6801 and Null-named attribute fails Arrow export with "field name cannot be null" #6802 medium.
• Likely duplicate. Broadcast hash join on string keys drops rows when the build side is a large DISTINCT #6718 and Broadcast hash join produces incorrect results with sliced Arrow string/binary batches #6779 are very likely one defect. Both are from unikdahal, both drop rows from a broadcast join whose build side is a DISTINCT over string keys, and Broadcast hash join produces incorrect results with sliced Arrow string/binary batches #6779 traces it to sliced string vectors in Utils.serializeBatches and coalesceBroadcastBatches. Both are labeled, and the reviewer may want to close Broadcast hash join on string keys drops rows when the build side is a large DISTINCT #6718 in favor of Broadcast hash join produces incorrect results with sliced Arrow string/binary batches #6779.
• The guide's area table still lacks area:joins, area:Iceberg, area:udf and area:memory, so this pass added none of them. Issues that would have taken one and don't already carry it: area:joins for Broadcast hash join on string keys drops rows when the build side is a large DISTINCT #6718, Broadcast hash join produces incorrect results with sliced Arrow string/binary batches #6779, Native hash joins lose the streamed side's order when it arrives sorted from outside the native plan #6787 and [followup] Broadcast build-side auto-bridge only accelerates the innermost join when a fact has multiple unsupported-format dimensions #6832, area:Iceberg for Iceberg data tables whose names end with a metadata-table name fall back to Spark #6719 and Design question: long-term, should native Iceberg scans run through DataFusion's ParquetSource instead of iceberg-rust's ArrowReader? #6737, area:udf for Identical nondeterministic expressions share one codegen dispatcher kernel and continue each other's state #6711 and Interrupt a native scalar Arrow UDF when its task is killed #6811, and area:memory for Native PartialMerge over input sorted on a prefix of its keys fails to spill on DataFusion 55.2 #6880.
• The guide lists spark 4 as an area indicator, but the repository only has spark 3.x, spark 4.0, spark 4.1 and spark 4.2, so nothing was applied to arrays_overlap keeps -0.0 and 0.0 apart on Spark 4.0.5, 4.1.4 and 4.2.1, which merge them (SPARK-59602) #6768 and Native sequence raises "Unreachable code reached." where Spark 4.0.5, 4.1.4 and 4.2.1 return the sequence (SPARK-58821) #6769, which affect only Spark 4 releases.
• regression is not in the guide, so this pass didn't add it. It fits the IF and nullif shapes of IN treats a column-dependent CASE WHEN / IF candidate as the constant NULL #6762, which come from the eager CASE path in perf: optimize native CASE WHEN and IF (up to 12x faster) #6350, merged after 1.1.0.
• Spark 3.4: global aggregate over a single-split Iceberg scan returns no row when DPP prunes the split #6667 was closed while this pass was running and isn't included. Rename Cast.is_spark4_plus and get_json_object_spark34 to describe the behavior they select #6891 was opened during the pass and is included.

Bugs

priority:critical

• [EPIC] In-memory cache audit: correctness and performance follow-ups for InMemoryTableScanExec support (#6834)
  • Area labels: area:scan; also carries performance, EPIC, correctness
  • Rationale: Its first open item returns wrong results with default settings on Spark 3.5+, where the cache is on by default: a sorted cache under a native hash join loses its order (the root cause of Native hash joins lose the streamed side's order when it arrives sorted from outside the native plan #6787, fix in fix: keep a streamed side sorted outside the native plan in order through outer hash joins #6835). An EPIC of bugs takes the priority of its worst open item, as [EPIC] Timezone handling bugs #6335 and [EPIC] Match Spark's -0.0 and NaN semantics systematically instead of per expression #6385 did, so step 1 (see escalations).
• Native string to float/double and decimal casts accept strings Spark rejects (#6839)
  • Area labels: area:expressions
  • Rationale: CometCast reports string to FLOAT, DOUBLE and DECIMAL as Compatible, so with default configs 'NaNF', '-nan' and '1e-2147483648' cast to NaN, infinity or 0.00 where Spark returns NULL, or raises CAST_INVALID_INPUT under ANSI. Incorrect cast output is the guide's own step 1 example. Fix in fix: do not fail planning on a cast of a literal that fails to evaluate #6826.
• Native Iceberg scan returns wrong values for a column renamed after a VERSION AS OF snapshot (#6822)
  • Area labels: area:scan; also carries correctness, area:Iceberg
  • Rationale: A VERSION AS OF read of a snapshot from before a rename returns NULL, or the other column's values after a swap, from the native Iceberg scan, which is on by default. Silent wrong results at step 1, on branch-1.1 too.
• from_csv: wrong results in permissive modes and failure with a variant schema (#6800)
  • Area labels: area:expressions, spark sql tests; also carries expression
  • Rationale: The report says results differ from Spark in permissive modes, which is step 1. It was seen only with localTableScan on, but from_csv goes through the codegen dispatcher, which is on by default, so a native projection over another source may reach it too (see escalations).
• percentile of -0.0 returns 0.0 (#6799)
  • Area labels: area:aggregation, spark sql tests; also carries correctness
  • Rationale: CometPercentile reports Compatible for numeric input, so the native percentile that returns 0.0 where Spark returns -0.0 runs by default. Step 1, the same class as percentile_approx returns different percentiles from Spark when its input holds a NaN with the sign bit set #6519 and Native map construction doesn't match Spark 4.0+ float key normalization (-0.0 keys, missing DUPLICATED_MAP_KEY) #6549 under [EPIC] Match Spark's -0.0 and NaN semantics systematically instead of per expression #6385.
• Native Iceberg scan returns nested values that Spark reads as NULL in files without field ids or a name mapping (#6794)
  • Area labels: area:scan; also carries correctness, area:Iceberg
  • Rationale: For data files with no field ids and no name mapping, the native Iceberg scan returns the stored nested values where Spark returns NULL, so filters and counts differ (count(*) WHERE s.a IS NULL is 50 in Spark and 0 in Comet). Step 1 on the default native scan (see escalations).
• Native Iceberg scan returns NULL for nested fields of migrated tables that use a name mapping (#6790)
  • Area labels: area:scan; also carries correctness, area:Iceberg
  • Rationale: Tables migrated with migrate, snapshot or add_files get a name mapping, and the native Iceberg scan, on by default, returns NULL for their nested fields whenever the query also reads a top-level primitive. Silent wrong results at step 1, and 1.1.0 has it too.
• Native hash joins lose the streamed side's order when it arrives sorted from outside the native plan (#6787)
  • Area labels: none
  • Rationale: Spark drops a sort above the join as redundant, and the native join appends the unmatched probe rows after the matched ones, so sortWithinPartitions returns unsorted partitions with no error. [EPIC] In-memory cache audit: correctness and performance follow-ups for InMemoryTableScanExec support #6834 reproduces it with default settings through the native cache scan, and forceShuffledHashJoin leaves sortWithinPartitions output unsorted and fails partitioned inserts #6770, the same class, is critical. Fix in fix: keep a streamed side sorted outside the native plan in order through outer hash joins #6835.
• Broadcast hash join produces incorrect results with sliced Arrow string/binary batches (#6779)
  • Area labels: area:ffi
  • Rationale: A broadcast build side made of sliced string or binary vectors is serialized with its unused buffer prefix, so join keys are corrupted and matching rows are dropped with no error. That matches the guide's step 1 example of FFI data corruption from arrays with a non-zero offset.
• arrays_overlap keeps -0.0 and 0.0 apart on Spark 4.0.5, 4.1.4 and 4.2.1, which merge them (SPARK-59602) (#6768)
  • Area labels: area:expressions
  • Rationale: On those Spark releases arrays_overlap returns false where Spark returns true for signed zeros, with default configs. Step 1, under [EPIC] Match Spark's -0.0 and NaN semantics systematically instead of per expression #6385 (see escalations).
• IN treats a column-dependent CASE WHEN / IF candidate as the constant NULL (#6762)
  • Area labels: area:expressions
  • Rationale: IN returns NULL for every row, with no error, when a candidate is an IF, CASE WHEN or nullif that reads a column. The reproduction runs over range, but the constant check is in the native IN itself, so a native projection over a Parquet scan takes the same path by default. Step 1. Fix in fix: return an empty array from CASE WHEN for an empty batch #6763.
• least orders a null inside a struct or array last, where Spark orders it first (#6759)
  • Area labels: area:expressions
  • Rationale: least over structs or arrays returns a different value from Spark when a nested value is null, because the DataFusion comparator it uses puts nulls last. The plan is native and raises nothing, so step 1. Fix in fix: order nulls inside nested values first in least #6761.
• greatest/least compare struct arguments by field name instead of position when names differ only in case (#6758)
  • Area labels: area:expressions
  • Rationale: With the default spark.sql.caseSensitive=false, struct arguments whose field names differ only in case are matched by name instead of by position, so greatest and least return a different struct from Spark. Step 1. Fix in fix: compare greatest and least struct arguments by position #6760.
• Native CSV scan reads every partition through the first partition's object store (#6747)
  • Area labels: area:scan
  • Rationale: A native CSV scan over two buckets returns the first bucket's rows twice, with no error, when the keys match. That is step 1 on the opt-in native CSV scan, as input_file_name() returns empty values above a converted Spark Parquet scan #6573 was on its opt-in path (see escalations). Fix in fix: read each native scan partition through the object store its files live in #6753.
• Native Parquet scan reads files from another bucket when one partition spans buckets (#6746)
  • Area labels: area:scan
  • Rationale: When Spark packs small files from two buckets into one partition, the default native Parquet scan fetches the second bucket's keys from the first and returns duplicated rows with no error. Step 1. Fix in fix: read each native scan partition through the object store its files live in #6753.
• arrays_overlap returns wrong answers or panics with a constant array beside a column (#6742)
  • Area labels: area:expressions
  • Rationale: With default configs, a constant first argument compares only row 0 and uses that answer for the whole batch, and a constant second argument panics. The wrong answers make it step 1. Fix in fix: compare every row when arrays_overlap gets a scalar beside a column #6743.
• Broadcast hash join on string keys drops rows when the build side is a large DISTINCT (#6718)
  • Area labels: area:ffi
  • Rationale: About 0.012% of the matching rows go missing with no error, which is step 1. It is very likely the defect Broadcast hash join produces incorrect results with sliced Arrow string/binary batches #6779 traces to sliced string vectors in the broadcast path (see the notes).
• Identical nondeterministic expressions share one codegen dispatcher kernel and continue each other's state (#6711)
  • Area labels: area:expressions
  • Rationale: Two identical dispatched subtrees share one kernel and one counter, so the second monotonically_increasing_id() continues the first one's sequence. The dispatcher is on by default, so this is silent wrong results at step 1, like Codegen dispatcher initializes kernels with an incorrect partition index under UNION ALL and coalesce #6570. Fix in fix: give each nondeterministic dispatched expression its own kernel #6712.

priority:medium

• Native PartialMerge over input sorted on a prefix of its keys fails to spill on DataFusion 55.2 (#6880)
  • Area labels: area:aggregation
  • Rationale: Since fix: keep one row per key in a native PartialMerge that runs short of memory #6867, this shape fails the task under memory pressure instead of returning the wrong count from Native PartialMerge aggregate emits duplicate keys under memory pressure, so count(DISTINCT) over-counts #6857. The failure is visible and turning off the native aggregate avoids it, so step 3, like Native aggregate fails after spilling when groups are few and large (collect_list, collect_set) #6363.
• Native date functions and date-to-string cast return NULL or fail for dates outside chrono's NaiveDate range (#6843)
  • Area labels: area:expressions
  • Rationale: year, month, trunc, next_day and others return NULL, and cast(d AS string) and last_day fail, for dates beyond year 262142. Held at medium like Native make_date rejects Spark-valid years outside chrono range #5208 and Native Iceberg write panics or writes a NULL partition value for dates and timestamps beyond year 262143 #6145, since real data doesn't reach those years (see escalations).
• Native cast string to date returns NULL for years outside chrono's NaiveDate range (#6842)
  • Area labels: area:expressions
  • Rationale: CAST, TRY_CAST and to_date return NULL for years beyond chrono's range, even under ANSI. The cast's support note documents the range, and it is held at medium like Native make_date rejects Spark-valid years outside chrono range #5208 and Native Iceberg write panics or writes a NULL partition value for dates and timestamps beyond year 262143 #6145 (see escalations). Fix in fix: cast string to date for years outside chrono's NaiveDate range #6844.
• CometScanRule reads fs.comet.libhdfs.schemes from SQLConf, but native reads it from the Hadoop conf (#6837)
  • Area labels: area:scan
  • Rationale: A list set with SET spark.hadoop.fs.comet.libhdfs.schemes makes Comet claim a scan that then fails at execution, and one set only in core-site.xml makes the scan fall back. A visible failure in an uncommon setup, with a workaround (set it at startup), so step 3. Fix in fix: read fs.comet.libhdfs.schemes for the scan rule from the Hadoop conf #6838.
• Cast of a literal that fails under ANSI fails the query at planning time, even in a branch no row reaches (#6825)
  • Area labels: area:expressions
  • Rationale: Under ANSI, the default on Spark 4, CometCast evaluates a literal cast while planning and fails a query that Spark completes because no row reaches that branch. A visible failure with workarounds, step 3, like Array functions evaluate their second argument on rows where the array is NULL, raising errors that Spark skips #6613. Fix in fix: do not fail planning on a cast of a literal that fails to evaluate #6826.
• IN evaluates later candidates over the whole batch, so an ANSI error can fire for a row that already matched (#6823)
  • Area labels: area:expressions
  • Rationale: Under ANSI, Comet raises for a row that an earlier IN candidate already matched, where Spark returns a result. A visible failure, step 3, like Array functions evaluate their second argument on rows where the array is NULL, raising errors that Spark skips #6613, and one instance of Track evaluation masks for data-dependent errors beyond unbase64 operator fallback #6006.
• Native timezone conversions stop applying DST after 2099, while Spark keeps applying the zone's rules (#6816)
  • Area labels: area:expressions; also carries correctness
  • Rationale: After about 2100, hour, casts to string or date, and date_trunc in a DST zone can be off by the DST offset, and the first two run natively by default. That is silent wrong results, documented in the compatibility guide. Raised from the author's priority:low to medium, where this pass holds the other far-from-present date issues (see escalations).
• Non-AQE DPP subquery doesn't reuse the Comet broadcast when the dimension is a UNION (#6815)
  • Area labels: none; also carries performance, area:joins
  • Rationale: Results are correct, but with AQE off the DPP subquery computes and broadcasts a UNION ALL dimension a second time instead of reusing the join's broadcast. Lost exchange reuse is step 3, as it was for Spark 3.4 SQL test SPARK-34637 fails since #6547 because AQE re-plans flip the DPP join's build side #6645.
• ShuffleScan metrics are dropped under AQE: shuffle read and decode time missing on the direct-read path (#6804)
  • Area labels: area:shuffle
  • Rationale: With AQE and direct read, both on by default, the shuffle read and decode time never reach any SQL metric. Results are unaffected, but the metrics are broken, so step 3 (see escalations).
• Null-named attribute fails Arrow export with "field name cannot be null" (#6802)
  • Area labels: area:ffi, spark sql tests
  • Rationale: An NPE in Arrow export fails the query, but it needs the opt-in localTableScan and a null-named attribute (SPARK-57725). A visible failure on an opt-in path, step 3 (see escalations).
• GROUP BY on a struct containing a map fails with CometNativeException (#6801)
  • Area labels: spark sql tests
  • Rationale: The query fails with CometNativeException where Spark succeeds. It was seen only with the opt-in localTableScan and isn't diagnosed yet, so step 3 for a visible failure (see escalations).
• Native errors surface as SparkException where Spark raises a specific error class (#6798)
  • Area labels: area:expressions, spark sql tests; also carries correctness
  • Rationale: Comet raises where Spark raises, but as a generic SparkException instead of Spark's error class. Only the error differs, the tier of ANSI integral SUM overflow reports "integer overflow" without Spark's try_add suggestion #6661 and ANSI integral overflow errors: wrong error class for Byte/Short arithmetic and missing try_ suggestion #6217, so step 3.
• Math functions differ from Spark in the last digits (#6797)
  • Area labels: area:expressions, spark sql tests; also carries expression, correctness
  • Rationale: Native asinh, cosh, tan, cot, atanh, cbrt, exp, pow and atan2 run by default and return values that differ from Spark in the last digits. Held at medium because the values differ only in floating-point precision (see escalations).
• Native Iceberg scan fails with field not found when an equality delete is keyed on a nested field (#6782)
  • Area labels: area:scan; also carries area:Iceberg
  • Rationale: Every native Iceberg query on such a table fails where Spark answers. A visible failure with a workaround (turning off the native Iceberg scan), so step 3.
• Native Parquet writer starts a new row group for every batch when writing to HDFS (#6774)
  • Area labels: area:writer; also carries performance
  • Rationale: Files written natively to HDFS get a row group every 8,192 rows instead of up to 1,048,576, so readers do more work. The data is right, so this is performance degradation on the off-by-default writer, step 3.
• Native fanout Iceberg writes fail when their open partitions outgrow the memory pool (#6771)
  • Area labels: area:writer; also carries area:Iceberg
  • Rationale: The task fails with Additional allocation failed for IcebergWriteExec where iceberg-java's fanout writer succeeds, and the retry fails the same way. A failure is a bug under the guide's type table, even on the off-by-default writer, so this is reclassified from the author's enhancement to a step 3 bug. Fix in feat: close fanout partitions early when the memory pool refuses the native Iceberg writer #6773.
• Native sequence raises "Unreachable code reached." where Spark 4.0.5, 4.1.4 and 4.2.1 return the sequence (SPARK-58821) (#6769)
  • Area labels: area:expressions
  • Rationale: On Spark 4.0.5, 4.1.4 and 4.2.1, which include SPARK-58821, Comet fails a query that Spark answers. A visible failure on edge-case inputs, step 3.
• spark.comet.expression.<name>.enabled has no effect on expressions inside a codegen-dispatched tree (#6756)
  • Area labels: area:expressions; also carries area:udf
  • Rationale: The flag the user guide documents for forcing a Spark fallback has no effect inside a dispatched tree. Results match Spark, so this is a broken feature with workarounds, step 3.
• Native Parquet writes on Spark 3.4/3.5 ignore maxRecordsPerFile (#6751)
  • Area labels: area:writer
  • Rationale: With the native writer opted in on Spark 3.4 and 3.5, Comet writes one file per task where Spark rolls a new file every N rows. The rows are right and the writer needs allowIncompatible, so step 3, as for Native Parquet writes on Spark 3.4/3.5 leave INSERT INTO targets stale until REFRESH TABLE #6315.
• [Bug] Native make_date mishandles Date32 overflow and skipped evaluation (#6741)
  • Area labels: area:expressions
  • Rationale: For a year past Date32 (6,000,000 in the report), make_date returns NULL where Spark raises ARITHMETIC_OVERFLOW, held at medium like Native make_date rejects Spark-valid years outside chrono range #5208. The NULL-skip and LIMIT items are errors where Spark returns a result, step 3 like Array functions evaluate their second argument on rows where the array is NULL, raising errors that Spark skips #6613, and come from reading the code (see escalations).
• Native mode treats all NaNs as one value on Spark 3.4, where Spark counts each NaN separately (#6738)
  • Area labels: area:aggregation
  • Rationale: Only with spark.comet.expression.Mode.allowIncompatible=true, Comet returns a different mode from Spark 3.4, 3.5.0 and 3.5.1. Wrong results behind an Incompatible opt-in follow the [Bug] Native make_interval overflows valid time components or loses seconds precision #5131 rule at step 3 (see escalations).
• Iceberg data tables whose names end with a metadata-table name fall back to Spark (#6719)
  • Area labels: area:scan
  • Rationale: An ordinary Iceberg table named, say, user_files or order_history falls back with a misleading reason, because the metadata-table check matches the end of the name. Results are correct, but those tables lose the native scan, so step 3.
• Cached AQE queries fail after transition reversion on Spark 4.0+ (#6687)
  • Area labels: none
  • Rationale: A cached AQE query fails with FilterExec has column support mismatch, but only with the opt-in spark.comet.exec.transitionRevert.enabled=true. A visible failure with a workaround, step 3. Fix in fix: preserve columnar output when reverting a cached AQE result stage #6852.
• TakeOrderedAndProject fails with "Subquery not found" when the sort order has a scalar subquery and the input has several partitions (#6676)
  • Area labels: none
  • Rationale: With default configs, a scalar subquery in the sort keys over a multi-partition input fails the query where Spark returns rows. A visible failure with a workaround, step 3.
• Codegen dispatcher fails with "Subquery has not finished" when a scalar subquery is inside the dispatched expression (#6674)
  • Area labels: area:expressions; also carries area:udf
  • Rationale: With the dispatcher on by default, a scalar subquery inside a Scala UDF's argument, or inside to_json on Spark 4, fails the task where Spark succeeds, on 1.0 and 1.1 as well. A visible failure with a workaround, step 3.

priority:low

• SQLAppStatusListenerMemoryLeakSuite "no memory leak" is flaky in the Spark SQL tests (#6890)
  • Area labels: area:ci, spark sql tests; also carries test
  • Rationale: A CI flake caused by a race in the Spark test itself (2 of 252 runs), which upstream fixed in SPARK-57711. Step 4, confirming the author's label. It hasn't evicted a PR from the merge queue, so the escalation trigger doesn't apply yet.
• S3 test suites can no longer pull the minio/minio image (#6884)
  • Area labels: area:ci
  • Rationale: Docker Hub now refuses minio/minio, so every suite on CometS3TestBase fails in beforeAll. Test-only, step 4, and the PR workflows don't run these suites. Fix in test: run the S3 suites against an available MinIO image #6887.
• CometIcebergWriteDetectionSuite assumes a split Iceberg write plan (#6803)
  • Area labels: area:writer, spark sql tests; also carries area:Iceberg
  • Rationale: Five tests fail only once localTableScan is on by default, as in draft feat: enable LocalTableScanExec acceleration by default #6780, because the write becomes fully native. Test-only, step 4.
• collect_set / collect_list return elements in a different order than Spark (#6796)
  • Area labels: area:aggregation, spark sql tests
  • Rationale: Spark documents the order of collect_list and collect_set results as non-deterministic, so the failing tests depend on an order Spark doesn't promise. Test-only, step 4.
• test: Spark SQL tests that inspect the physical plan fail when LocalTableScanExec is native (#6795)
  • Area labels: spark sql tests; also carries test
  • Rationale: About 40 upstream tests assert on Spark nodes that become Comet nodes once localTableScan is on by default. Test-only, step 4.
• Spark SQL Writer Tests (manual) never runs Comet's native writer (#6749)
  • Area labels: area:writer, area:ci
  • Rationale: The manual workflow never turns on the native writer, so a green run tests Spark's writer. CI tooling, step 4. Its second finding, that the documented environment variable for allowIncompatible is ignored, is a minor config defect with an easy workaround.

Enhancements

• Rename Cast.is_spark4_plus and get_json_object_spark34 to describe the behavior they select (#6891)
  • Area labels: area:expressions
  • Rationale: A rename with no behavior change, a refactor under the guide's type table. Confirms the author's label.
• Write blog post for 1.2.0 release (#6889)
  • Area labels: none
  • Rationale: Release work under Comet 1.2.0 Release #6550, like the 1.1.0 post (Write blog post for 1.1.0 release #6120). Confirms the author's label.
• Add a GroupsAccumulator for the native regr_* aggregates (#6886)
  • Area labels: area:aggregation; also carries performance
  • Rationale: A performance optimization. The slowdown it warns about would only follow if fix: match Spark statistical aggregate updates and merges #6076 lands first, so nothing is broken on main. The author's priority:medium was left in place.
• Native Iceberg writes: count the DataFiles a task keeps in its memory reservation (#6885)
  • Area labels: area:writer; also carries area:Iceberg, area:memory
  • Rationale: A memory accounting improvement for the native Iceberg writer, with nothing failing. Confirms the author's label.
• [EPIC] Nested-type conversion performance: C2R, R2C and JVM shuffle slower than Spark (#6855)
  • Area labels: area:shuffle, area:scan; also carries performance, EPIC
  • Rationale: A performance EPIC. Every item is a gap against Spark or a benchmark follow-up, with correct results.
• Push Iceberg residual predicates on decimal columns into the native scan (#6854)
  • Area labels: area:scan; also carries performance, area:Iceberg
  • Rationale: Results are correct, and this adds row group and page pruning on decimal columns.
• Follow-ups to native NULL short-circuiting of array and map functions (#6850)
  • Area labels: area:expressions
  • Rationale: Follow-up cleanups to fix: skip an array or map function's later arguments where the array or map is NULL #6716. The one behavior difference it lists needs spark.sql.subexpressionElimination.enabled=false. Confirms the author's label.
• Check the timezone label of every native expression that returns a timestamp (#6849)
  • Area labels: area:expressions; also carries test
  • Rationale: New test coverage. test: fail Comet tests when a native plan mislabels a timestamp's timezone #6868, merged today, checks labels where a native plan hands batches to the JVM, and these tests would also catch a label consumed inside one plan. The author's priority:low was left in place.
• [followup] Broadcast build-side auto-bridge only accelerates the innermost join when a fact has multiple unsupported-format dimensions (#6832)
  • Area labels: none
  • Rationale: The outer joins run on Spark, as they do without the feature, and the issue says correctness is unaffected, so this is a missed optimization in the opt-in bridge from feat: run broadcast joins natively when an unsupported scan is on the build side  #6786. Reclassified from the author's bug (see the notes).
• Native fanout Iceberg writes: sort the remaining rows by partition when the memory pool runs short, instead of writing small files (#6818)
  • Area labels: area:writer; also carries area:Iceberg, area:memory
  • Rationale: An improvement on the early closes in feat: close fanout partitions early when the memory pool refuses the native Iceberg writer #6773, which already let the write finish. Confirms the author's label.
• Interrupt a native scalar Arrow UDF when its task is killed (#6811)
  • Area labels: none
  • Rationale: A follow-up to the unmerged feat: execute scalar Arrow Python UDFs in native Comet #6130, for a path that isn't on main yet. Confirms the author's label.
• perf: without AQE, native operators read Comet shuffle through a JVM round trip per block, making small reduce tasks up to 4x slower than Spark (#6807)
  • Area labels: area:shuffle, area:ffi; also carries performance
  • Rationale: A performance gap against Spark with correct results and no regression identified, like perf: Comet hash joins use more task time than Spark (skewed SHJ with nested payload, BHJ) #6528. Confirms the author's label.
• perf: every Comet broadcast hash join probe task re-decodes the whole broadcast in the JVM (#6806)
  • Area labels: none; also carries performance, area:joins
  • Rationale: A performance optimization of the broadcast decode path, related to perf: broadcast hash join rebuilds the build-side hash table in every task #6013. Confirms the author's label.
• Draft the Apache Comet proposal and board resolution (#6791)
  • Area labels: none
  • Rationale: Project governance work under [EPIC] Promote Comet to a top level Apache project #6793, labeled like the planning trackers in earlier passes. Confirms the author's label.
• Agree on the scope and repositories for Apache Comet (#6789)
  • Area labels: none
  • Rationale: Project governance work under [EPIC] Promote Comet to a top level Apache project #6793. Confirms the author's label.
• Agree on the initial PMC, committers, and chair for Apache Comet (#6788)
  • Area labels: none
  • Rationale: Project governance work under [EPIC] Promote Comet to a top level Apache project #6793. Confirms the author's label.
• perf: divide once per row in pmod and div_round_half_up (#6781)
  • Area labels: area:shuffle, area:expressions; also carries performance
  • Rationale: A performance optimization of pmod in shuffle partitioning and of wide decimal rescaling.
• Hand natively written Iceberg files' Parquet footers to the JVM instead of reading them back (#6772)
  • Area labels: area:writer; also carries area:Iceberg
  • Rationale: A performance optimization that saves two storage reads for each natively written file. Confirms the author's label.
• Warn when unrecognized spark.comet.* configuration keys are set (#6767)
  • Area labels: none
  • Rationale: A new warning for mistyped configuration keys. Confirms the author's label.
• perf: reuse the nested unsafe views in ArrowWriter's row path, detaching them once per row (#6765)
  • Area labels: area:scan; also carries performance
  • Rationale: A performance optimization of Spark-to-Arrow conversion, under [EPIC] Faster Spark-to-Arrow conversion in CometSparkToColumnarExec #6565.
• perf: converting unsafe arrays of maps to Arrow is up to 1.7x slower when C2 inlines MapWriter.writeEntries (#6757)
  • Area labels: area:scan; also carries performance
  • Rationale: A JIT tuning question in the perf: convert Spark's unsafe rows to Arrow straight from their memory #6730 writer, which is still faster than before. Not a regression, under [EPIC] Faster Spark-to-Arrow conversion in CometSparkToColumnarExec #6565.
• Codegen dispatcher: run more Scala UDF null-guard shapes inside the kernel (#6755)
  • Area labels: area:expressions; also carries performance, area:udf
  • Rationale: A performance follow-up to perf: run Spark's null guard for a Scala UDF inside the codegen kernel #6715 that doesn't change results. Confirms the author's label.
• Stop ending Spark-to-Arrow batches at input file boundaries (#6752)
  • Area labels: area:scan; also carries performance
  • Rationale: A batch size optimization that depended on fix: keep a Spark scan unconverted when the plan reads input_file_name #6703, now merged. Confirms the author's label.
• Add an end-to-end HDFS write test for both native Parquet writers (#6750)
  • Area labels: area:writer
  • Rationale: New test coverage. Confirms the author's label.
• Extend per-scan HTTP attempt and retry metrics to native Azure and GCS (#6748)
  • Area labels: area:scan
  • Rationale: Extends the scan HTTP metrics from the unmerged feat: expose per-scan S3 HTTP coverage and retries #6429 to Azure and GCS.
• Design question: long-term, should native Iceberg scans run through DataFusion's ParquetSource instead of iceberg-rust's ArrowReader? (#6737)
  • Area labels: area:scan
  • Rationale: A design discussion about the long-term read path for native Iceberg scans, with nothing broken.
• Explore Kubernetes in-place pod memory resize (Spark 4.2 ExecutorResizePlugin) for Comet's untracked native memory (#6727)
  • Area labels: none; also carries area:memory
  • Rationale: An exploration of Spark 4.2's ExecutorResizePlugin for Comet's untracked native memory. Confirms the author's label.
• Native Iceberg scan reads every field of a nested column even when Spark prunes it (#6724)
  • Area labels: area:scan; also carries performance, area:Iceberg
  • Rationale: Results are correct, and this removes the I/O and decode cost of nested fields Spark has pruned. Fix in perf: read only the nested fields Spark keeps in the native Iceberg scan #6725.
• Row-input conversionTime metric includes the time the child spends producing rows (#6723)
  • Area labels: area:scan
  • Rationale: A metric that counts more than its description says. The issue proposes relabeling or dropping it, since timing each row would cost more than the conversion. Confirms the author's label.
• Admit all array and map types in CometSparkToColumnarExec (#6722)
  • Area labels: area:scan; also carries performance
  • Rationale: These types fall back today, so admitting them is new coverage. Fix in feat: admit every array and map type in Spark-to-Comet conversion #6726. Confirms the author's label.
• Reuse Arrow buffers across batches in Spark-to-Arrow conversion instead of reallocating and zeroing them (#6720)
  • Area labels: area:scan; also carries performance
  • Rationale: A performance optimization under [EPIC] Faster Spark-to-Arrow conversion in CometSparkToColumnarExec #6565. Confirms the author's label.
• Vectorize the lower-bound check in native date_trunc for fine units (#6717)
  • Area labels: area:expressions; also carries performance
  • Rationale: A performance optimization of a check feat: Use DataFusion date_trunc for scalar-format timestamp truncation #5956 added. The author's priority:low was left in place.
• Native Parquet scan: FLOAT and DOUBLE comparisons in data filters don't prune with row-level pushdown (#6702)
  • Area labels: area:scan; also carries performance
  • Rationale: Lost pruning in the opt-in row-level pushdown mode, with correct results. Confirms the author's label.
• Support custom vectorized Java UDFs that operate on Arrow vectors (#6694)
  • Area labels: none; also carries area:udf
  • Rationale: A new UDF API. Its prototype, feat: custom vectorized Java and Scala UDFs [experimental] #6697, was closed. Confirms the author's label.
• Native Iceberg writes: write row lineage for copy-on-write DML and compaction on format-version 3 tables (#6669)
  • Area labels: area:writer; also carries area:Iceberg
  • Rationale: These writes fall back to iceberg-java today, so native support is new functionality. Confirms the author's label.

Escalations to consider

• Native timezone conversions stop applying DST after 2099, while Spark keeps applying the zone's rules (#6816)
  • Raised from the author's priority:low to medium, the level this pass and Native make_date rejects Spark-valid years outside chrono range #5208 hold far-from-present dates at. Step 1 reads critical, and there is a realistic trigger: in a southern-hemisphere DST zone such as Australia/Sydney, a 9999-12-31 sentinel timestamp falls in summer, so hour() and casts to string come out an hour off from Spark. The reviewer may restore low, since the compatibility guide documents the difference, or raise it.
• [EPIC] In-memory cache audit: correctness and performance follow-ups for InMemoryTableScanExec support (#6834)
  • Labeled critical for its open wrong-results item, Native hash joins lose the streamed side's order when it arrives sorted from outside the native plan #6787, whose fix is fix: keep a streamed side sorted outside the native plan in order through outer hash joins #6835. Once that lands, the open items are performance work and the opt-in Cached AQE queries fail after transition reversion on Spark 4.0+ #6687, so medium would fit.
• from_csv: wrong results in permissive modes and failure with a variant schema (#6800)
  • Labeled critical from the report alone. It isn't diagnosed and was seen only with localTableScan on. from_csv goes through the codegen dispatcher, which runs Spark's own code and is on by default, so a reproduction on main over a Parquet source would settle whether a default path reaches it.
• arrays_overlap keeps -0.0 and 0.0 apart on Spark 4.0.5, 4.1.4 and 4.2.1, which merge them (SPARK-59602) (#6768)
  • Labeled critical, but only Spark 4.0.5, 4.1.4 and 4.2.1 are affected, and they were release candidates when the issue was filed. Spark 4.0.4, 4.1.3 and 4.2.0 aren't affected.
• Native CSV scan reads every partition through the first partition's object store (#6747)
  • Labeled critical on the opt-in native CSV scan (spark.comet.scan.csv.v2.enabled), as input_file_name() returns empty values above a converted Spark Parquet scan #6573 was for spark.comet.convert.parquet.enabled. The reviewer may prefer priority:high under the "core path over experimental" principle.
• Native Iceberg scan returns nested values that Spark reads as NULL in files without field ids or a name mapping (#6794)
  • Labeled critical because Comet's answer differs from Spark's. Here Comet returns the values stored in the file and Spark returns NULL, as the Iceberg spec requires. It needs data files added without field ids and without a name mapping, which the migration procedures don't produce.
• Math functions differ from Spark in the last digits (#6797)
  • Held at medium. Step 1 reads critical for any default-config difference from Spark, but these values differ only in the last digits. If any of the functions turns out to differ by more than floating-point precision, critical fits.
• Native date functions and date-to-string cast return NULL or fail for dates outside chrono's NaiveDate range (#6843)
  • Held at medium, with Native cast string to date returns NULL for years outside chrono's NaiveDate range #6842 and the Date32 overflow in [Bug] Native make_date mishandles Date32 overflow and skipped evaluation #6741, following Native make_date rejects Spark-valid years outside chrono range #5208 and Native Iceberg write panics or writes a NULL partition value for dates and timestamps beyond year 262143 #6145. Step 1 reads critical, but they need years beyond 262142 (6,000,000 for [Bug] Native make_date mishandles Date32 overflow and skipped evaluation #6741), which real data doesn't reach. Date-to-timestamp casts can overflow or panic for wide dates #5456, a wide-date cast that overflows into wrong values, is critical, so the reviewer may want to compare.
• ShuffleScan metrics are dropped under AQE: shuffle read and decode time missing on the direct-read path (#6804)
  • Held at medium. Metrics-only defects have gone both ways: Broadcast/hash join fallback reasons are lost from the AQE-final plan #6442 (lost fallback reasons) is low, and a reviewer raised Native metrics from several plan instances in one task overwrite each other, so a coalesced scan reports only its last partition #5879 (overwritten scan metrics) to high.
• GROUP BY on a struct containing a map fails with CometNativeException (#6801)
  • Held at medium with Null-named attribute fails Arrow export with "field name cannot be null" #6802, because both were seen only with the opt-in localTableScan. If either is a native panic or an NPE that a default path can reach, priority:high fits.
• Native mode treats all NaNs as one value on Spark 3.4, where Spark counts each NaN separately (#6738)
  • Held at medium under the [Bug] Native make_interval overflows valid time components or loses seconds precision #5131 rule for Incompatible opt-ins. Spark 3.4's behavior here is the bug SPARK-45599 fixed, which is why a reviewer lowered array_distinct and array_union diverge from Spark on -0.0 for Spark versions without SPARK-54918 #5701 from critical, so low is also defensible.

Skipped (needs more info)

#6576 has no reproduction on a supported path. The others are tracking or record issues rather than bug reports or feature requests. requires-triage was left in place on all five, so they reappear in the next pass until they are closed or classified.

• Arrow struct writer should respect projected schema width (#6576)
  • There is still no reproduction on a supported path. The fix: write columnar structs to Arrow schema width #6578 review on 2026-10-07 asked which producer hands Comet a struct column wider than the plan's attribute, and that is unanswered. If one exists, main converts a mismatched prefix silently, which is step 1. If none does, this is hardening for Support Native Iceberg MOR write #6240.
• Audit the PRs in 1.1.0 for regressions since 1.0.0 (#6399)
  • An audit task, not a bug report or a feature request. Every phase is done and the results are in [EPIC] Regressions in 1.1.0 since 1.0.0 #6402, so it can be closed.
• Bug triage results: 2026-10-05 (#6665)
  • The summary of the 2026-10-05 pass. No type or priority label it applied has changed since, so it can be closed once reviewed.
• Bug triage results: 2026-09-28 (#6321)
  • The summary of the 2026-09-28 pass, with nothing outstanding, so it can be closed.
• Bug triage results: 2026-08-24 (#5454)
  • The summary of the 2026-08-24 pass. dwsmith1983 suggested closing it on 2026-10-04, as the other summaries from that period were.