Bug triage results: 2026-08-11

[andygrove]

Triage pass over the open requires-triage queue, per the project Bug Triage Guide.

• Date: 2026-08-11
• Total issues processed: 44 (43 triaged, 1 skipped, 0 failed)
• Type counts: 13 bugs, 30 enhancements
• Priority counts applied: priority:critical 2, priority:high 3, priority:medium 6, priority:low 2
• Guide: docs/source/contributor-guide/bug_triage.md

Labels have already been applied. A reviewer should spot-check the calls below and close this issue when satisfied; corrections should be made directly on the affected issue.

Notes on label availability: the guide lists spark 4 as a pre-existing area indicator, but the repository only has spark 4.0 / spark 4.1 / spark 4.2, so no spark 4 label was applied in this pass. Where an issue already carried non-guide labels (correctness, performance, test, array expressions, crash, area:Iceberg), those were left untouched and are included in the area lists below for context.

Bugs

priority:critical

• A Rust UDF silently answers calls to an ordinary Scala UDF registered under the same name (#5295)
  • Area labels: area:expressions
  • Rationale: a Scala UDF registered under a name a Rust UDF already claimed is evaluated by the Rust library and returns wrong values with no error, which is the guide's "silent wrong results" case at the top of the decision tree.
• arrays_overlap on flat float arrays does not canonicalize NaN like Spark (#5270)
  • Area labels: area:expressions, correctness, array expressions
  • Rationale: Comet returns false where Spark returns true for non-canonical NaN bit patterns with no error, a silent correctness divergence; escalated from the reporter's priority:medium per the guide's "correctness over crashes" principle.

priority:high

• Native collect_set(struct) aggregate crashes on spill emit (#5239)
  • Area labels: area:aggregation
  • Rationale: aborts in GroupedHashAggregateStream::emit with an ArrowError on the spill path, a crash on a supported code path per the guide's step 2.
• [Bug] Preserve CalendarInterval microseconds across Comet boundaries and native kernels (#5279)
  • Area labels: area:expressions
  • Rationale: an uncaught ArithmeticException: long overflow aborts the query even with ANSI mode disabled, where Spark returns a valid result; visible failure rather than silent, so priority:high.
• arrays_overlap panics on nested double arrays built from expressions (#5269)
  • Area labels: area:expressions, array expressions, crash
  • Rationale: native panic (index out of bounds in GenericListArray::value), matching the guide's crash-bug category; confirms the reporter's priority:high.

priority:medium

• Iceberg serde: delete-file fields fall back to wrong defaults on reflection failure (#5256)
  • Area labels: area:scan, area:Iceberg, correctness
  • Rationale: a reflection failure substitutes defaults that leak deleted rows or delete the wrong rows, which would be priority:critical on correctness grounds, but the issue states the paths are latent and not reachable on any supported Iceberg version today; kept at the reporter's priority:medium and listed under escalations.
• Iceberg serde: schema and metadata reflection helpers return empty collections on failure (#5257)
  • Area labels: area:scan, area:Iceberg, correctness
  • Rationale: same family as Iceberg serde: delete-file fields fall back to wrong defaults on reflection failure #5256 — empty mappings can resolve a column to a different field id or silently change which schema a task is serialized with, but only on a reflection failure that is not reachable today.
• Iceberg write split-operator plan: CI failures when enabled by default (#5259)
  • Area labels: area:writer, area:ci, area:Iceberg
  • Rationale: 8 CI jobs fail when the split-operator plan is enabled, but the feature ships off by default so nothing is broken for users today; a broken feature with a workaround is priority:medium. Re-labeled from enhancement to bug since the body describes concrete test failures.
• Bare -0.0 literals in SQL file tests are coerced through decimal, so signed-zero coverage is vacuous (#5271)
  • Area labels: area:expressions, test
  • Rationale: test-only defect, which the guide places at priority:low, but the fixtures give false confidence in signed-zero coverage across ~35 files and have already let a real divergence through review, so the reporter's priority:medium is retained.
• Native Parquet writer derives schema nullability and field IDs from Arrow rather than Catalyst (#5305)
  • Area labels: area:writer
  • Rationale: written files differ from Spark's in nullability and field IDs, a functional gap in the still-experimental native writer, which the guide's native-writer note places at lower priority.
• Native Parquet write of a zero-partition RDD produces no output file (#5303)
  • Area labels: area:writer
  • Rationale: missing schema-only output versus Spark, behind an off-by-default config with a fallback available, so priority:medium per step 3 of the decision tree.

priority:low

• Reading Iceberg tables show 0.0B for the Input column on Spark's executor UI (#5264)
  • Area labels: area:scan
  • Rationale: cosmetic metrics-reporting gap in the Spark UI; query results are unaffected.
• Discuss spark.comet.enabled parameter behavior (#5274)
  • Area labels: none
  • Rationale: a misleading "native library initialized" log line is emitted when Comet is disabled — cosmetic, no functional impact. Re-labeled from enhancement to bug since the body reports incorrect observable behavior rather than requesting new functionality.

Enhancements

• Sort-aware Iceberg reads in Comet via a per-partition streaming merge (#5323)
  • Area labels: area:scan
  • Rationale: new capability — report scan ordering and merge sorted files natively so Spark can eliminate redundant sorts.
• Support Iceberg SparkCopyOnWriteScan in IcebergReflection.ICEBERG_SCAN_CLASSES (#5319)
  • Area labels: area:scan
  • Rationale: extends native scan coverage to a scan class Comet does not claim to support yet; falling back to Spark is the expected behavior today.
• refactor: centralize Spark-to-Arrow batch encoding in ArrowWriter (#5317)
  • Area labels: area:ffi
  • Rationale: internal refactor of the Spark-to-Arrow conversion paths; the guide lists refactoring under enhancement.
• Support S3-compliant object stores (#5313)
  • Area labels: area:scan
  • Rationale: new functionality for non-AWS S3 endpoints; the current gate correctly falls back to Spark rather than misbehaving.
• Evaluate extending collect_set input normalization to Spark 3.4 through 4.1 (#5312)
  • Area labels: area:aggregation
  • Rationale: evaluation task about widening a documented, version-gated compatibility restriction, not a defect against the current contract.
• WideDecimalBinaryExpr: write nulls directly for overflow (#5309)
  • Area labels: area:expressions
  • Rationale: performance optimization of the overflow path; behavior is explicitly to be preserved.
• feat: Write Iceberg Parquet data files natively via iceberg-rust (part 3 of 3) (#5308)
  • Area labels: area:writer
  • Rationale: new native write capability, part 3 of the Writes to Apache Iceberg Tables #4322 deliverable.
• Native Parquet writes report row counts but not row contents to WriteTaskStatsTrackers (#5307)
  • Area labels: area:writer
  • Rationale: only third-party trackers that inspect row contents are affected; Spark's only shipped tracker ignores the row argument, and the gap is warned about today, so this is hardening rather than a defect.
• Reconcile the two config namespaces for native writes (#5306)
  • Area labels: area:writer
  • Rationale: naming cleanup across two testing-category configs; nothing user-facing is broken.
• Native Parquet writer ignores Spark's Parquet writer properties (#5304)
  • Area labels: area:writer
  • Rationale: feature gap in the experimental native writer — properties are not yet plumbed through, rather than plumbed through incorrectly.
• Add a C++ UDF example and a published C header for the Comet UDF ABI (#5301)
  • Area labels: none
  • Rationale: new artifacts (header, example, CI smoke test) plus documentation.
• perf: explore bulk copies for no-null fixed-width Arrow writes (#5299)
  • Area labels: area:ffi
  • Rationale: performance investigation, explicitly conditional on benchmarks.
• Rust UDF library cache holds its write lock across dlopen (#5297)
  • Area labels: performance
  • Rationale: contention and lock-poisoning hardening in a path the reporter notes has bounded impact in practice; no incorrect results or crashes in normal operation.
• Rust UDF adapter rebuilds the kernel impl and re-resolves the return type on every batch (#5296)
  • Area labels: performance
  • Rationale: redundant per-batch work; an optimization with an ABI-contract decision attached.
• Scope the Rust UDF registry to the session that registered the UDF (#5294)
  • Area labels: none
  • Rationale: design change to remove a process-wide singleton per the contributor guide; the wrong-results half of the same problem is tracked separately as the bug A Rust UDF silently answers calls to an ordinary Scala UDF registered under the same name #5295.
• Determine whether "Apache Comet" is a suitable project name (#5291)
  • Area labels: none
  • Rationale: project-administration task, not a code defect.
• Comet 1.0.1 Release (#5283)
  • Area labels: none
  • Rationale: release-tracking issue.
• Remove orphaned files in comet iceberg split writers (#5277)
  • Area labels: area:writer
  • Rationale: cleanup-on-abort is functionality the split writer does not implement yet, behind an off-by-default config.
• Build spark_size LargeList lengths as Int32 without Int64 cast (#5272)
  • Area labels: area:expressions
  • Rationale: allocation-removing optimization on the spark_size LargeList path; results are already correct.
• Use the offset buffer for Map in spark_size (#5266)
  • Area labels: area:expressions
  • Rationale: vectorization of the Map branch, explicitly droppable if the benchmark comes back flat.
• IcebergReflection: Option helpers cannot distinguish an absent accessor from a failed reflective call (#5258)
  • Area labels: area:scan, area:Iceberg
  • Rationale: API design change to the reflection layer so callers can tell the two outcomes apart; the concrete wrong-behavior consequences are tracked as the bugs Iceberg serde: delete-file fields fall back to wrong defaults on reflection failure #5256 and Iceberg serde: schema and metadata reflection helpers return empty collections on failure #5257.
• Add a forward-flexible property slot to the Comet UDF C ABI (#5254)
  • Area labels: none
  • Rationale: ABI extension for forward compatibility.
• Relax the arrow version requirement in comet-udf-sdk (#5253)
  • Area labels: none
  • Rationale: dependency-range relaxation to unblock future SDK consumers; nothing broken today.
• Remove the per-UDF Mutex serializing Rust UDF batches (#5252)
  • Area labels: performance
  • Rationale: removes defensive locking that serializes concurrent batches; an optimization, not a defect.
• Implement return_field() for imported Rust UDFs (#5251)
  • Area labels: none
  • Rationale: new capability to carry a kernel's full Field (non-nullability, metadata) through to the plan.
• Make kernel ownership transfer explicit in the Comet UDF C ABI (#5250)
  • Area labels: none
  • Rationale: the issue states the current behavior is correct; this makes the contract harder to get wrong.
• Honor the deterministic flag for Rust UDFs (#5249)
  • Area labels: none
  • Rationale: new support for deterministic = false, which feat: custom Rust UDFs via arrow-ffi [experimental] #4459 currently rejects at registration so the flag cannot silently lie.
• Avoid re-running DecimalPrecision.promote during recursive expression serialization (#5248)
  • Area labels: area:expressions
  • Rationale: removes redundant traversals; the correctness consequence was already fixed by fix: avoid duplicate CheckOverflow evaluation for decimal division #5225, leaving a planner performance audit.
• Add Criterion bench for native RLike (is_match loop) (#5246)
  • Area labels: area:expressions
  • Rationale: adds benchmark coverage for a hot path.
• Add AQE test coverage for CometInMemoryTableScanExec (#5245)
  • Area labels: none
  • Rationale: adds test coverage; no defect is reported, only an untested interaction.

Escalations to consider

• Iceberg serde: delete-file fields fall back to wrong defaults on reflection failure (#5256)
  • Trigger: the guide places silent wrong results at priority:critical, and the described failure modes leak deleted rows or delete the wrong rows. Held at priority:medium only because the issue's own reachability analysis says the paths cannot fire on any supported Iceberg version; if that analysis is wrong, this is a priority:critical.
• Iceberg serde: schema and metadata reflection helpers return empty collections on failure (#5257)
  • Trigger: same as Iceberg serde: delete-file fields fall back to wrong defaults on reflection failure #5256 — silent wrong results on a currently unreachable path. Escalate together with Iceberg serde: delete-file fields fall back to wrong defaults on reflection failure #5256 if reachability changes.
• Native collect_set(struct) aggregate crashes on spill emit (#5239)
  • Trigger: the guide escalates a priority:high crash to priority:critical if it also produces wrong results silently. The report has no reproduction steps, so it is unknown whether a non-spilling variant of the same schema mismatch yields wrong output rather than aborting.

Skipped — needs more info

• Bug triage results: 2026-08-03 (#5231)
  • This is the previous triage summary issue, awaiting human review. It is neither a bug nor an enhancement, so no type label was applied and requires-triage was left in place; it will drop out of the queue when a reviewer closes it.