From 37b2ba6d9cde30fef594de6acc2df36ecc418570 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Mon, 28 Sep 2026 13:25:46 -0600 Subject: [PATCH] docs: add 1.1.0 changelog --- docs/source/changelog/1.1.0.md | 484 +++++++++++++++++++++++++++++++++ docs/source/changelog/index.md | 1 + 2 files changed, 485 insertions(+) create mode 100644 docs/source/changelog/1.1.0.md diff --git a/docs/source/changelog/1.1.0.md b/docs/source/changelog/1.1.0.md new file mode 100644 index 00000000000..3270cc480e4 --- /dev/null +++ b/docs/source/changelog/1.1.0.md @@ -0,0 +1,484 @@ + + +# DataFusion Comet 1.1.0 Changelog + +This release consists of 398 commits from 40 contributors. See credits at the end of this changelog for more information. + +**Fixed bugs:** + +- fix: skip null slots when checking overflow in unary negation [#5162](https://github.com/apache/datafusion-comet/pull/5162) (Smallfu666) +- fix: surface next_day and make_date ANSI errors as Spark exceptions [#5167](https://github.com/apache/datafusion-comet/pull/5167) (peterxcli) +- fix: normalize nested field nullability in ShuffleScanExec and ExpandExec [#5138](https://github.com/apache/datafusion-comet/pull/5138) (andygrove) +- fix: propagate the Spark task ClassLoader to JVM UDF calls [#5282](https://github.com/apache/datafusion-comet/pull/5282) (andygrove) +- fix: avoid duplicate CheckOverflow evaluation for decimal division [#5225](https://github.com/apache/datafusion-comet/pull/5225) (peterxcli) +- fix: match Spark whitespace trimming in to_time and try_to_time [#5364](https://github.com/apache/datafusion-comet/pull/5364) (sunchao) +- fix: preserve Catalyst nullability and field IDs in native Parquet writes [#5369](https://github.com/apache/datafusion-comet/pull/5369) (sunchao) +- fix: guard against silent fail_on_error loss in scalar wiring (#5074) [#5359](https://github.com/apache/datafusion-comet/pull/5359) (sam-1112) +- fix: preserve Spark semantics for dictionary-encoded Parquet inputs and reject dictionary targets [#5234](https://github.com/apache/datafusion-comet/pull/5234) (peterxcli) +- fix: canonicalize NaN in flat arrays_overlap float keys [#5376](https://github.com/apache/datafusion-comet/pull/5376) (sunchao) +- fix: report native shuffle write metrics accurately [#5370](https://github.com/apache/datafusion-comet/pull/5370) (sunchao) +- fix: Native shuffle fails with a 2GB task serialization OOM on jobs with many partitions [#5392](https://github.com/apache/datafusion-comet/pull/5392) (parthchandra) +- fix: format NativeMemoryConsumer id in toString [#5398](https://github.com/apache/datafusion-comet/pull/5398) (ywskycn) +- fix: ignore reader-side parquet.hadoop.vectored.io.enabled in Iceberg native-write detection [#5410](https://github.com/apache/datafusion-comet/pull/5410) (snmvaughan) +- fix: support map casts with NullType elements [#5045](https://github.com/apache/datafusion-comet/pull/5045) (peterxcli) +- fix: distinguish reflection failure from absent accessor in IcebergReflection [#5412](https://github.com/apache/datafusion-comet/pull/5412) (unikdahal) +- fix: use current shuffle config in aggregate test [#5439](https://github.com/apache/datafusion-comet/pull/5439) (peterxcli) +- fix: support wide years in native make_date [#5443](https://github.com/apache/datafusion-comet/pull/5443) (peterxcli) +- fix: report native child spill metrics in shuffle tasks [#5445](https://github.com/apache/datafusion-comet/pull/5445) (sunchao) +- fix: expose native memory usage to Spark [#5408](https://github.com/apache/datafusion-comet/pull/5408) (ywskycn) +- fix: release native shuffle reservation after spill failure [#5461](https://github.com/apache/datafusion-comet/pull/5461) (peterxcli) +- fix: track peak memory before JVM shuffle spill [#5463](https://github.com/apache/datafusion-comet/pull/5463) (peterxcli) +- fix: RAII for memory pool registration [#5464](https://github.com/apache/datafusion-comet/pull/5464) (peterxcli) +- fix: tighten RSS writer visibility and JNI array limits [#5475](https://github.com/apache/datafusion-comet/pull/5475) (pingzh) +- fix: report native operator spill metrics in Spark task metrics for non-shuffle stages [#5497](https://github.com/apache/datafusion-comet/pull/5497) (peterxcli) +- fix: report bounded shuffle allocator memory usage [#5516](https://github.com/apache/datafusion-comet/pull/5516) (ywskycn) +- fix: use Spark type names in ANSI abs overflow errors [#5357](https://github.com/apache/datafusion-comet/pull/5357) (Smallfu666) +- fix: make collect_list/collect_set argument coercion a normalization barrier [#5159](https://github.com/apache/datafusion-comet/pull/5159) (andygrove) +- fix: make task-shared memory pool as ref-counted RAII guard [#5494](https://github.com/apache/datafusion-comet/pull/5494) (peterxcli) +- fix: accept UTC timezone aliases in Python Arrow input [#5556](https://github.com/apache/datafusion-comet/pull/5556) (sunchao) +- fix: normalize scalar float sort and window rank keys [#5469](https://github.com/apache/datafusion-comet/pull/5469) (sunchao) +- fix: make CometExplodeExec respect batch size [#5362](https://github.com/apache/datafusion-comet/pull/5362) (andygrove) +- fix: align Spark 4.2 Python worker configuration [#5561](https://github.com/apache/datafusion-comet/pull/5561) (sunchao) +- fix: prevent memory leak after failed Arrow vector import [#5539](https://github.com/apache/datafusion-comet/pull/5539) (1fanwang) +- fix: preserve Arrow Field metadata across C Data exports [#5552](https://github.com/apache/datafusion-comet/pull/5552) (peterxcli) +- fix: rebase map offsets in mapsort so sliced maps do not overrun entries [#5630](https://github.com/apache/datafusion-comet/pull/5630) (viirya) +- fix: scope Celeborn bootstrap hooks to Comet clients [#5627](https://github.com/apache/datafusion-comet/pull/5627) (pingzh) +- fix: skip codegen dispatcher null short-circuit when a foldable subtree can raise [#5623](https://github.com/apache/datafusion-comet/pull/5623) (andygrove) +- fix: accept dictionary encodings in remote shuffle [#5650](https://github.com/apache/datafusion-comet/pull/5650) (pingzh) +- fix: make CometDiskBlockWriter spill registry per-task instead of executor-global [#5493](https://github.com/apache/datafusion-comet/pull/5493) (peterxcli) +- fix: Bump iceberg-rust so native Iceberg writes URL-escape partition paths [#5651](https://github.com/apache/datafusion-comet/pull/5651) (andygrove) +- fix(celeborn): reject unsafe native push completion tracking [#5665](https://github.com/apache/datafusion-comet/pull/5665) (pingzh) +- fix: match Spark's null short-circuiting in array_join and enable it natively [#5558](https://github.com/apache/datafusion-comet/pull/5558) (Visorgood) +- fix: make native shuffle spill metrics independent of input batching [#5628](https://github.com/apache/datafusion-comet/pull/5628) (sunchao) +- fix: expand object store option references, uniquify constant metadata names, drop dead parquet JNI [#5653](https://github.com/apache/datafusion-comet/pull/5653) (dwsmith1983) +- fix: match Spark's ANSI bound check for float/double to integral casts [#5683](https://github.com/apache/datafusion-comet/pull/5683) (peterxcli) +- fix: return NULL from rpad/lpad when the length column is NULL instead of panicking [#5680](https://github.com/apache/datafusion-comet/pull/5680) (peterxcli) +- fix: fall back for concat_ws with array arguments instead of failing natively [#5679](https://github.com/apache/datafusion-comet/pull/5679) (peterxcli) +- fix: prevent silent overflow when reading Parquet TIMESTAMP_MILLIS values [#5177](https://github.com/apache/datafusion-comet/pull/5177) (peterxcli) +- fix: recover native Celeborn shuffle from oversized rows [#5668](https://github.com/apache/datafusion-comet/pull/5668) (pingzh) +- fix: report native shuffle read metrics [#5554](https://github.com/apache/datafusion-comet/pull/5554) (peterxcli) +- fix: correctly rounded decimal to double/float cast matching BigDecimal.doubleValue/floatValue [#5684](https://github.com/apache/datafusion-comet/pull/5684) (peterxcli) +- fix: restore columnar transitions under the native Iceberg write [#5696](https://github.com/apache/datafusion-comet/pull/5696) (andygrove) +- fix: make columnar-to-row benchmarks exercise Comet [#5718](https://github.com/apache/datafusion-comet/pull/5718) (rich7420) +- fix: distinguish "nothing spilled" from a spill backend with no local path [#5726](https://github.com/apache/datafusion-comet/pull/5726) (andygrove) +- fix: propagate Arrow array copy errors [#5747](https://github.com/apache/datafusion-comet/pull/5747) (rich7420) +- fix: keep the dictionary hash fast path off nested and reseeded buffers [#5757](https://github.com/apache/datafusion-comet/pull/5757) (viirya) +- fix: preserve ANSI errors for rejected TIMESTAMP_NTZ casts [#5752](https://github.com/apache/datafusion-comet/pull/5752) (peterxcli) +- fix: apply the parent struct's null mask before hashing its fields [#5754](https://github.com/apache/datafusion-comet/pull/5754) (viirya) +- fix: read Iceberg tables partitioned by an unknown transform [#5759](https://github.com/apache/datafusion-comet/pull/5759) (andygrove) +- fix: native Iceberg write panics on an evolved partition spec and on a timestamptz partition path [#5729](https://github.com/apache/datafusion-comet/pull/5729) (andygrove) +- fix: let AQE optimize queries over Comet caches [#5733](https://github.com/apache/datafusion-comet/pull/5733) (peterxcli) +- fix: dispatch Iceberg system functions wrapped as ApplyFunctionExpression [#5773](https://github.com/apache/datafusion-comet/pull/5773) (andygrove) +- fix: attach tokio runtime threads to the JVM as daemon threads [#5748](https://github.com/apache/datafusion-comet/pull/5748) (zhangfengcdt) +- fix: check nested TIMESTAMP_MILLIS overflow in unfiltered scans [#5740](https://github.com/apache/datafusion-comet/pull/5740) (peterxcli) +- fix: match iceberg-java's exception for unclustered input to a clustered Iceberg write [#5779](https://github.com/apache/datafusion-comet/pull/5779) (andygrove) +- fix: enable FIRST/LAST partial merge [#5041](https://github.com/apache/datafusion-comet/pull/5041) (peterxcli) +- fix: preserve aggregate result identity during exchange reuse [#5470](https://github.com/apache/datafusion-comet/pull/5470) (sunchao) +- fix: ignore structural tags when lifting expression coverage [#5471](https://github.com/apache/datafusion-comet/pull/5471) (sunchao) +- fix: align string to timestamp parsing with Spark's segment rules [#5682](https://github.com/apache/datafusion-comet/pull/5682) (peterxcli) +- fix: roll native Iceberg data files on iceberg-java's 1000-row grid [#5780](https://github.com/apache/datafusion-comet/pull/5780) (andygrove) +- fix: decide libhdfs routing from the scheme as written [#5825](https://github.com/apache/datafusion-comet/pull/5825) (comphead) +- fix: list a fanout Iceberg write's data files in a stable order [#5810](https://github.com/apache/datafusion-comet/pull/5810) (andygrove) +- fix: isolate object-store registration by backend and configuration [#5503](https://github.com/apache/datafusion-comet/pull/5503) (sunchao) +- fix: Delete completed tasks' data files when an Iceberg write job fails [#5663](https://github.com/apache/datafusion-comet/pull/5663) (andygrove) +- fix: apply Spark's Parquet conversion rules to nested struct/list/map fields [#5681](https://github.com/apache/datafusion-comet/pull/5681) (peterxcli) +- fix: preserve nulls for Boolean/Byte/Short/Integer columns in FuzzDataGenerator [#5855](https://github.com/apache/datafusion-comet/pull/5855) (Smallfu666) +- fix: Accept explicit positive years in timestamp casts [#5858](https://github.com/apache/datafusion-comet/pull/5858) (peterxcli) +- fix: decline structs with duplicate field names before they reach Java Arrow [#5866](https://github.com/apache/datafusion-comet/pull/5866) (dwsmith1983) +- fix: explain ObjectHashAggregate fallback when Comet shuffle is disabled [#5746](https://github.com/apache/datafusion-comet/pull/5746) (0lai0) +- fix: preserve join and generator semantics in plan identity [#5828](https://github.com/apache/datafusion-comet/pull/5828) (ErikBPF) +- fix: propagate Parquet field-name folding failures [#5845](https://github.com/apache/datafusion-comet/pull/5845) (sunchao) +- fix: decode dictionary input for PyArrow UDFs [#5560](https://github.com/apache/datafusion-comet/pull/5560) (sunchao) +- fix: size JVM shuffle pointer array growth from the array, not the data pages [#5907](https://github.com/apache/datafusion-comet/pull/5907) (andygrove) +- fix: render float and double Iceberg partition values like iceberg-java [#5840](https://github.com/apache/datafusion-comet/pull/5840) (andygrove) +- fix: align time parsing and native second extraction with Spark [#5738](https://github.com/apache/datafusion-comet/pull/5738) (peterxcli) +- fix: normalize floating-point values in native collect_set [#5166](https://github.com/apache/datafusion-comet/pull/5166) (peterxcli) +- fix: keep Iceberg complex null checks on native scans [#5732](https://github.com/apache/datafusion-comet/pull/5732) (ErikBPF) +- fix: decode invalid UTF-8 at the JVM to native FFI import boundary [#5310](https://github.com/apache/datafusion-comet/pull/5310) (manuzhang) +- fix: fall back when a struct repeats a Parquet field id [#6004](https://github.com/apache/datafusion-comet/pull/6004) (comphead) +- fix: normalize signed zero in nested float array comparisons [#5235](https://github.com/apache/datafusion-comet/pull/5235) (divyankshah) +- fix: fall back to Spark for native Iceberg writes to gs:// through HadoopFileIO [#5935](https://github.com/apache/datafusion-comet/pull/5935) (zhangfengcdt) +- fix: gate the regr_r2 degenerate-case swap on the Spark patch release [#6042](https://github.com/apache/datafusion-comet/pull/6042) (dwsmith1983) +- fix: include ABFS container in object store cache key [#5053](https://github.com/apache/datafusion-comet/pull/5053) (peterxcli) +- fix: preserve Spark row index read errors [#6046](https://github.com/apache/datafusion-comet/pull/6046) (liupoyi-1031) +- fix: remove the ineffective spark.executor.memoryOverhead adjustment from the driver plugin [#6054](https://github.com/apache/datafusion-comet/pull/6054) (andygrove) +- fix: count each memory pool once in analyze_trace [#5991](https://github.com/apache/datafusion-comet/pull/5991) (andygrove) +- fix: revert unsafe partial aggregates after final fallback [#5421](https://github.com/apache/datafusion-comet/pull/5421) (sunchao) +- fix: restore the site's mermaid diagrams and make a dropped one fail CI [#6064](https://github.com/apache/datafusion-comet/pull/6064) (andygrove) +- fix: write cached batches to the schema width, not the batch width [#6090](https://github.com/apache/datafusion-comet/pull/6090) (andygrove) +- fix(iceberg): guard native Iceberg scan driver-metric double-post, add metrics docs and tests [#6085](https://github.com/apache/datafusion-comet/pull/6085) (parthchandra) +- fix: let decimal SUM recover from an intermediate overflow like Spark [#6041](https://github.com/apache/datafusion-comet/pull/6041) (dwsmith1983) +- fix: Nested floating-point IN membership does not match Spark for signed zero [#6073](https://github.com/apache/datafusion-comet/pull/6073) (mizulun) +- fix: let Comet memory pools overcommit on grow instead of panicking [#6128](https://github.com/apache/datafusion-comet/pull/6128) (andygrove) +- fix: correct two nightly test failures on Spark 3.4 and 4.2 [#6156](https://github.com/apache/datafusion-comet/pull/6156) (andygrove) +- fix: support null calendar interval literals [#5133](https://github.com/apache/datafusion-comet/pull/5133) (peterxcli) +- fix: wrap Iceberg split-write failures the way Spark does when abort fails [#6153](https://github.com/apache/datafusion-comet/pull/6153) (andygrove) +- fix: support Utf8/LargeUtf8/Utf8View in native RLike without panicking [#5215](https://github.com/apache/datafusion-comet/pull/5215) (sam-1112) +- fix: normalize noncanonical NaN literals in comparisons [#5472](https://github.com/apache/datafusion-comet/pull/5472) (sunchao) +- fix: preserve map field metadata and honor target sorted flag in cast_map_to_map [#5227](https://github.com/apache/datafusion-comet/pull/5227) (Smallfu666) +- fix: reject duplicate Parquet field names before decoding [#5786](https://github.com/apache/datafusion-comet/pull/5786) (ErikBPF) +- fix: preserve Parquet conversion errors during join filtering [#6067](https://github.com/apache/datafusion-comet/pull/6067) (pingzh) +- fix: plan Iceberg writes with Spark's own operator when Comet is disabled [#6151](https://github.com/apache/datafusion-comet/pull/6151) (andygrove) +- fix: read shuffle write buffer, spill limit and off-heap sizes in bytes [#6191](https://github.com/apache/datafusion-comet/pull/6191) (andygrove) +- fix: bound shuffle schema cache retention and preserve eviction order [#6098](https://github.com/apache/datafusion-comet/pull/6098) (sunchao) +- fix: do not run Comet in on-heap mode without `spark.comet.exec.onHeap.enabled` [#6195](https://github.com/apache/datafusion-comet/pull/6195) (andygrove) +- fix: remove misleading native opt-in for dispatch-only datetime expressions [#6182](https://github.com/apache/datafusion-comet/pull/6182) (LinSimon-901101) +- fix: support CalendarIntervalType hashing [#5135](https://github.com/apache/datafusion-comet/pull/5135) (peterxcli) +- fix: report `Input` column when native Iceberg scan is enabled or native shuffle is enabled [#5265](https://github.com/apache/datafusion-comet/pull/5265) (hsiang-c) +- fix: skip the executor memory overhead warning when a factor is set or in local mode [#6198](https://github.com/apache/datafusion-comet/pull/6198) (andygrove) +- fix: keep operators above a cached relation native after the AQE re-plan [#6208](https://github.com/apache/datafusion-comet/pull/6208) (andygrove) +- fix: preserve current AQE logical-stage links on Comet operators [#5483](https://github.com/apache/datafusion-comet/pull/5483) (sunchao) +- fix: check each fair_unified reservation against its own share [#6205](https://github.com/apache/datafusion-comet/pull/6205) (andygrove) +- fix(iceberg): don't push transform residuals as their source column, fail on residual errors [#6154](https://github.com/apache/datafusion-comet/pull/6154) (andygrove) +- fix: match Spark's duplicate field and field id semantics in parquet field lookup [#5654](https://github.com/apache/datafusion-comet/pull/5654) (dwsmith1983) +- fix: park the native scan loop instead of busy-polling while waiting on native I/O [#6219](https://github.com/apache/datafusion-comet/pull/6219) (andygrove) +- fix: refresh S3 policy locations when a location's credential fails [#6223](https://github.com/apache/datafusion-comet/pull/6223) (andygrove) +- fix: inject Comet session extension rules only once per session [#6230](https://github.com/apache/datafusion-comet/pull/6230) (andygrove) +- fix: [branch-1.1] reject a file without field ids at any depth whether or not id matching is on (#6116) [#6266](https://github.com/apache/datafusion-comet/pull/6266) (andygrove) +- fix: [branch-1.1] decline native Iceberg writes with a custom location provider (#6216) [#6305](https://github.com/apache/datafusion-comet/pull/6305) (andygrove) +- fix: [branch-1.1] Native S3 scan on EKS/IRSA turns a transient STS throttle into a hard 403 storm (#6025) [#6323](https://github.com/apache/datafusion-comet/pull/6323) (andygrove) +- fix: [branch-1.1] log partial memory grants at DEBUG and drop the memory usage dump (#6269) [#6346](https://github.com/apache/datafusion-comet/pull/6346) (andygrove) + +**Performance related:** + +- perf: optimize spark_base64 in spark-expr [#4885](https://github.com/apache/datafusion-comet/pull/4885) (andygrove) +- perf: optimize `spark_floor` (up to 4x faster) [#4911](https://github.com/apache/datafusion-comet/pull/4911) (andygrove) +- perf: cache Iceberg reflection lookups on the planning path [#5222](https://github.com/apache/datafusion-comet/pull/5222) (andygrove) +- perf: vectorize integer-to-decimal cast [#4939](https://github.com/apache/datafusion-comet/pull/4939) (andygrove) +- perf: compute spark_size list lengths with Arrow length kernel [#5233](https://github.com/apache/datafusion-comet/pull/5233) (0lai0) +- perf: make CometShuffleBenchmark completable and fast [#5388](https://github.com/apache/datafusion-comet/pull/5388) (andygrove) +- perf: vectorize Map in spark_size via offset buffer [#5395](https://github.com/apache/datafusion-comet/pull/5395) (0lai0) +- perf: improve ArrowWriter performance for fixed-length vectors [#5046](https://github.com/apache/datafusion-comet/pull/5046) (peterxcli) +- perf: reuse Arrow IPC compression context across shuffle blocks [#5038](https://github.com/apache/datafusion-comet/pull/5038) (peterxcli) +- perf: use Arrow cast for decimal rescale check [#5440](https://github.com/apache/datafusion-comet/pull/5440) (peterxcli) +- perf: bulk copy fixed-width columns in ArrowWriter [#5442](https://github.com/apache/datafusion-comet/pull/5442) (peterxcli) +- perf: serialize Python input directly from Comet Arrow vectors [#5368](https://github.com/apache/datafusion-comet/pull/5368) (sunchao) +- perf: reuse per-partition scratch in the shuffle write path [#5568](https://github.com/apache/datafusion-comet/pull/5568) (dwsmith1983) +- perf: validate shuffle IPC context reuse savings [#5727](https://github.com/apache/datafusion-comet/pull/5727) (peterxcli) +- perf: slice the child instead of gathering it when unnesting [#5667](https://github.com/apache/datafusion-comet/pull/5667) (andygrove) +- perf: evaluate posexplode array expressions once per batch [#5737](https://github.com/apache/datafusion-comet/pull/5737) (rich7420) +- perf: cache expected schemas for remote shuffle decoding [#5722](https://github.com/apache/datafusion-comet/pull/5722) (pingzh) +- perf: use Arrow cast for date to timestamp NTZ [#5735](https://github.com/apache/datafusion-comet/pull/5735) (peterxcli) +- perf: reduce allocations when collecting cache statistics [#5734](https://github.com/apache/datafusion-comet/pull/5734) (peterxcli) +- perf: avoid repeated decimal promotion in expression serialization [#5736](https://github.com/apache/datafusion-comet/pull/5736) (peterxcli) +- perf: give collect_list and collect_set a native GroupsAccumulator [#5803](https://github.com/apache/datafusion-comet/pull/5803) (andygrove) +- perf: vectorize the native map lookup behind element_at and GetMapValue [#5806](https://github.com/apache/datafusion-comet/pull/5806) (andygrove) +- perf: compile user regex patterns once per planned expression [#5612](https://github.com/apache/datafusion-comet/pull/5612) (dwsmith1983) +- perf: batch the nested-element list hash for flat struct elements [#5778](https://github.com/apache/datafusion-comet/pull/5778) (viirya) +- perf: optimize `map_sort` singleton normalization (18x faster) [#5887](https://github.com/apache/datafusion-comet/pull/5887) (viirya) +- perf: optimize `map_sort` for multi-entry string maps (up to 3x faster) [#5901](https://github.com/apache/datafusion-comet/pull/5901) (viirya) +- perf: skip calendar reconstruction in hour/minute/second and dayofweek/weekday [#5771](https://github.com/apache/datafusion-comet/pull/5771) (peterxcli) +- perf: reuse zstd compression contexts across shuffle blocks [#5565](https://github.com/apache/datafusion-comet/pull/5565) (dwsmith1983) +- perf: cache parsed plan data across a stage tasks [#5615](https://github.com/apache/datafusion-comet/pull/5615) (dwsmith1983) +- perf: spill every shuffle partition of a task into one file [#5916](https://github.com/apache/datafusion-comet/pull/5916) (peterxcli) +- perf: decode shuffle blocks against a cached schema instead of re-parsing per block [#5809](https://github.com/apache/datafusion-comet/pull/5809) (peterxcli) +- perf: project cached batches by buffer selection, prune on collated strings [#5543](https://github.com/apache/datafusion-comet/pull/5543) (andygrove) +- perf: make one thread-local access per tracked allocation [#6166](https://github.com/apache/datafusion-comet/pull/6166) (andygrove) +- perf: reuse quantile summary buffers during merge [#4932](https://github.com/apache/datafusion-comet/pull/4932) (peterxcli) +- perf: optimize list_extract without defaults using Arrow take [#5174](https://github.com/apache/datafusion-comet/pull/5174) (peterxcli) +- perf: track decimal overflow without rescanning results [#5044](https://github.com/apache/datafusion-comet/pull/5044) (peterxcli) + +**Implemented enhancements:** + +- feat: support `explode_outer` [#5192](https://github.com/apache/datafusion-comet/pull/5192) (comphead) +- feat: support timestampadd and timestampdiff via codegen dispatch [#5030](https://github.com/apache/datafusion-comet/pull/5030) (andygrove) +- feat: support _metadata constant columns in native Parquet scan [#5237](https://github.com/apache/datafusion-comet/pull/5237) (mbutrovich) +- feat: add `make_interval` support (codegen dispatch + native) [#5039](https://github.com/apache/datafusion-comet/pull/5039) (peterxcli) +- feat: Optionally split the Iceberg V2 write operator into distinct writer and committer operations [#4658](https://github.com/apache/datafusion-comet/pull/4658) (jordepic) +- feat: detect Iceberg V2 writes and emit fall-back reasons [#5298](https://github.com/apache/datafusion-comet/pull/5298) (jordepic) +- feat: remove native cast from boolean to decimal [#5185](https://github.com/apache/datafusion-comet/pull/5185) (andygrove) +- feat: add micro benchmark runner and EC2 guide [#5374](https://github.com/apache/datafusion-comet/pull/5374) (andygrove) +- feat: build gate + inert wiring for contrib Delta scans [Delta contrib split, part 2] [#4952](https://github.com/apache/datafusion-comet/pull/4952) (schenksj) +- feat: Support native scans with unprojected Spark 4 VARIANT columns [#5377](https://github.com/apache/datafusion-comet/pull/5377) (sunchao) +- feat: expose native aggregate spill and memory metrics [#5423](https://github.com/apache/datafusion-comet/pull/5423) (sunchao) +- feat: support `WindowGroupLimitExec` [#4870](https://github.com/apache/datafusion-comet/pull/4870) (comphead) +- feat: add RSS partition writer and task-owned JNI callback (1/n) [#5473](https://github.com/apache/datafusion-comet/pull/5473) (pingzh) +- feat: native `spark_unbase64` kernel [#5451](https://github.com/apache/datafusion-comet/pull/5451) (0lai0) +- feat: add partition writer destinations to native shuffle plans (2/n) [#5476](https://github.com/apache/datafusion-comet/pull/5476) (pingzh) +- feat: add destination-aware native shuffle execution (3/n) [#5481](https://github.com/apache/datafusion-comet/pull/5481) (pingzh) +- feat: bind task-owned RSS callbacks to native shuffle plans (4/n) [#5491](https://github.com/apache/datafusion-comet/pull/5491) (pingzh) +- feat: add Celeborn shuffle manager and partition pusher (5/n) [#5501](https://github.com/apache/datafusion-comet/pull/5501) (pingzh) +- feat: complete Celeborn map-side shuffle push lifecycle (6/n) [#5513](https://github.com/apache/datafusion-comet/pull/5513) (pingzh) +- feat: add experimental native support for in-memory cache, disabled by default [#5051](https://github.com/apache/datafusion-comet/pull/5051) (andygrove) +- feat: add raw Celeborn native shuffle reader (7/n) [#5531](https://github.com/apache/datafusion-comet/pull/5531) (pingzh) +- feat: support Map for `CreateArray` literal [#5452](https://github.com/apache/datafusion-comet/pull/5452) (comphead) +- feat: route `round` on float/double through the codegen dispatcher [#5600](https://github.com/apache/datafusion-comet/pull/5600) (andygrove) +- feat: enable native-only Celeborn shuffle planning (8/n) [#5537](https://github.com/apache/datafusion-comet/pull/5537) (pingzh) +- feat: support unicode case sensitive field names for reading parquet [#5602](https://github.com/apache/datafusion-comet/pull/5602) (comphead) +- feat: implement native Iceberg V2 writer via iceberg-rust [#5361](https://github.com/apache/datafusion-comet/pull/5361) (jordepic) +- feat: Support Iceberg system functions (bucket, truncate, years/months/days/hours) natively [#5638](https://github.com/apache/datafusion-comet/pull/5638) (andygrove) +- feat: implement regr_slope, regr_intercept, regr_r2, regr_sxx, regr_syy, regr_sxy aggregates [#4775](https://github.com/apache/datafusion-comet/pull/4775) (andygrove) +- feat: carry VariantType identity through schema serialization [#5631](https://github.com/apache/datafusion-comet/pull/5631) (peterxcli) +- feat: route unrecognized StaticInvoke and Invoke through the codegen dispatcher [#5692](https://github.com/apache/datafusion-comet/pull/5692) (andygrove) +- feat: support S3 compliant filesystems [#5314](https://github.com/apache/datafusion-comet/pull/5314) (comphead) +- feat: add native spark_sequence kernel for integral element types [#5614](https://github.com/apache/datafusion-comet/pull/5614) (0lai0) +- feat: support nested types as native shuffle hash partitioning keys [#5567](https://github.com/apache/datafusion-comet/pull/5567) (viirya) +- feat: route next_day and levenshtein collated input through the codegen dispatcher [#5720](https://github.com/apache/datafusion-comet/pull/5720) (0lai0) +- chore: add_benches hash function aggregators [#5730](https://github.com/apache/datafusion-comet/pull/5730) (coderfender) +- test: cover Spark-to-Arrow batch conversion edge cases [#5713](https://github.com/apache/datafusion-comet/pull/5713) (peterxcli) +- feat: normalize marked Variant arrays at the native Parquet boundary [#5715](https://github.com/apache/datafusion-comet/pull/5715) (peterxcli) +- feat: support native concat_ws with string arrays [#5725](https://github.com/apache/datafusion-comet/pull/5725) (peterxcli) +- feat: native dynamic filter pushdown for hash join into Parquet scans [#5699](https://github.com/apache/datafusion-comet/pull/5699) (pingzh) +- feat: enable codegen dispatch for `lpad` and `rpad` [#5764](https://github.com/apache/datafusion-comet/pull/5764) (Satyr09) +- feat: address remaining issues for `CreateArray` [#5766](https://github.com/apache/datafusion-comet/pull/5766) (comphead) +- ci: stop non-gating labels from skipping Preflight [#5784](https://github.com/apache/datafusion-comet/pull/5784) (comphead) +- feat: route `abs` on interval types through the codegen dispatcher [#5622](https://github.com/apache/datafusion-comet/pull/5622) (kazantsev-maksim) +- ci: shard Iceberg Spark tests across four runners [#5459](https://github.com/apache/datafusion-comet/pull/5459) (sunchao) +- test: expand ANSI coverage for round, conv and elt [#5799](https://github.com/apache/datafusion-comet/pull/5799) (rich7420) +- test: strengthen ANSI exception assertions [#5800](https://github.com/apache/datafusion-comet/pull/5800) (rich7420) +- chore: Improve network retry configuration for maven and artifact upload [#5782](https://github.com/apache/datafusion-comet/pull/5782) (comphead) +- ci: gate the Delta contrib build on symbols, not on libcomet size [#5827](https://github.com/apache/datafusion-comet/pull/5827) (andygrove) +- test: add helpers to assert whether an expression ran natively or via codegen dispatch [#5610](https://github.com/apache/datafusion-comet/pull/5610) (andygrove) +- feat: support Spark encode expression via codegen dispatch [#5037](https://github.com/apache/datafusion-comet/pull/5037) (andygrove) +- feat: support native aggregate function `mode` [#4782](https://github.com/apache/datafusion-comet/pull/4782) (andygrove) +- test: name the whole dispatched subtree in the decimal promotion assertion [#5849](https://github.com/apache/datafusion-comet/pull/5849) (andygrove) +- feat: route translate and to_csv through codegen dispatch by default [#5032](https://github.com/apache/datafusion-comet/pull/5032) (andygrove) +- feat: expose native Parquet scan I/O and read-amplification metrics [#5453](https://github.com/apache/datafusion-comet/pull/5453) (sunchao) +- ci: move the job routing policy out of ci.yml expressions and into compute-changes.py [#5850](https://github.com/apache/datafusion-comet/pull/5850) (andygrove) +- feat: support max_by and min_by aggregate expressions [#4817](https://github.com/apache/datafusion-comet/pull/4817) (andygrove) +- ci: add a Required Checks aggregator job so main can have a required status check [#5842](https://github.com/apache/datafusion-comet/pull/5842) (andygrove) +- feat: adapt Parquet storage for Variant projection [#5794](https://github.com/apache/datafusion-comet/pull/5794) (peterxcli) +- ci: move the Spark 3.4/3.5/4.0, Iceberg, macOS and benchmark suites behind a merge queue [#5843](https://github.com/apache/datafusion-comet/pull/5843) (andygrove) +- feat(iceberg): Iceberg table format V3: apply deletion vector on reads [#5853](https://github.com/apache/datafusion-comet/pull/5853) (mbutrovich) +- test: restore ANSI array access error coverage [#5798](https://github.com/apache/datafusion-comet/pull/5798) (rich7420) +- test: cover native memory accounting boundaries [#5856](https://github.com/apache/datafusion-comet/pull/5856) (rich7420) +- test: cover date maps in Parquet temporal fuzz tests [#5877](https://github.com/apache/datafusion-comet/pull/5877) (rich7420) +- feat: Enable adaptive partial aggregation for eligible native shuffle plans [#5723](https://github.com/apache/datafusion-comet/pull/5723) (peterxcli) +- ci: move the Spark 4.1 sql_hive shards behind the merge queue [#5871](https://github.com/apache/datafusion-comet/pull/5871) (andygrove) +- ci: retry the Maven wrapper bootstrap in every job that calls ./mvnw directly [#5852](https://github.com/apache/datafusion-comet/pull/5852) (andygrove) +- test: cover slice over expression-produced non-null element arrays (#… [#5839](https://github.com/apache/datafusion-comet/pull/5839) (sam-1112) +- test: run the libhdfs suite manually instead of in CI [#5892](https://github.com/apache/datafusion-comet/pull/5892) (andygrove) +- feat: Add Lance contrib build gate [#5728](https://github.com/apache/datafusion-comet/pull/5728) (wirybeaver) +- chore: drop support for JDK 11 [#5897](https://github.com/apache/datafusion-comet/pull/5897) (manuzhang) +- test: cover regex expression routing configurations [#5917](https://github.com/apache/datafusion-comet/pull/5917) (rich7420) +- test: cover round expression routing configurations [#5878](https://github.com/apache/datafusion-comet/pull/5878) (rich7420) +- test: cover to_csv routing configurations [#5921](https://github.com/apache/datafusion-comet/pull/5921) (rich7420) +- test: cover interval expression routing configurations [#5920](https://github.com/apache/datafusion-comet/pull/5920) (rich7420) +- test: cover array and map expression routing configurations [#5918](https://github.com/apache/datafusion-comet/pull/5918) (rich7420) +- test: make the cancelled Iceberg abort test yield deterministically [#5919](https://github.com/apache/datafusion-comet/pull/5919) (andygrove) +- chore: deprecate Spark 3.4 rather than removing it in 1.1.0 [#5885](https://github.com/apache/datafusion-comet/pull/5885) (andygrove) +- test: restore the TPC-H suite’s 2 GiB off-heap budget [#5904](https://github.com/apache/datafusion-comet/pull/5904) (ErikBPF) +- test: cover string expression routing and native opt-in [#5915](https://github.com/apache/datafusion-comet/pull/5915) (rich7420) +- chore: run only the cache-writing jobs on push to main [#5930](https://github.com/apache/datafusion-comet/pull/5930) (andygrove) +- ci: publish nightly SNAPSHOT jars to repository.apache.org [#5902](https://github.com/apache/datafusion-comet/pull/5902) (andygrove) +- ci: fold the Delta build gate and PyArrow UDF suite into the merge queue tiers [#5926](https://github.com/apache/datafusion-comet/pull/5926) (andygrove) +- refactor: use Arrow decimal precision validation [#5160](https://github.com/apache/datafusion-comet/pull/5160) (Smallfu666) +- ci: shrink the pull request tier to the Linux build on the default Spark profile [#5939](https://github.com/apache/datafusion-comet/pull/5939) (andygrove) +- ci: run the non-default Spark and Iceberg suites nightly instead of in the merge queue [#5963](https://github.com/apache/datafusion-comet/pull/5963) (andygrove) +- test: cover lpad and rpad routing configurations [#5895](https://github.com/apache/datafusion-comet/pull/5895) (rich7420) +- test: preserve Spark SQL baselines for ordering-sensitive fixtures [#5914](https://github.com/apache/datafusion-comet/pull/5914) (rich7420) +- test: cover json_array_length routing configurations [#5940](https://github.com/apache/datafusion-comet/pull/5940) (peterxcli) +- test: cover SQL fixture metadata and statement parsing [#5941](https://github.com/apache/datafusion-comet/pull/5941) (peterxcli) +- test: cover crc32 on binary inputs [#5942](https://github.com/apache/datafusion-comet/pull/5942) (peterxcli) +- test: cover datetime and timezone routing configurations [#5950](https://github.com/apache/datafusion-comet/pull/5950) (rich7420) +- test: cover collation and predicate routing configurations [#5951](https://github.com/apache/datafusion-comet/pull/5951) (rich7420) +- test: cover array_intersect routing configurations [#5952](https://github.com/apache/datafusion-comet/pull/5952) (rich7420) +- test: guard native plan equality against omitted parameters [#5953](https://github.com/apache/datafusion-comet/pull/5953) (rich7420) +- test: expand replace compatibility regression coverage [#5409](https://github.com/apache/datafusion-comet/pull/5409) (sam-1112) +- feat: add native allocation accounting for memory observability [#5934](https://github.com/apache/datafusion-comet/pull/5934) (andygrove) +- feat: run rlike natively by default for Java-equivalent literal patterns [#5415](https://github.com/apache/datafusion-comet/pull/5415) (sam-1112) +- ci: write large actions/cache entries only on push to main [#5973](https://github.com/apache/datafusion-comet/pull/5973) (andygrove) +- refactor: extract shared runtime filter components [#5937](https://github.com/apache/datafusion-comet/pull/5937) (pingzh) +- test: skip the Spark RocksDB state-store suites when Comet is enabled [#5987](https://github.com/apache/datafusion-comet/pull/5987) (comphead) +- chore: bench nondeterministic and json kernels [#5989](https://github.com/apache/datafusion-comet/pull/5989) (coderfender) +- feat: add BinaryType support for SortMergeJoin [#5928](https://github.com/apache/datafusion-comet/pull/5928) (xhumanoid) +- refactor: replace remaining hand-rolled loops with Arrow kernels [#5367](https://github.com/apache/datafusion-comet/pull/5367) (0lai0) +- feat: support Spark 4 EmptyRelationExec as a native input [#5821](https://github.com/apache/datafusion-comet/pull/5821) (jeffw13) +- chore: Bench agg (welford) stats [#5988](https://github.com/apache/datafusion-comet/pull/5988) (coderfender) +- ci: rename the umbrella workflow from CI to Comet CI [#6003](https://github.com/apache/datafusion-comet/pull/6003) (andygrove) +- ci: bootstrap Maven from the setup actions so every mvnw job is covered [#5881](https://github.com/apache/datafusion-comet/pull/5881) (andygrove) +- ci: add dev/local-ci.sh to run the Spark SQL and Iceberg suites locally [#5974](https://github.com/apache/datafusion-comet/pull/5974) (comphead) +- feat: narrow strict floating-point admission for scalar sort keys [#5981](https://github.com/apache/datafusion-comet/pull/5981) (0lai0) +- feat: route compatible xxhash64 args through SparkXxhash64 [#5960](https://github.com/apache/datafusion-comet/pull/5960) (sam-1112) +- feat: support string arrays in Spark-to-Comet conversion [#5954](https://github.com/apache/datafusion-comet/pull/5954) (rich7420) +- feat: run length on binary input natively [#5874](https://github.com/apache/datafusion-comet/pull/5874) (dwsmith1983) +- chore: drop the redundant width_bucket shim registrations and guard serde uniqueness [#5873](https://github.com/apache/datafusion-comet/pull/5873) (dwsmith1983) +- feat: hook native Parquet writes into Spark's WriteFilesExec seam on Spark 4.0+ [#5763](https://github.com/apache/datafusion-comet/pull/5763) (andygrove) +- feat(iceberg): report native Iceberg scan planning metrics and scan time in the Spark UI [#6027](https://github.com/apache/datafusion-comet/pull/6027) (parthchandra) +- test: guard page skipping in the native Iceberg scan [#6040](https://github.com/apache/datafusion-comet/pull/6040) (dwsmith1983) +- test: stop claiming Spark agrees on signed-zero array literals [#6055](https://github.com/apache/datafusion-comet/pull/6055) (cestercian) +- feat: trace Arrow memory held on the JVM side [#6048](https://github.com/apache/datafusion-comet/pull/6048) (andygrove) +- feat: unix_timestamp codegen dispatch for strings, and fix pre-epoch fractional rounding [#5789](https://github.com/apache/datafusion-comet/pull/5789) (Satyr09) +- chore: remove JaCoCo from the build [#6084](https://github.com/apache/datafusion-comet/pull/6084) (andygrove) +- refactor: compose CometScanRule and CometExecRule into a single CometRule [#6082](https://github.com/apache/datafusion-comet/pull/6082) (andygrove) +- chore: add a native Iceberg write benchmark [#6038](https://github.com/apache/datafusion-comet/pull/6038) (0lai0) +- ci: retry the Spark test pre-compile step on resolution failures [#6079](https://github.com/apache/datafusion-comet/pull/6079) (andygrove) +- ci: focus Miri on unsafe row and hash tests [#6072](https://github.com/apache/datafusion-comet/pull/6072) (rich7420) +- test: run the Iceberg Spark tests with the native Iceberg writer enabled [#5677](https://github.com/apache/datafusion-comet/pull/5677) (andygrove) +- feat: support listagg / string_agg aggregate (Spark 4.0+) [#4816](https://github.com/apache/datafusion-comet/pull/4816) (andygrove) +- ci: run label-triggered CI as a separate workflow [#6161](https://github.com/apache/datafusion-comet/pull/6161) (andygrove) +- feat: always count native allocations and log executor native memory usage [#6162](https://github.com/apache/datafusion-comet/pull/6162) (andygrove) +- feat: use DataFusion unnest_outer instead of Comet ListEmptyToNullExpr [#6132](https://github.com/apache/datafusion-comet/pull/6132) (comphead) +- chore: recommend --force-with-lease when updating PR branches [#6171](https://github.com/apache/datafusion-comet/pull/6171) (mizulun) +- feat: positional round robin shuffle keyed on a row ordinal [#6095](https://github.com/apache/datafusion-comet/pull/6095) (andygrove) +- chore: deprecate spark.comet.exec.memoryPool.fraction [#6163](https://github.com/apache/datafusion-comet/pull/6163) (andygrove) +- feat: add spark.comet.explain.planOnly.enabled [#5394](https://github.com/apache/datafusion-comet/pull/5394) (andygrove) +- feat: support direct Variant projection in native Parquet scans [#5868](https://github.com/apache/datafusion-comet/pull/5868) (peterxcli) +- test: cover JSON and cast expression routing [#6068](https://github.com/apache/datafusion-comet/pull/6068) (rich7420) +- feat: remove memory accounting from Comet's on-heap mode [#6066](https://github.com/apache/datafusion-comet/pull/6066) (andygrove) +- test: accept CometHashAggregateExec in the Spark 4.0 CollationSuite hash agg check [#6220](https://github.com/apache/datafusion-comet/pull/6220) (andygrove) +- feat: per-location credentials for the S3 credential SPI [#6031](https://github.com/apache/datafusion-comet/pull/6031) (snmvaughan) +- refactor: centralize data type support predicates [#5025](https://github.com/apache/datafusion-comet/pull/5025) (peterxcli) +- chore: [branch-1.1] change version from 1.1.0-SNAPSHOT to 1.1.0 [#6236](https://github.com/apache/datafusion-comet/pull/6236) (andygrove) +- feat: [branch-1.1] built-in S3 credential provider adapters for the native Parquet scan (#6023) [#6318](https://github.com/apache/datafusion-comet/pull/6318) (andygrove) + +**Documentation updates:** + +- docs: add note about run-iceberg-tests label in CI [#5247](https://github.com/apache/datafusion-comet/pull/5247) (mbutrovich) +- docs: add 1.0.0 TPC-DS benchmark results, remove TPC-H [#5284](https://github.com/apache/datafusion-comet/pull/5284) (mbutrovich) +- docs: add 1.0.0 changelog to main [#5263](https://github.com/apache/datafusion-comet/pull/5263) (andygrove) +- docs: correct Spark 4.2 version and CI test status in installation guide [#5315](https://github.com/apache/datafusion-comet/pull/5315) (andygrove) +- docs: add suggest-native-expression skill for assessing native expression candidates [#5348](https://github.com/apache/datafusion-comet/pull/5348) (andygrove) +- docs: document C2R cost for wide/nested schemas in tuning guide [#5458](https://github.com/apache/datafusion-comet/pull/5458) (DebadityaHait) +- docs: update stale interval multiplication note in expressions.md [#5518](https://github.com/apache/datafusion-comet/pull/5518) (peterxcli) +- docs: add contributing guide link [#5521](https://github.com/apache/datafusion-comet/pull/5521) (Dharan-K) +- doc: fix benchmark examples for MacOS [#5522](https://github.com/apache/datafusion-comet/pull/5522) (xhumanoid) +- docs: add comet meeting link [#5621](https://github.com/apache/datafusion-comet/pull/5621) (coderfender) +- docs: add a contributor guide page for CI and the merge queue [#5863](https://github.com/apache/datafusion-comet/pull/5863) (andygrove) +- docs: add contributor guide page on memory management [#5933](https://github.com/apache/datafusion-comet/pull/5933) (andygrove) +- docs: how to check the scheduled CI runs are actually running [#6000](https://github.com/apache/datafusion-comet/pull/6000) (andygrove) +- docs: explain allocator hazards and diagram where memory is allocated [#6014](https://github.com/apache/datafusion-comet/pull/6014) (andygrove) +- docs: pre-render mermaid diagrams to SVG at build time [#6021](https://github.com/apache/datafusion-comet/pull/6021) (andygrove) +- docs: split the PR review skill by area and correct the shuffle contributor docs [#6018](https://github.com/apache/datafusion-comet/pull/6018) (andygrove) +- docs: correct the stale range-partitioning strict floating-point rule [#6049](https://github.com/apache/datafusion-comet/pull/6049) (andygrove) +- docs: recommend setting spark.executor.memoryOverhead alongside off-heap memory [#6051](https://github.com/apache/datafusion-comet/pull/6051) (andygrove) +- docs: exempt testing-category and internal configs from the versioning policy [#6089](https://github.com/apache/datafusion-comet/pull/6089) (andygrove) +- docs: make doc changes weekly comet sync [#6160](https://github.com/apache/datafusion-comet/pull/6160) (coderfender) +- docs: correct the plugin and shuffle sections of the plugin overview [#6197](https://github.com/apache/datafusion-comet/pull/6197) (andygrove) +- docs: fix stale and missing native Iceberg write details [#6150](https://github.com/apache/datafusion-comet/pull/6150) (andygrove) +- docs: add an Iceberg writes contributor guide and review skill [#6149](https://github.com/apache/datafusion-comet/pull/6149) (andygrove) +- docs: correct which operators run separate native plans in a task [#6194](https://github.com/apache/datafusion-comet/pull/6194) (andygrove) +- docs: update the user guide for the 1.1.0 release [#6168](https://github.com/apache/datafusion-comet/pull/6168) (andygrove) +- docs: [branch-1.1] backport the 1.1.0 upgrade notes and tuning guide updates (#6237, #6244, #6248) [#6265](https://github.com/apache/datafusion-comet/pull/6265) (andygrove) +- ci: [branch-1.1] run every tier on release-branch pull requests, and the full suite before an RC (#6218) [#6284](https://github.com/apache/datafusion-comet/pull/6284) (andygrove) +- docs: [branch-1.1] generate release docs for 1.1.0 [#6344](https://github.com/apache/datafusion-comet/pull/6344) (andygrove) + +**Other:** + +- chore: start 1.1.0 development [#5242](https://github.com/apache/datafusion-comet/pull/5242) (andygrove) +- refactor: replace hand-coded rollup of expression fallback reasons onto operators [#5236](https://github.com/apache/datafusion-comet/pull/5236) (andygrove) +- test: rename CometCastSuite to CometNativeCastSuite [#5268](https://github.com/apache/datafusion-comet/pull/5268) (andygrove) +- refactor: use arity helper for Int to Decimal128 reinterpretation [#5193](https://github.com/apache/datafusion-comet/pull/5193) (0lai0) +- test: add guard for Iceberg version on Variant fallback test [#5278](https://github.com/apache/datafusion-comet/pull/5278) (mbutrovich) +- chore: update documentation links for 1.0.0 release [#5290](https://github.com/apache/datafusion-comet/pull/5290) (andygrove) +- chore(deps): bump reqsign-core from 3.2.0 to 3.2.1 in /native in the all-other-cargo-deps group [#5287](https://github.com/apache/datafusion-comet/pull/5287) (dependabot[bot]) +- chore(deps): bump object_store_opendal from 0.57.0 to 0.58.0 in /native [#5289](https://github.com/apache/datafusion-comet/pull/5289) (dependabot[bot]) +- chore(deps): bump the codeql-actions group with 2 updates [#5286](https://github.com/apache/datafusion-comet/pull/5286) (dependabot[bot]) +- bug: Revert "chore(deps): bump object_store_opendal from 0.57.0 to 0.58.0 in /native" [#5332](https://github.com/apache/datafusion-comet/pull/5332) (coderfender) +- test: cover the narrowing direction of cast_and_stamp_schema [#5285](https://github.com/apache/datafusion-comet/pull/5285) (andygrove) +- chore(deps): bump opendal from 0.57.0 to 0.58.1 in /native [#5324](https://github.com/apache/datafusion-comet/pull/5324) (manuzhang) +- chore: respect Cargo parallelism settings for native release builds [#5344](https://github.com/apache/datafusion-comet/pull/5344) (pingzh) +- test: make expression benchmark harness fair and reproducible [#5371](https://github.com/apache/datafusion-comet/pull/5371) (andygrove) +- chore(deps): bump the all-other-cargo-deps group in /native with 5 updates [#5360](https://github.com/apache/datafusion-comet/pull/5360) (dependabot[bot]) +- test: fix vacuous signed-zero coverage in SQL file tests [#5393](https://github.com/apache/datafusion-comet/pull/5393) (sam-1112) +- chore: fix clippy warnings for Rust 1.98 [#5400](https://github.com/apache/datafusion-comet/pull/5400) (ywskycn) +- chore(deps): bump the codeql-actions group with 2 updates [#5405](https://github.com/apache/datafusion-comet/pull/5405) (dependabot[bot]) +- test: strengthen signed-zero SQL assertions [#5404](https://github.com/apache/datafusion-comet/pull/5404) (sunchao) +- chore(deps): bump actions/checkout from 6 to 7 [#5406](https://github.com/apache/datafusion-comet/pull/5406) (dependabot[bot]) +- test: add expression fallback-invariance suite [#5329](https://github.com/apache/datafusion-comet/pull/5329) (4ktLuffy) +- refactor: delegate ANSI integer arithmetic to arrow checked kernels [#5280](https://github.com/apache/datafusion-comet/pull/5280) (kazantsev-maksim) +- test: docs and test-coverage hardening for native collect_list / collect_set [#5055](https://github.com/apache/datafusion-comet/pull/5055) (andygrove) +- test: deduplicate Throwable cause-chain traversal [#5441](https://github.com/apache/datafusion-comet/pull/5441) (peterxcli) +- refactor: use Arrow timezone type [#5129](https://github.com/apache/datafusion-comet/pull/5129) (Hashim1999164) +- chore: add md formatting to `make format` [#5460](https://github.com/apache/datafusion-comet/pull/5460) (comphead) +- test: fail unexpected vacuous fallback-invariance checks [#5417](https://github.com/apache/datafusion-comet/pull/5417) (sunchao) +- ci: cache Maven distributions and retry bootstrap downloads [#5422](https://github.com/apache/datafusion-comet/pull/5422) (sunchao) +- chore: add math benches [#5479](https://github.com/apache/datafusion-comet/pull/5479) (coderfender) +- chore: Bench string functions [#5492](https://github.com/apache/datafusion-comet/pull/5492) (coderfender) +- chore(deps): bump actions/setup-java from 5 to 6 [#5524](https://github.com/apache/datafusion-comet/pull/5524) (dependabot[bot]) +- chore(deps): bump the codeql-actions group with 2 updates [#5523](https://github.com/apache/datafusion-comet/pull/5523) (dependabot[bot]) +- chore: bench additional math function [#5520](https://github.com/apache/datafusion-comet/pull/5520) (coderfender) +- chore: remove redundant condition and stray println in shuffle code [#5562](https://github.com/apache/datafusion-comet/pull/5562) (viirya) +- test: cover struct data columns in native shuffle [#5564](https://github.com/apache/datafusion-comet/pull/5564) (viirya) +- chore: rename .claude directory to vendor-neutral .ai [#5594](https://github.com/apache/datafusion-comet/pull/5594) (andygrove) +- chore: bench additional scalar functions [#5598](https://github.com/apache/datafusion-comet/pull/5598) (coderfender) +- test: verify Celeborn reflection compatibility [#5604](https://github.com/apache/datafusion-comet/pull/5604) (pingzh) +- test: enable native path in lower/upper_enabled sql fixtures [#5619](https://github.com/apache/datafusion-comet/pull/5619) (cestercian) +- chore: add benches datetime [#5620](https://github.com/apache/datafusion-comet/pull/5620) (coderfender) +- chore: move dead and defensive serde guards out of convert [#5595](https://github.com/apache/datafusion-comet/pull/5595) (andygrove) +- test: exercise the Iceberg write split-operator plan in Iceberg's own suites [#5640](https://github.com/apache/datafusion-comet/pull/5640) (andygrove) +- chore(deps): bump the codeql-actions group with 2 updates [#5669](https://github.com/apache/datafusion-comet/pull/5669) (dependabot[bot]) +- test: add explode operator microbenchmark [#5381](https://github.com/apache/datafusion-comet/pull/5381) (andygrove) +- deps: bump DataFusion 55.0 and Arrow/Parquet 59.2 [#5262](https://github.com/apache/datafusion-comet/pull/5262) (mbutrovich) +- chore: add benches array functions [#5700](https://github.com/apache/datafusion-comet/pull/5700) (coderfender) +- test: restore Comet coverage for recursive HAVING and ORDER BY [#5755](https://github.com/apache/datafusion-comet/pull/5755) (rich7420) +- test: restore Parquet V2 writer and delta encoding coverage [#5760](https://github.com/apache/datafusion-comet/pull/5760) (rich7420) +- chore: Add benches for datetime funcs [#5767](https://github.com/apache/datafusion-comet/pull/5767) (coderfender) +- bench: add a benchmark for the Spark hash kernels [#5765](https://github.com/apache/datafusion-comet/pull/5765) (viirya) +- test: restore Spark 4.1 Variant shredding suites [#5745](https://github.com/apache/datafusion-comet/pull/5745) (rich7420) +- refactor: share one helper for pushing a struct's null mask into its children [#5769](https://github.com/apache/datafusion-comet/pull/5769) (viirya) +- ci: label pull requests by changed paths and title prefix [#5762](https://github.com/apache/datafusion-comet/pull/5762) (dwsmith1983) +- test: cover ambiguous exact nested Parquet field matches [#5751](https://github.com/apache/datafusion-comet/pull/5751) (peterxcli) +- bench: measure nested types as native shuffle hash partitioning keys [#5788](https://github.com/apache/datafusion-comet/pull/5788) (viirya) +- deps: bump to datafusion 55.1.0 [#5865](https://github.com/apache/datafusion-comet/pull/5865) (comphead) +- bench: isolate map normalization and nested key hashing [#5822](https://github.com/apache/datafusion-comet/pull/5822) (viirya) +- bench: add a shuffle read benchmark covering the per-block schema parse [#5805](https://github.com/apache/datafusion-comet/pull/5805) (peterxcli) +- chore(deps): bump actions/setup-java from 4 to 6 [#6012](https://github.com/apache/datafusion-comet/pull/6012) (dependabot[bot]) +- chore(deps): bump the codeql-actions group with 2 updates [#6011](https://github.com/apache/datafusion-comet/pull/6011) (dependabot[bot]) +- deps: bump the iceberg-rust pin to bb1e4a4 and document why it is pinned [#6094](https://github.com/apache/datafusion-comet/pull/6094) (andygrove) + +## Credits + +Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor. + +``` + 128 Andy Grove + 61 Peter Lee + 26 Chao Sun + 26 KUAN-HAO HUANG + 19 Ping Zhang + 15 Oleks V + 13 Bhargava Vadlamani + 13 Liang-Chi Hsieh + 13 dustin + 11 dependabot[bot] + 10 ChenChen Lai + 7 sam-1112 + 6 Matt Butrovich + 5 Han-Yin Chang + 4 Erik Bogado + 4 Wei Yan + 3 Jordan Epstein + 3 Manu Zhang + 3 Parth Chandra + 2 Alexey + 2 Cestercian + 2 Daipayan Mukherjee + 2 Feng Zhang + 2 Kazantsev Maksim + 2 Steve Vaughan + 2 YuLun Mao + 1 Dharan-K + 1 Hashim Khan + 1 Henos D + 1 Jeff Wang + 1 LinSimon-901101 + 1 Scott Schenkein + 1 Stefan Wang + 1 Unik Dahal + 1 Viacheslav Inozemtsev + 1 Xuanyi Li + 1 debaditya + 1 divyank sameer shah + 1 hsiang-c + 1 liupoyi-1031 +``` + +Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release. diff --git a/docs/source/changelog/index.md b/docs/source/changelog/index.md index e79878f7e70..792dfa2df3b 100644 --- a/docs/source/changelog/index.md +++ b/docs/source/changelog/index.md @@ -24,6 +24,7 @@ Per-release change logs for Apache DataFusion Comet. ```{toctree} :maxdepth: 1 +1.1.0 <1.1.0> 1.0.0 <1.0.0> 0.17.1 <0.17.1> 0.17.0 <0.17.0>