Skip to content

Commit 42af516

Browse files
andygroveandygrove
committed
update changelog
1 parent 1b611ea commit 42af516

1 file changed

Lines changed: 44 additions & 83 deletions

File tree

‎dev/changelog/37.0.0.md‎

Lines changed: 44 additions & 83 deletions
Original file line numberDiff line numberDiff line change
@@ -1,29 +1,5 @@
1-
<!---
2-
Licensed to the Apache Software Foundation (ASF) under one
3-
or more contributor license agreements. See the NOTICE file
4-
distributed with this work for additional information
5-
regarding copyright ownership. The ASF licenses this file
6-
to you under the Apache License, Version 2.0 (the
7-
"License"); you may not use this file except in compliance
8-
with the License. You may obtain a copy of the License at
9-
10-
http://www.apache.org/licenses/LICENSE-2.0
11-
12-
Unless required by applicable law or agreed to in writing,
13-
software distributed under the License is distributed on an
14-
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15-
KIND, either express or implied. See the License for the
16-
specific language governing permissions and limitations
17-
under the License.
18-
-->
19-
20-
## [37.0.0](https://github.com/apache/arrow-datafusion/tree/37.0.0) (2024-03-19)
21-
22-
[Full Changelog](https://github.com/apache/arrow-datafusion/compare/36.0.0...37.0.0)
23-
241
**Breaking changes:**
252

26-
- refactor: Fix typo: Rename`PruningPredicate::allways_true()` to `PruningPredicate::always_true()` [#9283](https://github.com/apache/arrow-datafusion/pull/9283) (SteveLauC)
273
- refactor: Change `SchemaProvider::table` to return `Result<Option<..>` rather than `Option<..>` [#9307](https://github.com/apache/arrow-datafusion/pull/9307) (crepererum)
284
- feat: issue_9285: port builtin reg function into datafusion-function-\* (1/3 regexpmatch) [#9329](https://github.com/apache/arrow-datafusion/pull/9329) (Lordworms)
295
- Cache common plan properties to eliminate recursive calls in physical plan [#9346](https://github.com/apache/arrow-datafusion/pull/9346) (mustafasrepo)
@@ -34,14 +10,10 @@
3410

3511
**Performance related:**
3612

37-
- Implement specialized group values for single Uft8/LargeUtf8/Binary/LargeBinary column [#8827](https://github.com/apache/arrow-datafusion/pull/8827) (alamb)
3813
- perf: improve to_field performance [#9722](https://github.com/apache/arrow-datafusion/pull/9722) (haohuaijin)
3914

4015
**Implemented enhancements:**
4116

42-
- feat: Implement `FunctionRegistry::register_udwf` and `FunctionRegistry::register_udaf` for SessionContext, MemoryFunctionRegistry and TaskContext [#9226](https://github.com/apache/arrow-datafusion/pull/9226) (Lordworms)
43-
- feat: support `FixedSizeList` Type Coercion [#9108](https://github.com/apache/arrow-datafusion/pull/9108) (Weijun-H)
44-
- feat: expand `unnest` to accept arbitrary single array expression [#9342](https://github.com/apache/arrow-datafusion/pull/9342) (jonahgao)
4517
- feat: support for defining ARRAY columns in `CREATE TABLE` [#9381](https://github.com/apache/arrow-datafusion/pull/9381) (jonahgao)
4618
- feat: support `unnest` in FROM clause [#9355](https://github.com/apache/arrow-datafusion/pull/9355) (jonahgao)
4719
- feat: support nvl2 function [#9364](https://github.com/apache/arrow-datafusion/pull/9364) (guojidan)
@@ -56,16 +28,13 @@
5628
- feat: implement more expr_to_sql functionality [#9578](https://github.com/apache/arrow-datafusion/pull/9578) (devinjdangelo)
5729
- feat: implement aggregation and subquery plans to SQL [#9606](https://github.com/apache/arrow-datafusion/pull/9606) (devinjdangelo)
5830
- feat: track memory usage for recursive CTE, enable recursive CTEs by default [#9619](https://github.com/apache/arrow-datafusion/pull/9619) (jonahgao)
31+
- feat: Between expr to sql string [#9803](https://github.com/apache/arrow-datafusion/pull/9803) (sebastian2296)
32+
- feat: Expose `array_empty` and `list_empty` functions as alias of `empty` function [#9807](https://github.com/apache/arrow-datafusion/pull/9807) (erenavsarogullari)
33+
- feat: Not expr to string [#9802](https://github.com/apache/arrow-datafusion/pull/9802) (sebastian2296)
34+
- feat: pass SessionState not SessionConfig to FunctionFactory::create [#9837](https://github.com/apache/arrow-datafusion/pull/9837) (tshauck)
5935

6036
**Fixed bugs:**
6137

62-
- fix: issue #9130 substitute redundant columns when doing cross join [#9154](https://github.com/apache/arrow-datafusion/pull/9154) (Lordworms)
63-
- fix: fix `GLIBC_2.35' not found error in ci [#9243](https://github.com/apache/arrow-datafusion/pull/9243) (ZhengLin-Li)
64-
- fix: add `ArrayAndElementAndOptionalIndex` for proper casting in `array_position` [#9233](https://github.com/apache/arrow-datafusion/pull/9233) (tshauck)
65-
- fix: issue #8838 discard extra sort when sorted element is wrapped [#9127](https://github.com/apache/arrow-datafusion/pull/9127) (Lordworms)
66-
- fix: issue #9213 substitute ArrayAgg to NthValue to optimize query plan [#9295](https://github.com/apache/arrow-datafusion/pull/9295) (Lordworms)
67-
- fix: throw error when incursion happen in dataframe api [#9330](https://github.com/apache/arrow-datafusion/pull/9330) (Lordworms)
68-
- fix: flight examples [#9335](https://github.com/apache/arrow-datafusion/pull/9335) (codonnell)
6938
- fix: use `JoinSet` to make spawned tasks cancel-safe [#9318](https://github.com/apache/arrow-datafusion/pull/9318) (DDtKey)
7039
- fix: nvl function's return type [#9357](https://github.com/apache/arrow-datafusion/pull/9357) (guojidan)
7140
- fix: panic in isnan() when no args are given [#9377](https://github.com/apache/arrow-datafusion/pull/9377) (SteveLauC)
@@ -80,14 +49,11 @@
8049
- fix: incorrect null handling in `range` and `generate_series` [#9574](https://github.com/apache/arrow-datafusion/pull/9574) (jonahgao)
8150
- fix: recursive cte hangs on joins [#9687](https://github.com/apache/arrow-datafusion/pull/9687) (jonahgao)
8251
- fix: parallel parquet can underflow when max_record_batch_rows < execution.batch_size [#9737](https://github.com/apache/arrow-datafusion/pull/9737) (devinjdangelo)
52+
- fix: change placeholder errors from Internal to Plan [#9745](https://github.com/apache/arrow-datafusion/pull/9745) (erratic-pattern)
53+
- fix: ensure mutual compatibility of the two input schemas from recursive CTEs [#9795](https://github.com/apache/arrow-datafusion/pull/9795) (jonahgao)
8354

8455
**Documentation updates:**
8556

86-
- Prepare 36.0.0-rc1 [#9251](https://github.com/apache/arrow-datafusion/pull/9251) (andygrove)
87-
- Update Docs with Copy partition_by support [#9275](https://github.com/apache/arrow-datafusion/pull/9275) (devinjdangelo)
88-
- docs: update contributor guide (format toml/inte test) [#9301](https://github.com/apache/arrow-datafusion/pull/9301) (SteveLauC)
89-
- docs: document range() alias generate_series() [#9321](https://github.com/apache/arrow-datafusion/pull/9321) (SteveLauC)
90-
- docs: update parquet_sql_multiple_files.rs with a relative path ex [#9310](https://github.com/apache/arrow-datafusion/pull/9310) (tshauck)
9157
- docs: put flatten in top fn list [#9376](https://github.com/apache/arrow-datafusion/pull/9376) (SteveLauC)
9258
- Update documentation so list_to_string alias to point to array_to_string [#9374](https://github.com/apache/arrow-datafusion/pull/9374) (monkwire)
9359
- Uplift keys/dependencies to use more workspace inheritance [#9293](https://github.com/apache/arrow-datafusion/pull/9293) (Jefffrey)
@@ -106,49 +72,6 @@
10672

10773
**Merged pull requests:**
10874

109-
- Prepare 36.0.0-rc1 [#9251](https://github.com/apache/arrow-datafusion/pull/9251) (andygrove)
110-
- fix: issue #9130 substitute redundant columns when doing cross join [#9154](https://github.com/apache/arrow-datafusion/pull/9154) (Lordworms)
111-
- minor: Add more doc to clarify permutation satisfying input order in `InputOrderMode` [#9255](https://github.com/apache/arrow-datafusion/pull/9255) (viirya)
112-
- fix: fix `GLIBC_2.35' not found error in ci [#9243](https://github.com/apache/arrow-datafusion/pull/9243) (ZhengLin-Li)
113-
- fix: add `ArrayAndElementAndOptionalIndex` for proper casting in `array_position` [#9233](https://github.com/apache/arrow-datafusion/pull/9233) (tshauck)
114-
- Support Copy To Partitioned Files [#9240](https://github.com/apache/arrow-datafusion/pull/9240) (devinjdangelo)
115-
- Minior: Add negative tests about log function [#9245](https://github.com/apache/arrow-datafusion/pull/9245) (caicancai)
116-
- Minor: fix warning of 'Unnecessary braces in use statement' [#9271](https://github.com/apache/arrow-datafusion/pull/9271) (jonahgao)
117-
- Remove circular dependency [#9279](https://github.com/apache/arrow-datafusion/pull/9279) (andygrove)
118-
- chore: use taplo to format Cargo.toml manifests [#9263](https://github.com/apache/arrow-datafusion/pull/9263) (waynexia)
119-
- feat: Implement `FunctionRegistry::register_udwf` and `FunctionRegistry::register_udaf` for SessionContext, MemoryFunctionRegistry and TaskContext [#9226](https://github.com/apache/arrow-datafusion/pull/9226) (Lordworms)
120-
- Implement specialized group values for single Uft8/LargeUtf8/Binary/LargeBinary column [#8827](https://github.com/apache/arrow-datafusion/pull/8827) (alamb)
121-
- Fix untracked files after rustdoc run [#9264](https://github.com/apache/arrow-datafusion/pull/9264) (Omega359)
122-
- Support List for Array aggregate order and distinct [#9234](https://github.com/apache/arrow-datafusion/pull/9234) (jayzhan211)
123-
- fix: issue #8838 discard extra sort when sorted element is wrapped [#9127](https://github.com/apache/arrow-datafusion/pull/9127) (Lordworms)
124-
- refactor: Fix typo: Rename`PruningPredicate::allways_true()` to `PruningPredicate::always_true()` [#9283](https://github.com/apache/arrow-datafusion/pull/9283) (SteveLauC)
125-
- Improve datafuson-proto and datafusion-substrait docs [#9260](https://github.com/apache/arrow-datafusion/pull/9260) (alamb)
126-
- Minor: Add ticket reference to test [#9270](https://github.com/apache/arrow-datafusion/pull/9270) (alamb)
127-
- Update Docs with Copy partition_by support [#9275](https://github.com/apache/arrow-datafusion/pull/9275) (devinjdangelo)
128-
- Continuing Work of [PR9127]: improved using existing orderings [#9273](https://github.com/apache/arrow-datafusion/pull/9273) (mustafasrepo)
129-
- Fix partition columns when they they are not at the end of the schema (fix Removing Partition Columns from RecordBatch) [#9276](https://github.com/apache/arrow-datafusion/pull/9276) (devinjdangelo)
130-
- Minor: rename datafusion/proto from `README-dev.md` to `CONTRIBUTING.md` [#9300](https://github.com/apache/arrow-datafusion/pull/9300) (alamb)
131-
- Fix array_ndims error handling [#9272](https://github.com/apache/arrow-datafusion/pull/9272) (Weijun-H)
132-
- Fix toml format script [#9309](https://github.com/apache/arrow-datafusion/pull/9309) (Omega359)
133-
- docs: update contributor guide (format toml/inte test) [#9301](https://github.com/apache/arrow-datafusion/pull/9301) (SteveLauC)
134-
- [MINOR]: Limit stream replace with slice [#9303](https://github.com/apache/arrow-datafusion/pull/9303) (mustafasrepo)
135-
- Support IGNORE NULLS for LAG window function [#9221](https://github.com/apache/arrow-datafusion/pull/9221) (comphead)
136-
- fix: issue #9213 substitute ArrayAgg to NthValue to optimize query plan [#9295](https://github.com/apache/arrow-datafusion/pull/9295) (Lordworms)
137-
- Minor: Adding missing fields to debug for `ScalarFunctionExpr` [#9325](https://github.com/apache/arrow-datafusion/pull/9325) (comphead)
138-
- docs: document range() alias generate_series() [#9321](https://github.com/apache/arrow-datafusion/pull/9321) (SteveLauC)
139-
- chore: statically link xz2 [#9311](https://github.com/apache/arrow-datafusion/pull/9311) (universalmind303)
140-
- fix: throw error when incursion happen in dataframe api [#9330](https://github.com/apache/arrow-datafusion/pull/9330) (Lordworms)
141-
- Support CopyTo::partition_by in datafusion proto [#9306](https://github.com/apache/arrow-datafusion/pull/9306) (PhVHoang)
142-
- Add test to prevent circular dependencies from being added [#9292](https://github.com/apache/arrow-datafusion/pull/9292) (andygrove)
143-
- Substrait: Support ScalarUDF [#9337](https://github.com/apache/arrow-datafusion/pull/9337) (jayzhan211)
144-
- Make agg_func_substitute test deterministic [#9340](https://github.com/apache/arrow-datafusion/pull/9340) (andygrove)
145-
- feat: support `FixedSizeList` Type Coercion [#9108](https://github.com/apache/arrow-datafusion/pull/9108) (Weijun-H)
146-
- feat: expand `unnest` to accept arbitrary single array expression [#9342](https://github.com/apache/arrow-datafusion/pull/9342) (jonahgao)
147-
- fix: flight examples [#9335](https://github.com/apache/arrow-datafusion/pull/9335) (codonnell)
148-
- docs: update parquet_sql_multiple_files.rs with a relative path ex [#9310](https://github.com/apache/arrow-datafusion/pull/9310) (tshauck)
149-
- tests: add tests for writing hive-partitioned parquet [#9316](https://github.com/apache/arrow-datafusion/pull/9316) (tshauck)
150-
- feature: support nvl(ifnull) function [#9284](https://github.com/apache/arrow-datafusion/pull/9284) (guojidan)
151-
- Move abs to datafusion_functions [#9313](https://github.com/apache/arrow-datafusion/pull/9313) (yyy1000)
15275
- refactor: Change `SchemaProvider::table` to return `Result<Option<..>` rather than `Option<..>` [#9307](https://github.com/apache/arrow-datafusion/pull/9307) (crepererum)
15376
- fix write_partitioned_parquet_results test case bug [#9360](https://github.com/apache/arrow-datafusion/pull/9360) (guojidan)
15477
- fix: use `JoinSet` to make spawned tasks cancel-safe [#9318](https://github.com/apache/arrow-datafusion/pull/9318) (DDtKey)
@@ -361,3 +284,41 @@
361284
- support format in options of COPY command [#9744](https://github.com/apache/arrow-datafusion/pull/9744) (tinfoil-knight)
362285
- Move lower, octet_length to datafusion-functions [#9747](https://github.com/apache/arrow-datafusion/pull/9747) (Omega359)
363286
- Fixed missing trim() in rust api [#9749](https://github.com/apache/arrow-datafusion/pull/9749) (Omega359)
287+
- refactor: Extract `array_length`, `array_reverse` and `array_sort` functions from `functions-array` subcrate' s `kernels` and `udf` containers [#9751](https://github.com/apache/arrow-datafusion/pull/9751) (erenavsarogullari)
288+
- refactor: Extract `array_empty` and `array_repeat` functions from `functions-array` subcrate' s `kernels` and `udf` containers [#9762](https://github.com/apache/arrow-datafusion/pull/9762) (erenavsarogullari)
289+
- Minor: remove an outdated TODO in `TypeCoercion` [#9752](https://github.com/apache/arrow-datafusion/pull/9752) (jonahgao)
290+
- refactor: Extract `array_resize` and `cardinality` functions from `functions-array` subcrate' s `kernels` and `udf` containers [#9766](https://github.com/apache/arrow-datafusion/pull/9766) (erenavsarogullari)
291+
- fix: change placeholder errors from Internal to Plan [#9745](https://github.com/apache/arrow-datafusion/pull/9745) (erratic-pattern)
292+
- Move levenshtein, uuid, overlay to datafusion-functions [#9760](https://github.com/apache/arrow-datafusion/pull/9760) (Omega359)
293+
- improve null handling for to_char [#9689](https://github.com/apache/arrow-datafusion/pull/9689) (tinfoil-knight)
294+
- Add Expr->String for ScalarFunction and InList [#9759](https://github.com/apache/arrow-datafusion/pull/9759) (yyy1000)
295+
- Move repeat, replace, split_part to datafusion_functions [#9784](https://github.com/apache/arrow-datafusion/pull/9784) (Omega359)
296+
- refactor: Extract `array_dims`, `array_ndims` and `flatten` functions from `functions-array` subcrate' s `kernels` and `udf` containers [#9786](https://github.com/apache/arrow-datafusion/pull/9786) (erenavsarogullari)
297+
- Minor: Improve documentation about `ColumnarValues::values_to_array` [#9774](https://github.com/apache/arrow-datafusion/pull/9774) (alamb)
298+
- Fix panic in `struct` function with mixed scalar/array arguments [#9775](https://github.com/apache/arrow-datafusion/pull/9775) (alamb)
299+
- refactor: Apply minor refactorings to `functions-array` crate [#9788](https://github.com/apache/arrow-datafusion/pull/9788) (erenavsarogullari)
300+
- Move bit_length and chr functions to datafusion_functions [#9782](https://github.com/apache/arrow-datafusion/pull/9782) (PsiACE)
301+
- Support tencent cloud COS storage in `datafusion-cli` [#9734](https://github.com/apache/arrow-datafusion/pull/9734) (harveyyue)
302+
- Make it easier to register configuration extension ... [#9781](https://github.com/apache/arrow-datafusion/pull/9781) (milenkovicm)
303+
- Expr to Sql : Case [#9798](https://github.com/apache/arrow-datafusion/pull/9798) (yyy1000)
304+
- feat: Between expr to sql string [#9803](https://github.com/apache/arrow-datafusion/pull/9803) (sebastian2296)
305+
- feat: Expose `array_empty` and `list_empty` functions as alias of `empty` function [#9807](https://github.com/apache/arrow-datafusion/pull/9807) (erenavsarogullari)
306+
- Support Expr `Like` to sql [#9805](https://github.com/apache/arrow-datafusion/pull/9805) (Weijun-H)
307+
- feat: Not expr to string [#9802](https://github.com/apache/arrow-datafusion/pull/9802) (sebastian2296)
308+
- [Minor]: Move some repetitive codes to functions(proto) [#9811](https://github.com/apache/arrow-datafusion/pull/9811) (mustafasrepo)
309+
- Implement IGNORE NULLS for LAST_VALUE [#9801](https://github.com/apache/arrow-datafusion/pull/9801) (huaxingao)
310+
- [MINOR]: Move some repetitive codes to functions [#9810](https://github.com/apache/arrow-datafusion/pull/9810) (mustafasrepo)
311+
- fix: ensure mutual compatibility of the two input schemas from recursive CTEs [#9795](https://github.com/apache/arrow-datafusion/pull/9795) (jonahgao)
312+
- Add support for constant expression evaluation in limit [#9790](https://github.com/apache/arrow-datafusion/pull/9790) (mustafasrepo)
313+
- Projection Pushdown through user defined LogicalPlan nodes. [#9690](https://github.com/apache/arrow-datafusion/pull/9690) (mustafasrepo)
314+
- chore(deps): update substrait requirement from 0.27.0 to 0.28.0 [#9809](https://github.com/apache/arrow-datafusion/pull/9809) (dependabot[bot])
315+
- Run TPC-H SF10 during PR benchmarks [#9822](https://github.com/apache/arrow-datafusion/pull/9822) (gruuya)
316+
- Expose `parser` on DFParser to enable user controlled parsing [#9729](https://github.com/apache/arrow-datafusion/pull/9729) (tshauck)
317+
- Disable parallel reading for gziped ndjson file [#9799](https://github.com/apache/arrow-datafusion/pull/9799) (Lordworms)
318+
- Optimize to_timestamp (with format) (#9090) [#9833](https://github.com/apache/arrow-datafusion/pull/9833) (vojtechtoman)
319+
- Create unicode module in datafusion/functions/src/unicode and unicode_expressions feature flag, move char_length function [#9825](https://github.com/apache/arrow-datafusion/pull/9825) (Omega359)
320+
- [Minor] Update TCPDS tests, remove some #[ignore]d tests [#9829](https://github.com/apache/arrow-datafusion/pull/9829) (Dandandan)
321+
- doc: Adding baseline benchmark example [#9827](https://github.com/apache/arrow-datafusion/pull/9827) (comphead)
322+
- Add name method to execution plan [#9793](https://github.com/apache/arrow-datafusion/pull/9793) (matthewmturner)
323+
- chore(deps-dev): bump express from 4.18.2 to 4.19.2 in /datafusion/wasmtest/datafusion-wasm-app [#9826](https://github.com/apache/arrow-datafusion/pull/9826) (dependabot[bot])
324+
- feat: pass SessionState not SessionConfig to FunctionFactory::create [#9837](https://github.com/apache/arrow-datafusion/pull/9837) (tshauck)

0 commit comments

Comments
 (0)