Repository navigation

[WIP] Map State TTL - #3

Closed

ericm-db wants to merge 496 commits into

masterfrom

list-state-ttl

Closed

[WIP] Map State TTL#3
ericm-db wants to merge 496 commits into
masterfrom
list-state-ttl

This pull request is big! We're only showing the most recent 250 commits

Commits on Mar 13, 2024
- [SPARK-47375][SQL] Add guidelines for timestamp mapping in `JdbcDialect#getCatalystType`
  yaooqinn
  committed
  6ea9835
- [SPARK-47378][PROTOBUF][TESTS] Make the related Protobuf UT run well in IDE
  
  panbingkun
  authored and
  dongjoon-hyun
  committed
  b75325c
- [SPARK-47344] Extend INVALID_IDENTIFIER error beyond catching '-' in an unquoted identifier and fix "IS ! NULL" et al.
  
  authored and
  gengliangwang
  committed
  ebe9f66
- [SPARK-46913][SS] Add support for processing/event time based timers with transformWithState operator
  
  anishshri-db
  authored and
  HeartSaVioR
  committed
  839bc9f
Commits on Mar 14, 2024
- [SPARK-41762][PYTHON][CONNECT][TESTS] Enable column name comparsion in `test_column_arithmetic_ops`
  zhengruifeng
  committed
  bc0ba7d
- [SPARK-47375][DOC][FOLLOWUP] Correct the preferTimestampNTZ option description in JDBC doc
  gengliangwang
  committed
  abfbd27
- [SPARK-47377][PYTHON][CONNECT][TESTS] Factor out tests from `SparkConnectSQLTestCase`
  
  zhengruifeng
  authored and
  HyukjinKwon
  committed
  5641637
- [MINOR][INFRA] Update the link text and ref in `notify_test_workflow.yml`
  
  panbingkun
  authored and
  HyukjinKwon
  committed
  9efae37
- [SPARK-47374][CONNECT][DOCS] Fix connect-repl `usage prompt` & `docs link`
  
  panbingkun
  authored and
  HyukjinKwon
  committed
  e980211
- [SPARK-47385] Fix tuple encoders with Option inputs
  
  chenhao-db
  authored and
  cloud-fan
  committed
  9986462
- [SPARK-46988][CONNECT][FOLLOWUP] Protobuf `abbreviate` method code clean up
  zhengruifeng
  committed
  49af659
- [SPARK-47390][SQL] PostgresDialect distinguishes TIMESTAMP from TIMESTAMP_TZ
  yaooqinn
  committed
  336801d
- [SPARK-47375][DOC][FOLLOWUP] Fix a mistake in JDBC's preferTimestampNTZ option doc
  
  gengliangwang
  authored and
  yaooqinn
  committed
  63b79c1
- [SPARK-47208][CORE] Allow overriding base overhead memory
  
  jpcorreia99
  authored and
  tgravescs
  committed
  4b073fd
- [SPARK-47388][SQL] Pass `messageParameters` by name to `require()`
  MaxGekk
  committed
  e848b96
- [SPARK-47394][SQL] Support TIMESTAMP WITH TIME ZONE for H2Dialect
  
  yaooqinn
  authored and
  dongjoon-hyun
  committed
  5ce1507
- [SPARK-47391][SQL] Remove the test case workaround for JDK 8
  
  beliefer
  authored and
  dongjoon-hyun
  committed
  168346f
- [SPARK-47384][BUILD] Upgrade RoaringBitmap to 1.0.5
  
  panbingkun
  authored and
  dongjoon-hyun
  committed
  205e826
- [SPARK-47400][BUILD] Upgrade `gcs-connector` to 2.2.20
  dongjoon-hyun
  committed
  481597c
- [SPARK-47401][K8S][DOCS] Update `YuniKorn` docs with v1.5
  dongjoon-hyun
  committed
  b98accd
- [SPARK-47390][SQL][FOLLOWUP] Fix TIME_WITH_TIMEZONE mapping in PostgresDialect
  
  yaooqinn
  authored and
  dongjoon-hyun
  committed
  d41d5ec
- [SPARK-47387][SQL] Remove some unused error classes
  
  panbingkun
  authored and
  MaxGekk
  committed
  7b4ab4f
- [SPARK-47396][SQL] Add a general mapping for TIME WITHOUT TIME ZONE to TimestampNTZType
  
  yaooqinn
  authored and
  dongjoon-hyun
  committed
  213399b
- [SPARK-47402][BUILD] Upgrade `ZooKeeper` to 3.9.2
  dongjoon-hyun
  committed
  fe0aa1e
Commits on Mar 15, 2024
- [SPARK-47366][SQL] Implement parse_json
  
  chenhao-db
  authored and
  cloud-fan
  committed
  7eea705
- [SPARK-47234][BUILD] Upgrade Scala to 2.13.13
  
  panbingkun
  authored and
  dongjoon-hyun
  committed
  56cfc89
- [SPARK-47406][SQL] Handle TIMESTAMP and DATETIME in MYSQLDialect
  yaooqinn
  committed
  19ac1fc
- [SPARK-47399][SQL] Disable generated columns on expressions with collations
  
  stefankandic
  authored and
  MaxGekk
  committed
  4b1f8c3
- [SPARK-47407][SQL] Support java.sql.Types.NULL map to NullType
  
  yaooqinn
  authored and
  MaxGekk
  committed
  b7aa974
- [SPARK-47419][CONNECT] Move `log4j2-defaults.properties` to `common/utils`
  
  panbingkun
  authored and
  dongjoon-hyun
  committed
  4437e6e
- [SPARK-44740][CONNECT][TESTS][FOLLOWUP] Deduplicate `test_metadata`
  
  zhengruifeng
  authored and
  dongjoon-hyun
  committed
  6bf0317
- [SPARK-47395] Add collate and collation to other APIs
  
  stefankandic
  authored and
  MaxGekk
  committed
  e2c0471
- [SPARK-47345][SQL][TESTS] Xml functions suite
  
  yhosny
  authored and
  MaxGekk
  committed
  7c81bdf
- [SPARK-47346][PYTHON] Make daemon mode configurable when creating Python planner workers
  
  allisonwang-db
  authored and
  ueshin
  committed
  add49b3
Commits on Mar 16, 2024
- [SPARK-47423][SQL] Collations - Set operation support for strings with collations
  
  dbatomic
  authored and
  MaxGekk
  committed
  653ac5b
- [SPARK-47327][SQL] Move sort keys concurrency test to CollationFactorySuite
  
  stefankandic
  authored and
  MaxGekk
  committed
  6719168
- [SPARK-47377][PYTHON][CONNECT][TESTS][FOLLOWUP] Factor out more tests from `SparkConnectSQLTestCase`
  
  zhengruifeng
  authored and
  HyukjinKwon
  committed
  714feb7
Commits on Mar 17, 2024
- [SPARK-47426][BUILD] Upgrade Guava used by the connect module to `33.1.0-jre`
  
  panbingkun
  authored and
  dongjoon-hyun
  committed
  2dba721
Commits on Mar 18, 2024
- [SPARK-45891][DOCS] Update README with more details
  
  gene-db
  authored and
  cloud-fan
  committed
  310dd52
- [SPARK-47436][PYTHON][DOCS] Fix docstring links and type hints in Python Data Source
  
  HyukjinKwon
  authored and
  zhengruifeng
  committed
  3f171ce
- [MINOR][TESTS] Collation - extending golden file coverage
  
  dbatomic
  authored and
  MaxGekk
  committed
  44a88ed
- [SPARK-47439][PYTHON] Document Python Data Source API in API reference page
  HyukjinKwon
  committed
  08866c2
- [SPARK-43435][PYTHON][CONNECT][TESTS] Reenable doctest `pyspark.sql.connect.dataframe.DataFrame.writeStream`
  
  zhengruifeng
  authored and
  HyukjinKwon
  committed
  42c4dad
- [SPARK-47437][PYTHON][CONNECT] Correct the error class for `DataFrame.sort*`
  
  zhengruifeng
  authored and
  HyukjinKwon
  committed
  d3f12df
- [SPARK-47434][WEBUI] Fix `statistics` link in `StreamingQueryPage`
  
  HuwCampbell
  authored and
  dongjoon-hyun
  committed
  9b466d3
- [MINOR][DOCS] Add `Web UI` link to `Other Documents` section of index.md
  
  mattayes
  authored and
  dongjoon-hyun
  committed
  57424b9
- [SPARK-47438][BUILD] Upgrade jackson to 2.17.0
  
  panbingkun
  authored and
  dongjoon-hyun
  committed
  4dc362d
- [SPARK-47442][CORE][TEST] Use port 0 to start worker servers in MasterSuite
  
  Ngone51
  authored and
  dongjoon-hyun
  committed
  1aafe60
- [SPARK-47435][SQL] Fix overflow issue of MySQL UNSIGNED TINYINT caused by SPARK-45561
  
  yaooqinn
  authored and
  dongjoon-hyun
  committed
  8bd42cb
- [SPARK-47383][CORE] Support `spark.shutdown.timeout` config
  
  robreeves
  authored and
  dongjoon-hyun
  committed
  ce93c9f
- [SPARK-47446][CORE] Make `BlockManager` warn before `removeBlockInternal`
  dongjoon-hyun
  committed
  a40940a
- [SPARK-47380][CONNECT] Ensure on the server side that the SparkSession is the same
  
  nemanja-boric-databricks
  authored and
  hvanhovell
  committed
  51e8634
- [SPARK-47448][CORE] Enable `spark.shuffle.service.removeShuffle` by default
  dongjoon-hyun
  committed
  cb20fca
Commits on Mar 19, 2024
- [SPARK-47450][INFRA][R] Use R 4.3.3 in `windows` R GitHub Action job
  dongjoon-hyun
  committed
  acf17fd
- [SPARK-47329][SS][DOCS] Add note to persist dataframe while using foreachbatch and stateful streaming query to prevent state from being re-loaded in each batch
  
  anishshri-db
  authored and
  HeartSaVioR
  committed
  9f8147c
- [SPARK-47345][SQL][TESTS][FOLLOW-UP] Rename JSON to XML within XmlFunctionsSuite
  
  HyukjinKwon
  authored and
  dongjoon-hyun
  committed
  e01ed0d
- [SPARK-47422][SQL] Support collated strings in array operations
  
  nikolamand-db
  authored and
  cloud-fan
  committed
  681b41f
- [SPARK-47453][SQL][DOCKER][BUILD][TESTS] Upgrade MySQL docker image version to 8.3.0
  
  yaooqinn
  authored and
  dongjoon-hyun
  committed
  5f48931
- [SPARK-47456][SQL] Support ORC Brotli codec
  
  cxzl25
  authored and
  dongjoon-hyun
  committed
  5e42ecc
- [SPARK-47452][INFRA] Use `Ubuntu 22.04` in `dev/infra/Dockerfile`
  dongjoon-hyun
  committed
  ef94f70
- [SPARK-47454][PYTHON][CONNECT][TESTS] Split `pyspark.sql.tests.test_dataframe`
  
  zhengruifeng
  authored and
  HyukjinKwon
  committed
  b6a8369
- [SPARK-47457][SQL] Fix `IsolatedClientLoader.supportsHadoopShadedClient` to handle Hadoop 3.4+
  dongjoon-hyun
  committed
  a6bffcc
- [SPARK-47458][CORE] Fix the problem with calculating the maximum concurrent tasks for the barrier stage
  
  wbo4958
  authored and
  tgravescs
  committed
  90560dc
- [SPARK-47461][CORE] Remove private function `totalRunningTasksPerResourceProfile` from `ExecutorAllocationManager`
  
  LuciferYang
  authored and
  dongjoon-hyun
  committed
  db531c6
- [SPARK-47464][INFRA] Update `labeler.yml` for module `common/sketch` and `common/variant`
  
  panbingkun
  authored and
  dongjoon-hyun
  committed
  32ee2d7
Commits on Mar 20, 2024
- [SPARK-47449][SS] Refactor and split list/timer unit tests
  
  jingz-db
  authored and
  HeartSaVioR
  committed
  f0061db
- [SPARK-47454][PYTHON][CONNECT][TESTS][FOLLOWUP] Further split `pyspark.sql.tests.test_dataframe`
  
  zhengruifeng
  authored and
  HyukjinKwon
  committed
  46b46b7
- [SPARK-47468][BUILD] Exclude `logback` dependency from SBT like Maven
  dongjoon-hyun
  committed
  c32d278
- [SPARK-47309][SQL] XML: Add schema inference tests for value tags
  
  shujingyang-db
  authored and
  HyukjinKwon
  committed
  b9c0e93
- [SPARK-47330][SQL][TESTS] XML: Added XmlExpressionsSuite
  
  yhosny
  authored and
  HyukjinKwon
  committed
  bc378f4
- [SPARK-47470][SQL][TESTS] Ignore `IntentionallyFaultyConnectionProvider` error in `CliSuite`
  dongjoon-hyun
  committed
  61d7b0f
- [SPARK-47007][SQL][PYTHON][R][CONNECT] Add the `map_sort` function
  
  authored and
  MaxGekk
  committed
  747846b
- [SPARK-47447][SQL] Allow reading Parquet TimestampLTZ as TimestampNTZ
  gengliangwang
  committed
  c3a04fa
- [SPARK-47455][BUILD] Fix resource leak during the initialization of `scalaStyleOnCompileConfig` in `SparkBuild.scala`
  LuciferYang
  committed
  85bf761
- [SPARK-47473][SQL] Fix correctness issue of converting postgres INFINITY timestamps
  yaooqinn
  committed
  ad8ac17
- [SPARK-47480][PYTHON][CONNECT][TESTS] Enable doctest for `createDataFrame`
  
  zhengruifeng
  authored and
  HyukjinKwon
  committed
  a5910a2
- [SPARK-41625][PYTHON][CONNECT][TESTS] Enable doctest `pyspark.sql.connect.dataframe.DataFrame.isStreaming`
  
  zhengruifeng
  authored and
  HyukjinKwon
  committed
  df56390
- [SPARK-47478][SQL][TESTS] Improve test coverage for mysql bool synonyms
  
  yaooqinn
  authored and
  HyukjinKwon
  committed
  a3c04ec
- [SPARK-47296][SQL][COLLATION] Fail unsupported functions for non-binary collations
  
  authored and
  cloud-fan
  committed
  8762e25
- [SPARK-46990][SQL] Fix loading empty Avro files emitted by event-hubs
  
  Ivan Sadikov
  authored and
  cloud-fan
  committed
  7626080
- [SPARK-47443][SQL] Window Aggregate support for collations
  
  dbatomic
  authored and
  MaxGekk
  committed
  3a3477b
- [SPARK-47462][SQL] Align mappings of other unsigned numeric types with TINYINT in MySQLDialect
  
  yaooqinn
  authored and
  dongjoon-hyun
  committed
  a34c8ce
- [SPARK-45393][BUILD] Upgrade Hadoop to 3.4.0
  
  dongjoon-hyun
  and
  LuciferYang
  committed
  49b4c3b
- [SPARK-47486][CONNECT] Remove unused private `ArrowDeserializers.getString` method
  
  LuciferYang
  authored and
  dongjoon-hyun
  committed
  f66274e
- [SPARK-47490][SS] Fix RocksDB Logger constructor use to avoid deprecation warning
  
  anishshri-db
  authored and
  dongjoon-hyun
  committed
  b8e7d99
- [SPARK-47494][DOC] Add migration doc for the behavior change of Parquet timestamp inference since Spark 3.3
  
  gengliangwang
  authored and
  dongjoon-hyun
  committed
  11247d8
Commits on Mar 21, 2024
- Revert "[SPARK-47007][SQL][PYTHON][R][CONNECT] Add the `map_sort` function"
  cloud-fan
  committed
  bb3e275
- [MINOR][CORE] Fix a comment typo `slf4j-to-jul` to `jul-to-slf4j`
  dongjoon-hyun
  committed
  bb0867f
- [SPARK-46920][YARN] Improve executor exit error message on YARN
  
  pan3793
  authored and
  LuciferYang
  committed
  5d3845f
- [SPARK-47474][CORE] Revert SPARK-47461 and add some comments
  LuciferYang
  committed
  bffffee
- [SPARK-41888][PYTHON][CONNECT][TESTS] Enable doctest for `DataFrame.observe`
  
  zhengruifeng
  authored and
  dongjoon-hyun
  committed
  91df046
- [SPARK-45393][BUILD][FOLLOWUP] Update IsolatedClientLoader fallback Hadoop version to 3.4.0
  
  pan3793
  authored and
  dongjoon-hyun
  committed
  3a1609a
- [SPARK-47496][SQL] Java SPI Support for dynamic JDBC dialect registering
  yaooqinn
  committed
  2b0e841
- [SPARK-47501][SQL] Add convertDateToDate like the existing convertTimestampToTimestamp for JdbcDialect
  
  yaooqinn
  authored and
  dongjoon-hyun
  committed
  32f3d4d
- [SPARK-47487][SQL] Simplify code in AnsiTypeCoercion
  
  cloud-fan
  authored and
  dongjoon-hyun
  committed
  25ecde9
- [SPARK-47398][SQL] Extract a trait for InMemoryTableScanExec to allow for extending functionality
  
  razajafri
  authored and
  tgravescs
  committed
  6a27789
Commits on Mar 22, 2024
- [SPARK-47502][INFRA] Make output the installation packages in `descending size`, add `titles`, and remove `unused` packages
  
  panbingkun
  authored and
  dongjoon-hyun
  committed
  b4c0922
- [SPARK-47512][SS] Tag operation type used with RocksDB state store instance lock acquisition/release
  
  anishshri-db
  authored and
  HeartSaVioR
  committed
  057acf9
- Revert "[SPARK-46654][SQL][PYTHON] Make `to_csv` explicitly indicate that it does not support some types of data"
  cloud-fan
  committed
  7b575d1
- [SPARK-47513][TESTS] Regenerate benchmark results
  dongjoon-hyun
  committed
  ef3244c
- [SPARK-47514][SQL][TESTS] Add a test coverage for createTable method (partitioned-table) in CatalogSuite
  
  panbingkun
  authored and
  dongjoon-hyun
  committed
  9ba70d6
- [SPARK-47479][SQL] Optimize cannot write data to relations with multiple paths error log
  
  huangxiaopingRD
  authored and
  MaxGekk
  committed
  5042263
- [SPARK-47501][SQL][FOLLOWUP] Rename convertDateToDate to convertJavaDateToDate
  
  yaooqinn
  authored and
  dongjoon-hyun
  committed
  0ef7b77
- [MINOR][SQL] Fix a typo in `DelegateSymlinkTextInputSplit` comment
  
  RocMarshal
  authored and
  dongjoon-hyun
  committed
  47bce8e
- [SPARK-47500][PYTHON][CONNECT] Factor column name handling out of `plan.py`
  
  zhengruifeng
  authored and
  HyukjinKwon
  committed
  aea13fc
- [SPARK-47499][PYTHON][CONNECT][TESTS] Enable `test_help_command` test
  
  zhengruifeng
  authored and
  dongjoon-hyun
  committed
  ca44489
- [SPARK-47483][SQL] Add support for aggregation and join operations on arrays of collated strings
  
  nikolamand-db
  authored and
  MaxGekk
  committed
  8cba15e
- [SPARK-47007][SQL] Add the `MapSort` expression
  
  authored and
  MaxGekk
  committed
  c94090e
- [MINOR][PYTHON][DOCS] Fix typos in `PandasCogroupedOps.applyInArrow`
  
  zhengruifeng
  authored and
  HyukjinKwon
  committed
  105f008
- [SPARK-47462][SQL][FOLLOWUP] Add migration guide for integral type mapping changes
  yaooqinn
  committed
  c486ed8
- [SPARK-47515][SQL] Save TimestampNTZType as DATETIME in MySQL
  yaooqinn
  committed
  f41cd17
- [SPARK-47440][SQL] Fix pushing unsupported syntax to MsSqlServer
  
  stefanbuk-db
  authored and
  yaooqinn
  committed
  227a50a
- [MINOR][DOCS] Fix typo in spark connect overview
  
  rorueda
  authored and
  dongjoon-hyun
  committed
  d7be50f
- [SPARK-47517][CORE][SQL] Prefer Utils.bytesToString for size display
  
  pan3793
  authored and
  yaooqinn
  committed
  32dfdd3
- [SPARK-47517][INFRA][FOLLOWUP] Prevent `byteCountToDisplaySize` via Scalastyle
  dongjoon-hyun
  committed
  d1be4fb
- [SPARK-47523][SQL] Replace deprecated `JsonParser#getCurrentName` with `JsonParser#currentName`
  
  LuciferYang
  authored and
  dongjoon-hyun
  committed
  93f98c0
- [SPARK-47521][CORE] Use `Utils.tryWithResource` during reading shuffle data from external storage
  
  maheshbehera
  authored and
  dongjoon-hyun
  committed
  2456690
- [SPARK-47522][SQL] Read MySQL FLOAT as FloatType to keep consistent with the write side
  
  yaooqinn
  authored and
  dongjoon-hyun
  committed
  36126a5
- [SPARK-47522][SQL][FOLLOWUP] Add float(p) values for MySQLIntegrationSuite
  
  yaooqinn
  authored and
  dongjoon-hyun
  committed
  39500a3
Commits on Mar 24, 2024
- [SPARK-47510][INFRA] Fix `DSTREAM` label pattern in `labeler.yml`
  dongjoon-hyun
  committed
  b9335b9
- [SPARK-47495][CORE] Fix primary resource jar added to spark.jars twice under k8s cluster mode
  
  jiale_tan
  and
  dongjoon-hyun
  committed
  c29d132
- [SPARK-47529][DOCS] Use hadoop 3.4.0 in some docs
  
  panbingkun
  authored and
  dongjoon-hyun
  committed
  9d6b9f7
- [SPARK-47530][BUILD][TESTS] Add `bcpkix-jdk18on` test dependencies to `hive` module for Hadoop 3.4.0
  dongjoon-hyun
  committed
  11f5d3f
- [SPARK-47531][BUILD] Upgrade `Arrow` to 15.0.2
  
  panbingkun
  authored and
  dongjoon-hyun
  committed
  063638b
- [SPARK-47497][SQL] Make `to_csv` support the output of `array/struct/map/binary` as pretty strings
  
  panbingkun
  authored and
  dongjoon-hyun
  committed
  18bb6a3
- [SPARK-47503][SQL] Make `makeDotNode` escape graph node name always
  
  Alexey
  and
  dongjoon-hyun
  committed
  15ea9de
- [SPARK-47526][BUILD] Upgrade `netty` to 4.1.108.Final and `netty-tcnative` to 2.0.65.Final
  
  panbingkun
  authored and
  dongjoon-hyun
  committed
  ba175e3
- [SPARK-47528][SQL] Add UserDefinedType support to DataTypeUtils.canWrite
  
  viirya
  authored and
  dongjoon-hyun
  committed
  767a52d
- [SPARK-47533][BUILD] Migrate scalafmt dialect to `scala213`
  
  panbingkun
  authored and
  dongjoon-hyun
  committed
  310fd65
- [SPARK-44708][PYTHON] Migrate test_reset_index assert_eq to use assertDataFrameEqual
  
  gdhuper
  authored and
  HyukjinKwon
  committed
  2ef15ca
- [MINOR][PYTHON][DOCS] Fix a `pandas_udf` example
  
  zhengruifeng
  authored and
  HyukjinKwon
  committed
  19e3de4
- [SPARK-47516][INFRA] Move `remove unused installation package logic` from `each test job` to `create the docker image`
  
  panbingkun
  authored and
  HyukjinKwon
  committed
  015719e
Commits on Mar 25, 2024
- [SPARK-47534][SQL] Move `o.a.s.variant` to `o.a.s.types.variant`
  dongjoon-hyun
  committed
  99fb84b
- [SPARK-47535][INFRA] Update `publish_snapshot.yml` to publish twice per day
  dongjoon-hyun
  committed
  b34a44f
- [SPARK-47536][BUILD] Upgrade `jmock-junit5` to 2.13.1
  
  LuciferYang
  authored and
  dongjoon-hyun
  committed
  d8d119a
- [SPARK-47506][SQL] Add support to all file source formats for collated data types
  
  stefankandic
  authored and
  MaxGekk
  committed
  804b2a4
- [SPARK-47537][SQL] Fix error data type mapping on MySQL Connector/J
  
  yaooqinn
  authored and
  dongjoon-hyun
  committed
  824d670
- [SPARK-47538][BUILD] Remove `commons-logging` dependency
  dongjoon-hyun
  committed
  e22ddcb
- [SPARK-47451][SQL] Support to_json(variant).
  
  chenhao-db
  authored and
  cloud-fan
  committed
  4a1f241
- [SPARK-47539][SQL] Make the return value of method `castToString` be `Any => UTF8String`
  
  panbingkun
  authored and
  HyukjinKwon
  committed
  b341787
- [SPARK-47539][SQL][FOLLOWUP] Fix UT about `variant`
  
  panbingkun
  authored and
  MaxGekk
  committed
  1b55fd3
- [SPARK-46743][SQL] Count bug after constant folding
  
  agubichev
  and
  cloud-fan
  committed
  b2f6474
- [SPARK-47256][SQL] Assign names to error classes _LEGACY_ERROR_TEMP_102[4-7]
  
  davidm-db
  authored and
  MaxGekk
  committed
  d7db869
- [SPARK-47548][BUILD] Remove unused `commons-beanutils` dependency
  dongjoon-hyun
  committed
  90d5065
- [SPARK-47550][K8S][BUILD] Update `kubernetes-client` to 6.11.0
  
  bjornjorgensen
  authored and
  dongjoon-hyun
  committed
  7b9b3cb
- [SPARK-47552][CORE] Set `spark.hadoop.fs.s3a.connection.establish.timeout` to 30s if missing
  dongjoon-hyun
  committed
  87cae7b
Commits on Mar 26, 2024
- [SPARK-47549][BUILD] Remove Spark 3.0~3.2 `pyspark/version.py` workaround from release scripts
  
  dongjoon-hyun
  authored and
  HyukjinKwon
  committed
  a5ea884
- [SPARK-47497][SQL][FOLLOWUP] Add a UT for `nested structure` for the function `to_csv`
  
  panbingkun
  authored and
  cloud-fan
  committed
  0340f80
- [SPARK-47554][BUILD] Upgrade `sbt-assembly` to `2.2.0` and `sbt-protoc` to `1.0.7`
  
  panbingkun
  authored and
  HyukjinKwon
  committed
  2a8bb5c
- Revert "[SPARK-47450][INFRA][R] Use R 4.3.3 in `windows` R GitHub Action job"
  HyukjinKwon
  committed
  0054a12
- [SPARK-47509][SQL] Block subquery expressions in lambda and higher-order functions
  
  dtenedor
  and
  cloud-fan
  committed
  acf13d6
- Revert "Revert "[SPARK-47450][INFRA][R] Use R 4.3.3 in `windows` R GitHub Action job""
  HyukjinKwon
  committed
  31db27d
- [SPARK-47372][SS] Add support for range scan based key state encoder for use with state store provider
  
  anishshri-db
  authored and
  HeartSaVioR
  committed
  ff38378
- [SPARK-47358][SQL][COLLATION] Improve repeat expression support to return correct datatype
  
  miland-db
  authored and
  MaxGekk
  committed
  a3aa086
- [SPARK-47367][PYTHON][CONNECT] Support Python data sources with Spark Connect
  
  allisonwang-db
  authored and
  HyukjinKwon
  committed
  a127df4
- [SPARK-47560][PYTHON][CONNECT] Avoid RPC to validate column name with cached schema
  zhengruifeng
  committed
  becbf8b
- [SPARK-47469][SS][TESTS] Add `Trigger.AvailableNow` tests for `transformWithState` operator
  
  jingz-db
  authored and
  HeartSaVioR
  committed
  3e83432
- [SPARK-47431][SQL] Add session level default Collation
  
  mihailomilosevic2001
  authored and
  cloud-fan
  committed
  6ee662c
- [SPARK-47367][PYTHON][CONNECT][TESTS][FOLLOW-UP] Recover the test case for the number of partitions
  
  HyukjinKwon
  authored and
  dongjoon-hyun
  committed
  ded8cdf
- [SPARK-47557][SQL][TEST] Audit MySQL ENUM/SET Types
  
  yaooqinn
  authored and
  dongjoon-hyun
  committed
  89104b9
- [SPARK-47544][PYTHON] SparkSession builder method is incompatible with visual studio code intellisense
  
  Niranjan Jayakar
  authored and
  dongjoon-hyun
  committed
  8e20f8e
- [SPARK-47561][SQL] Fix analyzer rule order issues about Alias
  
  cloud-fan
  authored and
  dongjoon-hyun
  committed
  e00eace
- [SPARK-47555][SQL] Show a warning message about SQLException if `JDBCTableCatalog.loadTable` fails
  
  xleoken
  authored and
  dongjoon-hyun
  committed
  fd4b8e8
- [MINOR][CORE] When failed to canceling the job group, add a warning log
  
  panbingkun
  authored and
  dongjoon-hyun
  committed
  7d87a94
Commits on Mar 27, 2024
- [SPARK-47498][TESTS][CORE] Refine some GPU fraction calculation tests
  
  wbo4958
  authored and
  Ngone51
  committed
  96b301c
- [SPARK-47273][SS][PYTHON] implement Python data stream writer interface
  
  chaoqin-li1123
  authored and
  HeartSaVioR
  committed
  5ac3918
- [SPARK-47570][SS] Integrate range scan encoder changes with timer implementation
  
  jingz-db
  authored and
  HeartSaVioR
  committed
  88b29c5
- [SPARK-47562][CONNECT] Factor literal handling out of `plan.py`
  zhengruifeng
  committed
  9db527e
- [SPARK-42040][SQL] SPJ: Introduce a new API for V2 input partition to report partition statistics
  
  qzhu
  authored and
  sunchao
  committed
  eef44f0
- [SPARK-47610][CORE] Always set `io.netty.tryReflectionSetAccessible=true`
  
  pan3793
  authored and
  LuciferYang
  committed
  5f392a2
- [SPARK-46575][SQL][FOLLOWUP] Add back `HiveThriftServer2.startWithContext(SQLContext)` method for compatibility
  
  dragqueen95
  and
  cloud-fan
  committed
  f9eb3f3
- [SPARK-47491][CORE] Add `slf4j-api` jar to the class path first before the others of `jars` directory
  dongjoon-hyun
  committed
  a600c0e
- [SPARK-47611][SQL] Cleanup dead code in MySQLDialect.getCatalystType
  
  yaooqinn
  authored and
  dongjoon-hyun
  committed
  b540cc5
- [SPARK-47430][SQL] Support GROUP BY for MapType
  
  authored and
  cloud-fan
  committed
  d57164a
- [SPARK-46575][SQL][FOLLOWUP] Correct @since annotation for HiveThriftServer2.startWithContext(SQLContext, exitonError)
  
  dragqueen95
  authored and
  yaooqinn
  committed
  d326cb9
- [SPARK-47563][SQL] Add map normalization on creation
  
  stevomitric
  authored and
  cloud-fan
  committed
  87449c3
- [SPARK-47564][SQL] Always throw FAILED_READ_FILE error when fail to read files
  
  cloud-fan
  authored and
  MaxGekk
  committed
  d10dbaa
- [SPARK-47616][SQL] Add User Document for Mapping Spark SQL Data Types from MySQL
  
  yaooqinn
  authored and
  dongjoon-hyun
  committed
  8d1539f
- [SPARK-47485][SQL][PYTHON][CONNECT] Create column with collations in dataframe API
  
  stefankandic
  authored and
  MaxGekk
  committed
  1b3e5e7
- [SPARK-47107][SS][PYTHON] Implement partition reader for python streaming data source
  
  chaoqin-li1123
  authored and
  HeartSaVioR
  committed
  e3e7135
Commits on Mar 28, 2024
- [SPARK-47543][CONNECT][PYTHON] Inferring `dict` as `MapType` from Pandas DataFrame to allow DataFrame creation
  
  itholic
  authored and
  HyukjinKwon
  committed
  0318648
- [SPARK-47546][SQL] Improve validation when reading Variant from Parquet
  
  cashmand
  authored and
  cloud-fan
  committed
  f2492dd
- [SPARK-47620][PYTHON][CONNECT] Add a helper function to sort columns
  zhengruifeng
  committed
  0eb96ae
- [SPARK-47619][PYTHON][DOCS] Refine docstring of `to_json/from_json`
  HyukjinKwon
  committed
  67bd52c
- [SPARK-47363][SS] Initial State without state reader implementation for State API v2
  
  jingz-db
  authored and
  HeartSaVioR
  committed
  4d72be3
- [SPARK-47614][CORE][DOC] Update some outdated comments about `JavaModuleOptions`
  
  panbingkun
  authored and
  yaooqinn
  committed
  299a659
- [MINOR][PYTHON][TESTS] Remove redundant parity tests
  zhengruifeng
  committed
  8a6e5bb
- [SPARK-47621][PYTHON][DOCS] Refine docstring of `try_sum`, `try_avg`, `avg`, `sum`, `mean`
  
  HyukjinKwon
  authored and
  zhengruifeng
  committed
  b594c4e
- [SPARK-47559][SQL] Codegen Support for variant `parse_json`
  
  panbingkun
  authored and
  cloud-fan
  committed
  8c4d676
- [SPARK-47628][SQL] Fix Postgres bit array issue 'Cannot cast to boolean'
  
  yaooqinn
  authored and
  dongjoon-hyun
  committed
  4b58a63
- [MINOR][CORE] Replace `get+getOrElse` with `getOrElse` with default value in `StreamingQueryException`
  
  LuciferYang
  authored and
  dongjoon-hyun
  committed
  32e73dd
- [SPARK-47475][CORE][K8S] Support `spark.kubernetes.jars.avoidDownloadSchemes` for K8s Cluster Mode
  
  jiale_tan
  authored and
  dongjoon-hyun
  committed
  2e4f2b0
- [SPARK-47630][BUILD] Upgrade `zstd-jni` to 1.5.6-1
  dongjoon-hyun
  committed
  1623b2d
- [SPARK-47632][BUILD] Ban `com.amazonaws:aws-java-sdk-bundle` dependency
  dongjoon-hyun
  committed
  d8dc0c3
- [SPARK-47492][SQL] Widen whitespace rules in lexer
  
  srielau
  authored and
  gengliangwang
  committed
  c832e2a
- [SPARK-47635][K8S] Use Java `21` instead of `21-jre` image in K8s Dockerfile
  dongjoon-hyun
  committed
  ca00013
- [SPARK-47366][PYTHON] Add pyspark and dataframe parse_json aliases
  
  gene-db
  and
  HyukjinKwon
  committed
  7f1eadc
- [SPARK-47631][SQL] Remove unused `SQLConf.parquetOutputCommitterClass` method
  
  dongjoon-hyun
  authored and
  HyukjinKwon
  committed
  009b50e
- [SPARK-47623][PYTHON][CONNECT][TESTS] Use `QuietTest` in parity tests
  
  zhengruifeng
  authored and
  HyukjinKwon
  committed
  6cf5f95
Commits on Mar 29, 2024
- [SPARK-47546][SQL][FOLLOWUP] Improve validation when reading Variant from Parquet using non-vectorized reader
  
  cashmand
  authored and
  HyukjinKwon
  committed
  fde7a7f
- [SPARK-47525][SQL] Support subquery correlation joining on map attributes
  
  jchen5
  authored and
  cloud-fan
  committed
  41cbd56
- [SPARK-47638][PS][CONNECT] Skip column name validation in PS
  zhengruifeng
  committed
  a57f94b
- [SPARK-47511][SQL] Canonicalize With expressions by re-assigning IDs
  
  kelvinjian-db
  authored and
  cloud-fan
  committed
  b314672
- [SPARK-47568][SS] Fix race condition between maintenance thread and load/commit for snapshot files
  
  sahnib
  authored and
  HeartSaVioR
  committed
  0b844e5
- [SPARK-47629][INFRA] Add `common/variant` and `connector/kinesis-asl` to maven daily test module list
  LuciferYang
  committed
  a8b247e
- [SPARK-47574][INFRA] Introduce Structured Logging Framework
  gengliangwang
  committed
  874d033
- [SPARK-47644][PYTHON][DOCS] Refine docstrings of try_*
  HyukjinKwon
  committed
  5259dcc
- [SPARK-47572][SQL] Enforce Window partitionSpec is orderable
  
  chenhao-db
  authored and
  cloud-fan
  committed
  609bd48
- [SPARK-47646][SQL] Make try_to_number return NULL for malformed input
  HyukjinKwon
  committed
  d709e20
- [SPARK-47543][CONNECT][PYTHON][TESTS][FOLLOW-UP] Skip the test if pandas and PyArrow are unavailable
  HyukjinKwon
  committed
  63529bf
- [SPARK-47642][BUILD] Exclude dependencies related to `org.junit.jupiter` and `org.junit.platform` from `jmock-junit5`
  
  LuciferYang
  authored and
  dongjoon-hyun
  committed
  5d5611e
- [SPARK-47641][SQL] Improve the performance for `UnaryMinus` and `Abs`
  
  panbingkun
  authored and
  dongjoon-hyun
  committed
  5318846
- [SPARK-47647][SQL] Make MySQL data source able to read bit(n>1) as BinaryType like Postgres
  
  yaooqinn
  authored and
  dongjoon-hyun
  committed
  12155e8
- [SPARK-47648][SQL][TESTS] Use `checkError()` to check Exception in `[CSV|Json|Xml]Suite` and `[Csv|Json]FunctionsSuite`
  
  panbingkun
  authored and
  dongjoon-hyun
  committed
  48def2b
- [SPARK-47575][INFRA] Implement logWarning API in structured logging framework
  gengliangwang
  committed
  d182810
- [SPARK-47637][SQL] Use errorCapturingIdentifier in more places
  
  srielau
  authored and
  gengliangwang
  committed
  db14be8
- [SPARK-47576][INFRA] Implement logInfo API in structured logging framework
  gengliangwang
  committed
  11d76c9
Commits on Mar 31, 2024
- [SPARK-47656][BUILD] Upgrade commons-io to 2.16.0
  
  panbingkun
  authored and
  dongjoon-hyun
  committed
  1bc812b
- [SPARK-47658][BUILD] Upgrade `tink` to 1.12.0
  
  panbingkun
  authored and
  dongjoon-hyun
  committed
  ef4c27b
Commits on Apr 1, 2024
- [SPARK-47662][SQL][DOCS] Add User Document for Mapping Spark SQL Data Types to MySQL
  
  yaooqinn
  authored and
  dongjoon-hyun
  committed
  18b582c
- [SPARK-47645][BUILD][CORE][SQL][YARN] Make Spark build with `-release` instead of `-target`
  
  LuciferYang
  authored and
  dongjoon-hyun
  committed
  4c1405d
- [SPARK-47654][INFRA] Structured logging framework: support log concatenation
  gengliangwang
  committed
  a5ca586
- [SPARK-47431][SQL][FOLLOWUP] Do not access SQLConf in Literal.apply
  
  mihailomilosevic2001
  authored and
  cloud-fan
  committed
  535734e
- [SPARK-47569][SQL] Disallow comparing variant
  
  chenhao-db
  authored and
  cloud-fan
  committed
  cf02b1a
- [SPARK-47664][PYTHON][CONNECT] Validate the column name with cached schema
  
  zhengruifeng
  authored and
  HyukjinKwon
  committed
  968cba2
- [SPARK-47666][SQL] Fix NPE when reading mysql bit array as LongType
  yaooqinn
  committed
  72c619e
- [SPARK-46840][SQL][TESTS] Add `CollationBenchmark`
  
  GideonPotok
  authored and
  cloud-fan
  committed
  e9f204a
- Add support for ValueState TTL.
  sahnib
  committed
  3c54264
- Fix existing testcases, and add TTL testcase.
  sahnib
  committed
  7c55e67
- Improved documentation, added NERF error classes for user errors.
  sahnib
  committed
  ca7a828
- Regenerate docs for SQL Error conditions.
  sahnib
  committed
  7e03677
- Add more testcases for value state ttl.
  sahnib
  committed
  b414a30
- Fix indentation.
  sahnib
  committed
  c17415c
- Add a placeholder TODO to use range scan once available.
  sahnib
  committed
  840e6da
- Remove unnecessary log in TransformWithStateExec.
  sahnib
  committed
  33ef577
- Rebase with latest master
  sahnib
  committed
  fc291df
- Suppress method checkstyle for TTLMode.
  sahnib
  committed
  eb8d50d
- Modify Spark Connect tws API to include ttlMode.
  sahnib
  committed
  6248ed4
- Rename ttl changes scala file to TTLState.scala
  sahnib
  committed
  5156882
- Do not write ttlExpiration in valueRow if ttlMode is set to NoTTL().
  sahnib
  committed
  af63a2f
- Incorporated comments.
  sahnib
  committed
  7465ec1
- Incorporate TTLMode for initialState changes.
  sahnib
  committed
  ede4898
- Modify eventTime ttl to use absolute time instead of a ttlDuration.
  sahnib
  committed
  902f7fa
- Use range scan encoder for ttl.
  sahnib
  committed
  9e95b79
- Add a testcase for multiple values (expired and unexpired) in a partition.
  sahnib
  committed
  da2b62b
- Add negative value check for expirationTime/ttlDuration.
  sahnib
  committed
  c96d4af
- Regenrate SQL error docs.
  sahnib
  committed
  69ad1e2
- initial commit after rebase
  ericm-db
  committed
  010e700
- all test cases pass
  ericm-db
  committed
  c7a75df
- adding exceptions
  ericm-db
  committed
  073572e

Commits

Commits on Mar 13, 2024

Commits on Mar 14, 2024

Commits on Mar 15, 2024

Commits on Mar 16, 2024

Commits on Mar 17, 2024

Commits on Mar 18, 2024

Commits on Mar 19, 2024

Commits on Mar 20, 2024

Commits on Mar 21, 2024

Commits on Mar 22, 2024

Commits on Mar 24, 2024

Commits on Mar 25, 2024