Skip to content

Spark and Fabric Spark Release #46226

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tvaron3 merged 4 commits into from
Jul 31, 2025
Merged

Spark and Fabric Spark Release #46226

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
e87980e
release changes
tvaron3 Jul 31, 2025
2827bd7
release changes
tvaron3 Jul 31, 2025
50a751f
Update sdk/cosmos/azure-cosmos-spark_3-5_2-12/CHANGELOG.md
tvaron3 Jul 31, 2025
3b4728e
Apply suggestions from code review
tvaron3 Jul 31, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions eng/versioning/version_client.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,9 +106,9 @@ com.azure:azure-cosmos;4.73.1;4.74.0-beta.1
com.azure:azure-cosmos-benchmark;4.0.1-beta.1;4.0.1-beta.1
com.azure.cosmos.spark:azure-cosmos-spark_3_2-12;1.0.0-beta.1;1.0.0-beta.1
com.azure:azure-cosmos-encryption;2.22.0;2.23.0-beta.1
com.azure.cosmos.spark:azure-cosmos-spark_3-3_2-12;4.37.2;4.38.0-beta.1
com.azure.cosmos.spark:azure-cosmos-spark_3-4_2-12;4.37.2;4.38.0-beta.1
com.azure.cosmos.spark:azure-cosmos-spark_3-5_2-12;4.37.2;4.38.0-beta.1
com.azure.cosmos.spark:azure-cosmos-spark_3-3_2-12;4.37.2;4.38.0
com.azure.cosmos.spark:azure-cosmos-spark_3-4_2-12;4.37.2;4.38.0
com.azure.cosmos.spark:azure-cosmos-spark_3-5_2-12;4.37.2;4.38.0
com.azure.cosmos.spark:azure-cosmos-spark-account-data-resolver-sample;1.0.0-beta.1;1.0.0-beta.1
com.azure.cosmos.spark:fabric-cosmos-spark-auth_3;1.0.0-beta.1;1.0.0-beta.1
com.azure:azure-cosmos-test;1.0.0-beta.13;1.0.0-beta.14
Expand Down
4 changes: 2 additions & 2 deletions sdk/cosmos/azure-cosmos-spark-account-data-resolver-sample/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@
<dependency>
<groupId>com.azure.cosmos.spark</groupId>
<artifactId>azure-cosmos-spark_3-5_2-12</artifactId>
<version>4.38.0-beta.1</version> <!-- {x-version-update;com.azure.cosmos.spark:azure-cosmos-spark_3-5_2-12;current} -->
<version>4.38.0</version> <!-- {x-version-update;com.azure.cosmos.spark:azure-cosmos-spark_3-5_2-12;current} -->
<scope>provided</scope>
</dependency>
<dependency>
Expand Down Expand Up @@ -292,7 +292,7 @@
<include>com.fasterxml.jackson.core:jackson-databind:[2.15.2]</include> <!-- {x-include-update;cosmos_com.fasterxml.jackson.core:jackson-databind;external_dependency} -->
<include>com.fasterxml.jackson.module:jackson-module-scala_2.12:[2.15.2]</include> <!-- {x-include-update;cosmos_com.fasterxml.jackson.module:jackson-module-scala_2.12;external_dependency} -->
<include>com.globalmentor:hadoop-bare-naked-local-fs:[0.1.0]</include> <!-- {x-include-update;cosmos_com.globalmentor:hadoop-bare-naked-local-fs;external_dependency} -->
<include>com.azure.cosmos.spark:azure-cosmos-spark_3-5_2-12:[4.38.0-beta.1]</include> <!-- {x-include-update;com.azure.cosmos.spark:azure-cosmos-spark_3-5_2-12;current} -->
<include>com.azure.cosmos.spark:azure-cosmos-spark_3-5_2-12:[4.38.0]</include> <!-- {x-include-update;com.azure.cosmos.spark:azure-cosmos-spark_3-5_2-12;current} -->
</includes>
</bannedDependencies>
</rules>
Expand Down
9 changes: 2 additions & 7 deletions sdk/cosmos/azure-cosmos-spark_3-3_2-12/CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,9 @@
## Release History

### 4.38.0-beta.1 (Unreleased)
### 4.38.0 (2025-07-31)

#### Features Added

#### Breaking Changes

#### Bugs Fixed

#### Other Changes
* Added telemetry support by adding OTEL span attribute naming schemes, introducing Azure Monitor integration, and sampled diagnostics. - See [PR 45924](https://github.com/Azure/azure-sdk-for-java/pull/45924)

### 4.37.2 (2025-05-14)

Expand Down
7 changes: 5 additions & 2 deletions sdk/cosmos/azure-cosmos-spark_3-3_2-12/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ https://github.com/Azure/azure-sdk-for-java/issues/new
#### azure-cosmos-spark_3-3_2-12
| Connector | Supported Spark Versions | Supported JVM Versions | Supported Scala Versions | Supported Databricks Runtimes |
|-----------|--------------------------|------------------------|--------------------------|-------------------------------|
| 4.38.0 | 3.3.0 - 3.3.2 | [8, 11] | 2.12 | 11.\*, 12.\* |
| 4.37.2 | 3.3.0 - 3.3.2 | [8, 11] | 2.12 | 11.\*, 12.\* |
| 4.37.1 | 3.3.0 - 3.3.2 | [8, 11] | 2.12 | 11.\*, 12.\* |
| 4.37.0 | 3.3.0 - 3.3.2 | [8, 11] | 2.12 | 11.\*, 12.\* |
Expand Down Expand Up @@ -72,6 +73,7 @@ https://github.com/Azure/azure-sdk-for-java/issues/new
#### azure-cosmos-spark_3-4_2-12
| Connector | Supported Spark Versions | Supported JVM Versions | Supported Scala Versions | Supported Databricks Runtimes |
|-----------|--------------------------|------------------------|--------------------------|-------------------------------|
| 4.38.0 | 3.4.0 - 3.4.1 | [8, 11] | 2.12 | 13.\* |
| 4.37.2 | 3.4.0 - 3.4.1 | [8, 11] | 2.12 | 13.\* |
| 4.37.1 | 3.4.0 - 3.4.1 | [8, 11] | 2.12 | 13.\* |
| 4.37.0 | 3.4.0 - 3.4.1 | [8, 11] | 2.12 | 13.\* |
Expand Down Expand Up @@ -107,6 +109,7 @@ https://github.com/Azure/azure-sdk-for-java/issues/new
#### azure-cosmos-spark_3-5_2-12
| Connector | Supported Spark Versions | Minimum Java Version | Supported Scala Versions | Supported Databricks Runtimes |
|-----------|--------------------------|----------------------|----------------------------|-------------------------------|
| 4.38.0 | 3.5.0 | [8, 11] | 2.12 | 14.*, 15.\* |
| 4.37.2 | 3.5.0 | [8, 11] | 2.12 | 14.*, 15.\* |
| 4.37.1 | 3.5.0 | [8, 11] | 2.12 | 14.*, 15.\* |
| 4.37.0 | 3.5.0 | [8, 11] | 2.12 | 14.*, 15.\* |
Expand All @@ -128,11 +131,11 @@ to use the same version of Scala that Spark was compiled for.
### Download

You can use the maven coordinate of the jar to auto install the Spark Connector to your Databricks Runtime 11 from Maven:
`com.azure.cosmos.spark:azure-cosmos-spark_3-3_2-12:4.37.2`
`com.azure.cosmos.spark:azure-cosmos-spark_3-3_2-12:4.38.0`

You can also integrate against Cosmos DB Spark Connector in your SBT project:
```scala
libraryDependencies += "com.azure.cosmos.spark" % "azure-cosmos-spark_3-3_2-12" % "4.37.2"
libraryDependencies += "com.azure.cosmos.spark" % "azure-cosmos-spark_3-3_2-12" % "4.38.0"
```

Cosmos DB Spark Connector is available on [Maven Central Repo](https://central.sonatype.com/search?namespace=com.azure.cosmos.spark).
Expand Down
2 changes: 1 addition & 1 deletion sdk/cosmos/azure-cosmos-spark_3-3_2-12/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
</parent>
<groupId>com.azure.cosmos.spark</groupId>
<artifactId>azure-cosmos-spark_3-3_2-12</artifactId>
<version>4.38.0-beta.1</version> <!-- {x-version-update;com.azure.cosmos.spark:azure-cosmos-spark_3-3_2-12;current} -->
<version>4.38.0</version> <!-- {x-version-update;com.azure.cosmos.spark:azure-cosmos-spark_3-3_2-12;current} -->
<packaging>jar</packaging>
<url>https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/cosmos/azure-cosmos-spark_3-3_2-12</url>
<name>OLTP Spark 3.3 Connector for Azure Cosmos DB SQL API</name>
Expand Down
9 changes: 2 additions & 7 deletions sdk/cosmos/azure-cosmos-spark_3-4_2-12/CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,9 @@
## Release History

### 4.38.0-beta.1 (Unreleased)
### 4.38.0 (2025-07-31)

#### Features Added

#### Breaking Changes

#### Bugs Fixed

#### Other Changes
* Added telemetry support by adding OTEL span attribute naming schemes, introducing Azure Monitor integration, and sampled diagnostics. - See [PR 45924](https://github.com/Azure/azure-sdk-for-java/pull/45924)

### 4.37.2 (2025-05-14)

Expand Down
7 changes: 5 additions & 2 deletions sdk/cosmos/azure-cosmos-spark_3-4_2-12/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ https://github.com/Azure/azure-sdk-for-java/issues/new
#### azure-cosmos-spark_3-4_2-12
| Connector | Supported Spark Versions | Supported JVM Versions | Supported Scala Versions | Supported Databricks Runtimes |
|-----------|--------------------------|------------------------|--------------------------|-------------------------------|
| 4.38.0 | 3.4.0 - 3.4.1 | [8, 11] | 2.12 | 13.\* |
| 4.37.2 | 3.4.0 - 3.4.1 | [8, 11] | 2.12 | 13.\* |
| 4.37.1 | 3.4.0 - 3.4.1 | [8, 11] | 2.12 | 13.\* |
| 4.37.0 | 3.4.0 - 3.4.1 | [8, 11] | 2.12 | 13.\* |
Expand Down Expand Up @@ -63,6 +64,7 @@ https://github.com/Azure/azure-sdk-for-java/issues/new
#### azure-cosmos-spark_3-3_2-12
| Connector | Supported Spark Versions | Supported JVM Versions | Supported Scala Versions | Supported Databricks Runtimes |
|-----------|--------------------------|------------------------|--------------------------|-------------------------------|
| 4.38.0 | 3.3.0 - 3.3.2 | [8, 11] | 2.12 | 11.\*, 12.\* |
| 4.37.2 | 3.3.0 - 3.3.2 | [8, 11] | 2.12 | 11.\*, 12.\* |
| 4.37.1 | 3.3.0 - 3.3.2 | [8, 11] | 2.12 | 11.\*, 12.\* |
| 4.37.0 | 3.3.0 - 3.3.2 | [8, 11] | 2.12 | 11.\*, 12.\* |
Expand Down Expand Up @@ -107,6 +109,7 @@ https://github.com/Azure/azure-sdk-for-java/issues/new
#### azure-cosmos-spark_3-5_2-12
| Connector | Supported Spark Versions | Minimum Java Version | Supported Scala Versions | Supported Databricks Runtimes |
|-----------|--------------------------|------------------------|--------------------------|-------------------------------|
| 4.38.0 | 3.5.0 | [8, 11] | 2.12 | 14.*, 15.\* |
| 4.37.2 | 3.5.0 | [8, 11] | 2.12 | 14.*, 15.\* |
| 4.37.1 | 3.5.0 | [8, 11] | 2.12 | 14.*, 15.\* |
| 4.37.0 | 3.5.0 | [8, 11] | 2.12 | 14.*, 15.\* |
Expand All @@ -128,11 +131,11 @@ to use the same version of Scala that Spark was compiled for.
### Download

You can use the maven coordinate of the jar to auto install the Spark Connector to your Databricks Runtime 11 from Maven:
`com.azure.cosmos.spark:azure-cosmos-spark_3-4_2-12:4.37.2`
`com.azure.cosmos.spark:azure-cosmos-spark_3-4_2-12:4.38.0`

You can also integrate against Cosmos DB Spark Connector in your SBT project:
```scala
libraryDependencies += "com.azure.cosmos.spark" % "azure-cosmos-spark_3-4_2-12" % "4.37.2"
libraryDependencies += "com.azure.cosmos.spark" % "azure-cosmos-spark_3-4_2-12" % "4.38.0"
```

Cosmos DB Spark Connector is available on [Maven Central Repo](https://central.sonatype.com/search?namespace=com.azure.cosmos.spark).
Expand Down
2 changes: 1 addition & 1 deletion sdk/cosmos/azure-cosmos-spark_3-4_2-12/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
</parent>
<groupId>com.azure.cosmos.spark</groupId>
<artifactId>azure-cosmos-spark_3-4_2-12</artifactId>
<version>4.38.0-beta.1</version> <!-- {x-version-update;com.azure.cosmos.spark:azure-cosmos-spark_3-4_2-12;current} -->
<version>4.38.0</version> <!-- {x-version-update;com.azure.cosmos.spark:azure-cosmos-spark_3-4_2-12;current} -->
<packaging>jar</packaging>
<url>https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/cosmos/azure-cosmos-spark_3-4_2-12</url>
<name>OLTP Spark 3.4 Connector for Azure Cosmos DB SQL API</name>
Expand Down
7 changes: 2 additions & 5 deletions sdk/cosmos/azure-cosmos-spark_3-5_2-12/CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,9 @@
## Release History

### 4.38.0-beta.1 (Unreleased)
### 4.38.0 (2025-07-31)

#### Features Added

#### Breaking Changes

#### Bugs Fixed
* Added telemetry support by adding OTEL span attribute naming schemes, introducing Azure Monitor integration, and sampled diagnostics. - See [PR 45924](https://github.com/Azure/azure-sdk-for-java/pull/45924)

#### Other Changes
* Added compatibility with CosmosDB Fabric Native Accounts using the `FabricAccountDataResolver` for authentication. - See [PR 45890](https://github.com/Azure/azure-sdk-for-java/pull/45890)
Expand Down
7 changes: 5 additions & 2 deletions sdk/cosmos/azure-cosmos-spark_3-5_2-12/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ https://github.com/Azure/azure-sdk-for-java/issues/new
#### azure-cosmos-spark_3-5_2-12
| Connector | Supported Spark Versions | Minimum Java Version | Supported Scala Versions | Supported Databricks Runtimes |
|-----------|--------------------------|-----------------------|---------------------------|-------------------------------|
| 4.38.0 | 3.5.0 | [8, 11] | 2.12 | 14.\*, 15.\* |
| 4.37.2 | 3.5.0 | [8, 11] | 2.12 | 14.\*, 15.\* |
| 4.37.1 | 3.5.0 | [8, 11] | 2.12 | 14.\*, 15.\* |
| 4.37.0 | 3.5.0 | [8, 11] | 2.12 | 14.\*, 15.\* |
Expand All @@ -49,6 +50,7 @@ to use the same version of Scala that Spark was compiled for.
#### azure-cosmos-spark_3-4_2-12
| Connector | Supported Spark Versions | Supported JVM Versions | Supported Scala Versions | Supported Databricks Runtimes |
|-----------|--------------------------|------------------------|--------------------------|-------------------------------|
| 4.38.0 | 3.4.0 - 3.4.1 | [8, 11] | 2.12 | 13.\* |
| 4.37.2 | 3.4.0 - 3.4.1 | [8, 11] | 2.12 | 13.\* |
| 4.37.1 | 3.4.0 - 3.4.1 | [8, 11] | 2.12 | 13.\* |
| 4.37.0 | 3.4.0 - 3.4.1 | [8, 11] | 2.12 | 13.\* |
Expand Down Expand Up @@ -84,6 +86,7 @@ to use the same version of Scala that Spark was compiled for.
#### azure-cosmos-spark_3-3_2-12
| Connector | Supported Spark Versions | Supported JVM Versions | Supported Scala Versions | Supported Databricks Runtimes |
|-----------|--------------------------|------------------------|--------------------------|-------------------------------|
| 4.38.0 | 3.3.0 - 3.3.2 | [8, 11] | 2.12 | 11.\*, 12.\* |
| 4.37.2 | 3.3.0 - 3.3.2 | [8, 11] | 2.12 | 11.\*, 12.\* |
| 4.37.1 | 3.3.0 - 3.3.2 | [8, 11] | 2.12 | 11.\*, 12.\* |
| 4.37.0 | 3.3.0 - 3.3.2 | [8, 11] | 2.12 | 11.\*, 12.\* |
Expand Down Expand Up @@ -128,11 +131,11 @@ to use the same version of Scala that Spark was compiled for.
### Download

You can use the maven coordinate of the jar to auto install the Spark Connector to your Databricks Runtime 14 from Maven:
`com.azure.cosmos.spark:azure-cosmos-spark_3-5_2-12:4.37.2`
`com.azure.cosmos.spark:azure-cosmos-spark_3-5_2-12:4.38.0`

You can also integrate against Cosmos DB Spark Connector in your SBT project:
```scala
libraryDependencies += "com.azure.cosmos.spark" % "azure-cosmos-spark_3-5_2-12" % "4.37.2"
libraryDependencies += "com.azure.cosmos.spark" % "azure-cosmos-spark_3-5_2-12" % "4.38.0"
```

Cosmos DB Spark Connector is available on [Maven Central Repo](https://central.sonatype.com/search?namespace=com.azure.cosmos.spark).
Expand Down
2 changes: 1 addition & 1 deletion sdk/cosmos/azure-cosmos-spark_3-5_2-12/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
</parent>
<groupId>com.azure.cosmos.spark</groupId>
<artifactId>azure-cosmos-spark_3-5_2-12</artifactId>
<version>4.38.0-beta.1</version> <!-- {x-version-update;com.azure.cosmos.spark:azure-cosmos-spark_3-5_2-12;current} -->
<version>4.38.0</version> <!-- {x-version-update;com.azure.cosmos.spark:azure-cosmos-spark_3-5_2-12;current} -->
<packaging>jar</packaging>
<url>https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/cosmos/azure-cosmos-spark_3-5_2-12</url>
<name>OLTP Spark 3.5 Connector for Azure Cosmos DB SQL API</name>
Expand Down
8 changes: 2 additions & 6 deletions sdk/cosmos/fabric-cosmos-spark-auth_3/CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,7 @@
## Release History
### 1.0.0-beta.1 (Unreleased)

### 1.0.0-beta.1 (2025-07-31)

#### Features Added
* Added account data resolver implementation for Azure Cosmos DB Spark Connector. This allows signed-in users in Fabric to authenticate to CosmosDB using Microsoft Entra ID. - See [PR 45890](https://github.com/Azure/azure-sdk-for-java/pull/45890)

#### Breaking Changes

#### Bugs Fixed

#### Other Changes
4 changes: 2 additions & 2 deletions sdk/cosmos/fabric-cosmos-spark-auth_3/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@
<dependency>
<groupId>com.azure.cosmos.spark</groupId>
<artifactId>azure-cosmos-spark_3-5_2-12</artifactId>
<version>4.38.0-beta.1</version> <!-- {x-version-update;com.azure.cosmos.spark:azure-cosmos-spark_3-5_2-12;current} -->
<version>4.38.0</version> <!-- {x-version-update;com.azure.cosmos.spark:azure-cosmos-spark_3-5_2-12;current} -->
<scope>provided</scope>
</dependency>
<dependency>
Expand Down Expand Up @@ -168,7 +168,7 @@
<include>com.fasterxml.jackson.module:jackson-module-scala_2.12:[2.18.4]</include> <!-- {x-include-update;com.fasterxml.jackson.module:jackson-module-scala_2.12;external_dependency} -->
<include>com.fasterxml.jackson.core:jackson-databind:[2.15.2]</include> <!-- {x-include-update;cosmos_com.fasterxml.jackson.core:jackson-databind;external_dependency} -->
<include>com.fasterxml.jackson.module:jackson-module-scala_2.12:[2.15.2]</include> <!-- {x-include-update;cosmos_com.fasterxml.jackson.module:jackson-module-scala_2.12;external_dependency} -->
<include>com.azure.cosmos.spark:azure-cosmos-spark_3-5_2-12:[4.38.0-beta.1]</include> <!-- {x-include-update;com.azure.cosmos.spark:azure-cosmos-spark_3-5_2-12;current} -->
<include>com.azure.cosmos.spark:azure-cosmos-spark_3-5_2-12:[4.38.0]</include> <!-- {x-include-update;com.azure.cosmos.spark:azure-cosmos-spark_3-5_2-12;current} -->
<include>com.microsoft.azure.synapse:synapseutils_2.12:[1.5.4]</include> <!-- {x-include-update;cosmos_com.microsoft.azure.synapse:synapseutils_2.12;external_dependency} -->
</includes>
</bannedDependencies>
Expand Down