Repository navigation
[Build] Add support for cross-Spark-version building and publishing - #5418
Conversation
tdas
changed the title
| val sparkVersion = settingKey[String]("Spark version") | ||
|
|
||
| // Flag to control cross-Spark publishing: set -DcrossSparkPublish=true to enable | ||
| val crossSparkPublishEnabled = sys.props.getOrElse("crossSparkPublish", "false").toBoolean |
|
I am actually thinking of changing this slightly. if we release this, python support of delta will break as pip-installed delta will dynamically injects spark configuration to force spark to download the maven jar. since the major jar name is changing, this will break. so my approach to not changing the maven jar name by default avoids the existing tests from changing.. but if we dont change the tests, the tests dont ensure that post-release things will work. So i am going to revise this. |
allisonport-db
left a comment
allisonport-db
left a comment
There was a problem hiding this comment.
Overall makes sense just a few comments. I think it would be good if @scottsand-db takes a look when he gets back as well since he did all the original cross compilation work.
| ) | ||
|
|
||
| /** Latest released Spark version */ | ||
| val LATEST_RELEASED = spark35 |
There was a problem hiding this comment.
It's kind of weird that sometimes we determine if isLatest by comparing with this, sometimes via the spec arg isLatest seems like two SOT for this. We can probably remove one.
There was a problem hiding this comment.
yes. good catch. removed.
| * Returns true if the current Spark version is the latest released version. | ||
| */ | ||
| def isLatestReleasedVersion(): Boolean = { | ||
| getSparkVersion() == SparkVersionSpec.LATEST_RELEASED.fullVersion |
There was a problem hiding this comment.
like this could also be getSparkVersionSpec().isLatest
| * Returns the Spark binary version (major.minor) for the given full Spark version. | ||
| * E.g., "3.5.7" -> "3.5", "4.0.2-SNAPSHOT" -> "4.0" | ||
| */ | ||
| def getSparkBinaryVersion(sparkVer: String): String = { |
There was a problem hiding this comment.
same thing here, this is the same exact impl as SparkVersionSpec:: shortVersion why do we have both
| * Latest released Spark version: "module-name" (e.g., delta-spark) | ||
| * Other Spark versions: "module-name_X.Y" (e.g., delta-spark_4.0) |
There was a problem hiding this comment.
Can you explain why we want to release like this? I worry it's confusing to sometimes have the suffix and sometimes not. Is there a main reason why we don't want to always have the suffix? (probably my preference)
There was a problem hiding this comment.
discussed offline. we are keeping the model of Delta default artifact (without Spark version) always supporting latest Spark release. We can break that model in the next major Delta version release if we want to.
There was a problem hiding this comment.
Ha, I asked the same thing here: #5418 (comment)
Btw I worry we are mixing up what "latest" means? Isn't spark 4.0 "latest"? not 3.5?
Question: What does Iceberg do?
There was a problem hiding this comment.
Just want to lay this out to think this through. So, in the current model in this PR, we will:
- delta spark version 4.1:
delta-spark_2.13::4.1is for spark 3.5delta-spark_4.0::4.1is for spark 4.0
Eventually we will move on to spark 4.0 and 4.1 ... suppose that is in delta-spark 4.2
- delta spark version 4.2:
delta-spark_2.13::4.2this is for spark 4.0delta-spark_4.1::4.2this is for spark 4.1
I wonder if it's weird that we have delta-spark_4.0::4.1 is for spark 4.0 for delta-spark 4.1 yet delta-spark_2.13::4.2 is also for spark 4.0 but for delta-spark 4.2 ..
There was a problem hiding this comment.
The invariant is as follows (I should write this up in the description more clearly)
- We always release
delta-spark_{scala_version}to be release on latest version of Spark available at the time of delta release - For older versions of Spark, we will release
delta-spark_{spark_version}_{scala_version}
With this, let me correct your claimed model
delta spark version 4.1: when we do release it, say spark 4.1 is available
delta-spark_2.13::4.1is for spark 4.1delta-spark_4.0_2.13::4.1is for spark 4.0 (note: the_2.13was missing)
delta spark version 4.2: says Spark 4.1 is still latest.
delta-spark_2.13::4.2is for spark 4.1 (because spark 4.1 will be available then)delta-spark_4.0_2.13::4.2is for spark 4.0 (note: the_2.13was missing)
delta spark version 4.3: now Spark 4.2 is available
delta-spark_2.13::4.3this is for spark 4.2delta-spark_4.1_2.13::4.3this is for spark 4.1
The current master branch is treating 4.0.2-snapshot as master is a distraction.. thats not how it should be. I dont want to change the spark versions in this PR in order to keep the change bounded.
Also Spark and Delta versions weirdly aligning right now is also confusing :P
There was a problem hiding this comment.
i added a slightly variation of this naming scheme to the description. hope thats clear.
| * @param allScalaVersions All Scala versions to cross-build | ||
| * @param scala213 Scala 2.13 version string | ||
| */ | ||
| def crossSparkSettings(defaultScalaVersion: SettingKey[String], allScalaVersions: Seq[String], scala213: String): Seq[Setting[_]] = { |
There was a problem hiding this comment.
I find it kind of weird to pass these in here. Should we consider adding the scala version information to the SparkVersionSpec? This seems to make sense to me
| # Scala modules | ||
| "delta-hudi_2.13-{version}.jar", | ||
| "delta-standalone_2.13-{version}.jar", | ||
| "delta-suite-generator_2.13-{version}.jar", |
There was a problem hiding this comment.
@scottsand-db @allisonport-db why are we publishing this? what is this?
There was a problem hiding this comment.
@ZiyaZa PTAL. I think we do not need to publish this, correct?
There was a problem hiding this comment.
It's for development use only, to generate Scala files with test suites. I also think we do not need to publish this.
There was a problem hiding this comment.
@ZiyaZa -- can you please udpate the build.sbt to not publish it, then?
There was a problem hiding this comment.
Created #5518, please let me know if something else also needs to be changed, I don't know how publishing works.
| # Template format: {version} = Delta version (e.g., "3.4.0-SNAPSHOT") | ||
| NON_SPARK_RELATED_JAR_TEMPLATES = [ | ||
| # Scala modules | ||
| "delta-hudi_2.13-{version}.jar", |
There was a problem hiding this comment.
do we need to cross-spark publish this as well?
@tdas -- Any reason not to instead use For example, couldn't we also just as equally argue that it should be |
There was a problem hiding this comment.
is build.yaml the most accurate and specific name here? I would have thought this name would have something to do with Run cross-Spark build test
There was a problem hiding this comment.
i am keeping this as the action for testing all build related stuff (especially those that are not implicitly tested in other actions). cross building is just one of such build related test today.
| ) | ||
|
|
||
| /** Latest released Spark version */ | ||
| val LATEST_RELEASED = spark35 |
There was a problem hiding this comment.
just to confirm: we are sticking with the current impl which is out-of-date in that we are calling spark35 "latest" and spark 4.0 "master"?
that is, in this PR we are explicitly not doing SPARK_VERSION_PREV (3.5) and SPARK_VERSION_LATEST (4.0) and even SPARK_VERSION_MASTER?
There was a problem hiding this comment.
yes. we are not changing the spark version used for building in this PR. thats a separate change. we are just extending the capability to support actually publishing of multiple spark versions.
| releaseSettings, | ||
| CrossSparkVersions.sparkDependentSettings(sparkVersion), | ||
| // Override Jackson versions to match Spark version and avoid conflicts with kernel | ||
| dependencyOverrides ++= { |
There was a problem hiding this comment.
changing the module naming somehow exposed a hidden problem with jackson version conflict
- jackson version is different in kernel and spark.
- when spark module was updated to implicitly depend on kernel (via sparkV2), this jackson module double version kicked in, but default version resolution upgraded to spark's latest version (higher than kernel) for the Spark 4.0 build and things were fine.
- somehow changing the module name xyz_2.13 to xyz_4.0_2.13 caused SBT to use maven version resolution logic instead of ivy resolution logic, causing the two versions of jackson classes to be put in the connectServer assembly jar with the Github actions was running
build/sbt -DsparkVersion=master "++ 2.13.16" clean connectServer/assembly - this change makes the dependency explicitly overridden and be spark-version-specific
allisonport-db
left a comment
allisonport-db
left a comment
There was a problem hiding this comment.
Framework looks awesome, will definitely be a lot easier to expand our multi-version support in the future!!
| * | ||
| * 1. Spark-Dependent Modules (requiresCrossSparkBuild := true): | ||
| * - These depend on Spark and need to be built for each Spark version | ||
| * - Examples: delta-spark, delta-connect-*, delta-sharing-spark, delta-iceberg |
There was a problem hiding this comment.
So do we want to release all of these with all spark versions? For example, I think Delta Sharing's release usually only supports 1 spark version so I'm not sure if this will be possible without also potentially cross-building that artifact. Just a heads up, maybe not urgent at the very moment.
There was a problem hiding this comment.
arent we cross compiling+testing sharing-spark with multiple spark version as well?
so we can release sharing for multiple spark version as well. but whether we should do it is a policy question .. not a ability question. lets decide that offline with the owners of delta-sharing-spark. we can put exclusions fairly easily if needed.
There was a problem hiding this comment.
I think it's a bit of an ability question because delta-sharing-client depends on a spark version (for example, it had to be upgraded for Spark 4.0)
| * Step 2: Publish ONLY Spark-dependent modules for each non-default Spark version | ||
| * build/sbt -DsparkVersion=4.0 "runOnlyForSparkModules publishSigned" | ||
| * | ||
| * This workflow is automated in the release process via crossSparkReleaseSteps(). |
There was a problem hiding this comment.
Does this mean after this PR we will by default release for all spark versions?
There was a problem hiding this comment.
well.. we have the choice. after branch cut we can choose whatever we want to do. thats a policy question, not a framework question. my goal in this PR to have the framework so that we can easily apply whatever policy we want.
| ) | ||
|
|
||
| private val spark40 = SparkVersionSpec( | ||
| fullVersion = "4.0.2-SNAPSHOT", |
There was a problem hiding this comment.
I think either let's rename this spark40snapshot or let's just make this the latest actual released Spark 4.0 version?
There was a problem hiding this comment.
lets do spark40snapshot. i dont want this framework PR to change anything actual builds. thats should be a separate PR.
| * Minimal settings for Spark-dependent modules that don't need full Spark integration. | ||
| * Use this for modules that need versioned artifacts but use default Scala settings. | ||
| * | ||
| * @param sparkVersionKey The sparkVersion setting key for this project | ||
| */ |
There was a problem hiding this comment.
These docs are a little unclear to me
There was a problem hiding this comment.
Maybe clarify when to use this versus sparkDependentSettings? Also I think the name is misleading, it's referred to as "minimal settings" but based on the name I assume it only sets the module name.
There was a problem hiding this comment.
simplified this from 3 variation to 2 variations of similar public methods. one only sets the module name, the other module + settings
| * Cross-Spark Build and Publish System | ||
| * ======================================================== | ||
| * | ||
| * This SBT plugin enables Delta Lake to be built and published for multiple Spark versions. |
There was a problem hiding this comment.
nit: enables Delta Spark
There was a problem hiding this comment.
its not just Delta Spark module. its multiple modules. So i think this statement is okay.
| * - set in the Delta master branch to a Spark released or snapshot version . | ||
| * - not set in the Delta release branches as we want to avoid building against Spark unreleased version. | ||
| * If MASTER is defined, then it can be selected by setting the sparkVersion property to "master". | ||
| * Artifacts for this version HAVE a Spark version suffix (e.g., delta-spark_4.0_2.13). |
There was a problem hiding this comment.
In this example, if MASTER is defined, wouldn't this be more like delta-spark_4.0.1-SNAPSHOT_2.13 ?
i.e. this looks like a final release version of spark, not a "master" version
There was a problem hiding this comment.
aah i was using only spark short version, not the full spark version in the name. have snapshot IN THE NAME is quite weird, but since this is not published beyond any local setup, i dont think it matter too much. i dont have a strong opinion at all, especially since its easy to tweak later of we do find a reason where this actually matters.
| * Artifacts for this version HAVE a Spark version suffix (e.g., delta-spark_4.0_2.13). | ||
| * | ||
| * - OTHER VERSIONS: Any non-default Spark version specified in ALL_SPECS. | ||
| * All non-default versions get a Spark version suffix in their artifact names. |
There was a problem hiding this comment.
perhaps give an example here, to differential this example against the MASTER example above?
| * Artifact Naming Convention | ||
| * ======================================================== | ||
| * | ||
| * Default Spark version artifacts (no suffix): |
There was a problem hiding this comment.
Can you include what the "default" and the "other" spark versions are in this example?
For example: eventually spark 4.0.2 may be the default -- and thus this example will be out of date and may be confusing
Sorry -- just trying to have these examples be as future proof and clear as possible for less confusion down the line
There was a problem hiding this comment.
100% agree to make this future proof. thanks for you eagle eyes catching these.
scottsand-db
left a comment
scottsand-db
left a comment
There was a problem hiding this comment.
I've left some minor comments asking for a bit more clarity in some of the comments/examples/explanations. After those are addressed, LGTM to merge
…elta-io#5418) #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [x] Other (fill in here) - Overall build ## Description This PR improves the cross-Spark version build support by coalescing all related code into one place, and extends it allow publishing artifacts for multiple Spark versions with differential names. **Note**: This is just the framework for version management. This PR is not changing any of the existing Spark build versions, nor is dictating a strict future policy on how many versions should we support. Policy is separate from the framework needed to enforce the policy, and this PR is just for the framework to make implementing any policy easier. ### Core Implementation - **New `CrossSparkVersions.scala` plugin**: Centralizes all Spark version-specific build configuration - Defines `SparkVersionSpec` which allows specifying - Handles JVM version targeting (Java 11 for Spark 3.5, Java 17 for Spark 4.0) - Manages version-specific source directories, ANTLR versions, and JVM options - **Artifact naming scheme**: - Non-Spark-dependent modules: `{module-name}_{scala-version}` (unchanged) - Spark-dependent modules: - when compiling with latest spark - `{module-name}_{scala-version}` - when compiling with non-latest spark - `{module-name}_{spark-version}_{scala-version}` - More details on the artifact naming policy below. - **New SBT command**: `runOnlyForSparkReleasedModules <task>` - Publishes only Spark-dependent modules for a specific Spark version - Usage: `build/sbt -DsparkVersion=4.0.2-SNAPSHOT "runOnlyForSparkModules publishM2"` - Typically this is expected to be run in conjunction with default command to generate all possible artifacts. That is, - Step 1: `build/sbt publishM2` - Step 2: `build/sbt -DsparkVersion=4.0.2-SNAPSHOT "runOnlyForSparkModules publishM2"` - releases all the other spark 4.0 related artifacts, without attempting re-release other non-Spark-related artifacts - **Maven Release integration**: `crossSparkReleaseSteps()` function - Automatically publishes artifacts for all supported Spark versions during releases - Step 1: Publish all modules for Spark 3.5.7 - Step 2: Publish Spark-dependent modules for Spark 4.0 - **Simplified `build.sbt`**: Reduced duplication by using reusable settings from `CrossSparkVersions` ### Artifact Naming Policy - We will always release `delta-spark_{scala_version}` (or such module) on latest version of Spark available at the time of the delta release - For older versions of Spark, we will release delta-spark_{spark_version}_{scala_version} - Example: - delta spark version 4.1: if we release today, latest spark version is 4.0 - `delta-spark_2.13::4.1` is for spark 4.0 - `delta-spark_3.5_2.13::4.1` is for spark 3.5 - delta spark version 4.2: says Spark 4.1 is now the latest. - `delta-spark_2.13::4.2` is for spark 4.1 - `delta-spark_4.0_2.13::4.2` is for spark 4.0 - delta spark version 4.3: Spark 4.2 is still the latest - `delta-spark_2.13::4.3` this is for spark 4.1 - `delta-spark_4.0_2.13::4.3` this is for spark 4.0 - **Note:** The current master branch is treating 4.0.2-snapshot as master is a distraction...thats not how it should be. I dont want to change the spark versions in this PR in order to keep the changes bounded. ## How was this patch tested? - **Comprehensive test suite**: `project/tests/test_cross_spark_publish.py` - Validates artifact naming for all Spark versions - Tests default publish vs selective publish workflows - Auto-validates Spark version consistency between test and build config - Integrated into CI pipeline - **CI integration**: Updated GitHub Actions workflow to run cross-Spark build tests ## Does this PR introduce any user-facing changes? Yes - New artifact naming for non-latest Spark versions: Python users: Automatic - configure_spark_with_delta_pip() detects Spark version and resolves correct artifact. Maven/SBT users: Must use new artifact names for non-latest Spark: // Spark 3.5.x (unchanged) "io.delta" %% "delta-spark" % "3.4.0" // Spark 4.0.x (new) "io.delta" %% "delta-spark_4.0" % "3.4.0" Developers: [Existing] `build/sbt publishM2` - Publish all modules for default Spark version (3.5.7) [New] `build/sbt -DsparkVersion=4.0.2-SNAPSHOT "runOnlyForSparkReleasedModules publishM2"` - publish Spark-dependent modules for Spark 4.0 [New] `build/sbt release` - Full cross-Spark release (automated in sbt-release)
…elta-io#5418) #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [x] Other (fill in here) - Overall build ## Description This PR improves the cross-Spark version build support by coalescing all related code into one place, and extends it allow publishing artifacts for multiple Spark versions with differential names. **Note**: This is just the framework for version management. This PR is not changing any of the existing Spark build versions, nor is dictating a strict future policy on how many versions should we support. Policy is separate from the framework needed to enforce the policy, and this PR is just for the framework to make implementing any policy easier. ### Core Implementation - **New `CrossSparkVersions.scala` plugin**: Centralizes all Spark version-specific build configuration - Defines `SparkVersionSpec` which allows specifying - Handles JVM version targeting (Java 11 for Spark 3.5, Java 17 for Spark 4.0) - Manages version-specific source directories, ANTLR versions, and JVM options - **Artifact naming scheme**: - Non-Spark-dependent modules: `{module-name}_{scala-version}` (unchanged) - Spark-dependent modules: - when compiling with latest spark - `{module-name}_{scala-version}` - when compiling with non-latest spark - `{module-name}_{spark-version}_{scala-version}` - More details on the artifact naming policy below. - **New SBT command**: `runOnlyForSparkReleasedModules <task>` - Publishes only Spark-dependent modules for a specific Spark version - Usage: `build/sbt -DsparkVersion=4.0.2-SNAPSHOT "runOnlyForSparkModules publishM2"` - Typically this is expected to be run in conjunction with default command to generate all possible artifacts. That is, - Step 1: `build/sbt publishM2` - Step 2: `build/sbt -DsparkVersion=4.0.2-SNAPSHOT "runOnlyForSparkModules publishM2"` - releases all the other spark 4.0 related artifacts, without attempting re-release other non-Spark-related artifacts - **Maven Release integration**: `crossSparkReleaseSteps()` function - Automatically publishes artifacts for all supported Spark versions during releases - Step 1: Publish all modules for Spark 3.5.7 - Step 2: Publish Spark-dependent modules for Spark 4.0 - **Simplified `build.sbt`**: Reduced duplication by using reusable settings from `CrossSparkVersions` ### Artifact Naming Policy - We will always release `delta-spark_{scala_version}` (or such module) on latest version of Spark available at the time of the delta release - For older versions of Spark, we will release delta-spark_{spark_version}_{scala_version} - Example: - delta spark version 4.1: if we release today, latest spark version is 4.0 - `delta-spark_2.13::4.1` is for spark 4.0 - `delta-spark_3.5_2.13::4.1` is for spark 3.5 - delta spark version 4.2: says Spark 4.1 is now the latest. - `delta-spark_2.13::4.2` is for spark 4.1 - `delta-spark_4.0_2.13::4.2` is for spark 4.0 - delta spark version 4.3: Spark 4.2 is still the latest - `delta-spark_2.13::4.3` this is for spark 4.1 - `delta-spark_4.0_2.13::4.3` this is for spark 4.0 - **Note:** The current master branch is treating 4.0.2-snapshot as master is a distraction...thats not how it should be. I dont want to change the spark versions in this PR in order to keep the changes bounded. ## How was this patch tested? - **Comprehensive test suite**: `project/tests/test_cross_spark_publish.py` - Validates artifact naming for all Spark versions - Tests default publish vs selective publish workflows - Auto-validates Spark version consistency between test and build config - Integrated into CI pipeline - **CI integration**: Updated GitHub Actions workflow to run cross-Spark build tests ## Does this PR introduce any user-facing changes? Yes - New artifact naming for non-latest Spark versions: Python users: Automatic - configure_spark_with_delta_pip() detects Spark version and resolves correct artifact. Maven/SBT users: Must use new artifact names for non-latest Spark: // Spark 3.5.x (unchanged) "io.delta" %% "delta-spark" % "3.4.0" // Spark 4.0.x (new) "io.delta" %% "delta-spark_4.0" % "3.4.0" Developers: [Existing] `build/sbt publishM2` - Publish all modules for default Spark version (3.5.7) [New] `build/sbt -DsparkVersion=4.0.2-SNAPSHOT "runOnlyForSparkReleasedModules publishM2"` - publish Spark-dependent modules for Spark 4.0 [New] `build/sbt release` - Full cross-Spark release (automated in sbt-release)
…elta-io#5418) #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [x] Other (fill in here) - Overall build ## Description This PR improves the cross-Spark version build support by coalescing all related code into one place, and extends it allow publishing artifacts for multiple Spark versions with differential names. **Note**: This is just the framework for version management. This PR is not changing any of the existing Spark build versions, nor is dictating a strict future policy on how many versions should we support. Policy is separate from the framework needed to enforce the policy, and this PR is just for the framework to make implementing any policy easier. ### Core Implementation - **New `CrossSparkVersions.scala` plugin**: Centralizes all Spark version-specific build configuration - Defines `SparkVersionSpec` which allows specifying - Handles JVM version targeting (Java 11 for Spark 3.5, Java 17 for Spark 4.0) - Manages version-specific source directories, ANTLR versions, and JVM options - **Artifact naming scheme**: - Non-Spark-dependent modules: `{module-name}_{scala-version}` (unchanged) - Spark-dependent modules: - when compiling with latest spark - `{module-name}_{scala-version}` - when compiling with non-latest spark - `{module-name}_{spark-version}_{scala-version}` - More details on the artifact naming policy below. - **New SBT command**: `runOnlyForSparkReleasedModules <task>` - Publishes only Spark-dependent modules for a specific Spark version - Usage: `build/sbt -DsparkVersion=4.0.2-SNAPSHOT "runOnlyForSparkModules publishM2"` - Typically this is expected to be run in conjunction with default command to generate all possible artifacts. That is, - Step 1: `build/sbt publishM2` - Step 2: `build/sbt -DsparkVersion=4.0.2-SNAPSHOT "runOnlyForSparkModules publishM2"` - releases all the other spark 4.0 related artifacts, without attempting re-release other non-Spark-related artifacts - **Maven Release integration**: `crossSparkReleaseSteps()` function - Automatically publishes artifacts for all supported Spark versions during releases - Step 1: Publish all modules for Spark 3.5.7 - Step 2: Publish Spark-dependent modules for Spark 4.0 - **Simplified `build.sbt`**: Reduced duplication by using reusable settings from `CrossSparkVersions` ### Artifact Naming Policy - We will always release `delta-spark_{scala_version}` (or such module) on latest version of Spark available at the time of the delta release - For older versions of Spark, we will release delta-spark_{spark_version}_{scala_version} - Example: - delta spark version 4.1: if we release today, latest spark version is 4.0 - `delta-spark_2.13::4.1` is for spark 4.0 - `delta-spark_3.5_2.13::4.1` is for spark 3.5 - delta spark version 4.2: says Spark 4.1 is now the latest. - `delta-spark_2.13::4.2` is for spark 4.1 - `delta-spark_4.0_2.13::4.2` is for spark 4.0 - delta spark version 4.3: Spark 4.2 is still the latest - `delta-spark_2.13::4.3` this is for spark 4.1 - `delta-spark_4.0_2.13::4.3` this is for spark 4.0 - **Note:** The current master branch is treating 4.0.2-snapshot as master is a distraction...thats not how it should be. I dont want to change the spark versions in this PR in order to keep the changes bounded. ## How was this patch tested? - **Comprehensive test suite**: `project/tests/test_cross_spark_publish.py` - Validates artifact naming for all Spark versions - Tests default publish vs selective publish workflows - Auto-validates Spark version consistency between test and build config - Integrated into CI pipeline - **CI integration**: Updated GitHub Actions workflow to run cross-Spark build tests ## Does this PR introduce any user-facing changes? Yes - New artifact naming for non-latest Spark versions: Python users: Automatic - configure_spark_with_delta_pip() detects Spark version and resolves correct artifact. Maven/SBT users: Must use new artifact names for non-latest Spark: // Spark 3.5.x (unchanged) "io.delta" %% "delta-spark" % "3.4.0" // Spark 4.0.x (new) "io.delta" %% "delta-spark_4.0" % "3.4.0" Developers: [Existing] `build/sbt publishM2` - Publish all modules for default Spark version (3.5.7) [New] `build/sbt -DsparkVersion=4.0.2-SNAPSHOT "runOnlyForSparkReleasedModules publishM2"` - publish Spark-dependent modules for Spark 4.0 [New] `build/sbt release` - Full cross-Spark release (automated in sbt-release)
…elta-io#5418) #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [x] Other (fill in here) - Overall build ## Description This PR improves the cross-Spark version build support by coalescing all related code into one place, and extends it allow publishing artifacts for multiple Spark versions with differential names. **Note**: This is just the framework for version management. This PR is not changing any of the existing Spark build versions, nor is dictating a strict future policy on how many versions should we support. Policy is separate from the framework needed to enforce the policy, and this PR is just for the framework to make implementing any policy easier. ### Core Implementation - **New `CrossSparkVersions.scala` plugin**: Centralizes all Spark version-specific build configuration - Defines `SparkVersionSpec` which allows specifying - Handles JVM version targeting (Java 11 for Spark 3.5, Java 17 for Spark 4.0) - Manages version-specific source directories, ANTLR versions, and JVM options - **Artifact naming scheme**: - Non-Spark-dependent modules: `{module-name}_{scala-version}` (unchanged) - Spark-dependent modules: - when compiling with latest spark - `{module-name}_{scala-version}` - when compiling with non-latest spark - `{module-name}_{spark-version}_{scala-version}` - More details on the artifact naming policy below. - **New SBT command**: `runOnlyForSparkReleasedModules <task>` - Publishes only Spark-dependent modules for a specific Spark version - Usage: `build/sbt -DsparkVersion=4.0.2-SNAPSHOT "runOnlyForSparkModules publishM2"` - Typically this is expected to be run in conjunction with default command to generate all possible artifacts. That is, - Step 1: `build/sbt publishM2` - Step 2: `build/sbt -DsparkVersion=4.0.2-SNAPSHOT "runOnlyForSparkModules publishM2"` - releases all the other spark 4.0 related artifacts, without attempting re-release other non-Spark-related artifacts - **Maven Release integration**: `crossSparkReleaseSteps()` function - Automatically publishes artifacts for all supported Spark versions during releases - Step 1: Publish all modules for Spark 3.5.7 - Step 2: Publish Spark-dependent modules for Spark 4.0 - **Simplified `build.sbt`**: Reduced duplication by using reusable settings from `CrossSparkVersions` ### Artifact Naming Policy - We will always release `delta-spark_{scala_version}` (or such module) on latest version of Spark available at the time of the delta release - For older versions of Spark, we will release delta-spark_{spark_version}_{scala_version} - Example: - delta spark version 4.1: if we release today, latest spark version is 4.0 - `delta-spark_2.13::4.1` is for spark 4.0 - `delta-spark_3.5_2.13::4.1` is for spark 3.5 - delta spark version 4.2: says Spark 4.1 is now the latest. - `delta-spark_2.13::4.2` is for spark 4.1 - `delta-spark_4.0_2.13::4.2` is for spark 4.0 - delta spark version 4.3: Spark 4.2 is still the latest - `delta-spark_2.13::4.3` this is for spark 4.1 - `delta-spark_4.0_2.13::4.3` this is for spark 4.0 - **Note:** The current master branch is treating 4.0.2-snapshot as master is a distraction...thats not how it should be. I dont want to change the spark versions in this PR in order to keep the changes bounded. ## How was this patch tested? - **Comprehensive test suite**: `project/tests/test_cross_spark_publish.py` - Validates artifact naming for all Spark versions - Tests default publish vs selective publish workflows - Auto-validates Spark version consistency between test and build config - Integrated into CI pipeline - **CI integration**: Updated GitHub Actions workflow to run cross-Spark build tests ## Does this PR introduce any user-facing changes? Yes - New artifact naming for non-latest Spark versions: Python users: Automatic - configure_spark_with_delta_pip() detects Spark version and resolves correct artifact. Maven/SBT users: Must use new artifact names for non-latest Spark: // Spark 3.5.x (unchanged) "io.delta" %% "delta-spark" % "3.4.0" // Spark 4.0.x (new) "io.delta" %% "delta-spark_4.0" % "3.4.0" Developers: [Existing] `build/sbt publishM2` - Publish all modules for default Spark version (3.5.7) [New] `build/sbt -DsparkVersion=4.0.2-SNAPSHOT "runOnlyForSparkReleasedModules publishM2"` - publish Spark-dependent modules for Spark 4.0 [New] `build/sbt release` - Full cross-Spark release (automated in sbt-release)
The Building section already states that Java 17 or newer is required (delta-io#6211), but the "Running python tests locally", "IntelliJ Setup" and "Troubleshooting" sections still instruct contributors to use JDK 11. Those instructions date back to delta-io#5377, before cross-Spark-version support (delta-io#5418) moved the build to JDK 17. JDK 17 is what the build actually requires today: every supported Spark version in project/CrossSparkVersions.scala declares targetJvm = "17", and the CI workflows install Java 17 for the Spark, Python, Iceberg, Flink and unidoc jobs. This only touches the contributor-facing setup instructions. The JVM 11 bytecode target in build.sbt and the deliberate JDK 11 runs in kernel_test.yaml / kernel_docs.yaml, which verify that Kernel artifacts stay runnable on JVM 11, are left unchanged. Signed-off-by: genius.choi <csj4032@gmail.com>
5 participants
Which Delta project/connector is this regarding?
Description
This PR improves the cross-Spark version build support by coalescing all related code into one place, and extends it allow publishing artifacts for multiple Spark versions with differential names.
Note: This is just the framework for version management. This PR is not changing any of the existing Spark build versions, nor is dictating a strict future policy on how many versions should we support. Policy is separate from the framework needed to enforce the policy, and this PR is just for the framework to make implementing any policy easier.
Core Implementation
New
CrossSparkVersions.scalaplugin: Centralizes all Spark version-specific build configurationSparkVersionSpecwhich allows specifyingArtifact naming scheme:
{module-name}_{scala-version}(unchanged){module-name}_{scala-version}{module-name}_{spark-version}_{scala-version}New SBT command:
runOnlyForSparkReleasedModules <task>build/sbt -DsparkVersion=4.0.2-SNAPSHOT "runOnlyForSparkModules publishM2"build/sbt publishM2build/sbt -DsparkVersion=4.0.2-SNAPSHOT "runOnlyForSparkModules publishM2"- releases all the other spark 4.0 related artifacts, without attempting re-release other non-Spark-related artifactsMaven Release integration:
crossSparkReleaseSteps()functionSimplified
build.sbt: Reduced duplication by using reusable settings fromCrossSparkVersionsArtifact Naming Policy
delta-spark_{scala_version}(or such module) on latest version of Spark available at the time of the delta release- For older versions of Spark, we will release delta-spark_{spark_version}_{scala_version}
delta spark version 4.1: if we release today, latest spark version is 4.0
delta-spark_2.13::4.1is for spark 4.0delta-spark_3.5_2.13::4.1is for spark 3.5delta spark version 4.2: says Spark 4.1 is now the latest.
delta-spark_2.13::4.2is for spark 4.1delta-spark_4.0_2.13::4.2is for spark 4.0delta spark version 4.3: Spark 4.2 is still the latest
delta-spark_2.13::4.3this is for spark 4.1delta-spark_4.0_2.13::4.3this is for spark 4.0How was this patch tested?
Comprehensive test suite:
project/tests/test_cross_spark_publish.pyCI integration: Updated GitHub Actions workflow to run cross-Spark build tests
Does this PR introduce any user-facing changes?
Yes - New artifact naming for non-latest Spark versions:
Python users: Automatic - configure_spark_with_delta_pip() detects Spark version and resolves correct artifact.
Maven/SBT users: Must use new artifact names for non-latest Spark:
// Spark 3.5.x (unchanged)
"io.delta" %% "delta-spark" % "3.4.0"
// Spark 4.0.x (new)
"io.delta" %% "delta-spark_4.0" % "3.4.0"
Developers:
[Existing]
build/sbt publishM2- Publish all modules for default Spark version (3.5.7)[New]
build/sbt -DsparkVersion=4.0.2-SNAPSHOT "runOnlyForSparkReleasedModules publishM2"- publish Spark-dependent modules for Spark 4.0[New]
build/sbt release- Full cross-Spark release (automated in sbt-release)