perf(ci): combine sbt lint steps; keep test step separate#4508
Merged
Yicong-Huang merged 2 commits intoApr 26, 2026
Conversation
The scala job ran four separate sbt invocations: scalafmtCheckAll, clean package, scalafixAll --check, and test. Each one paid the same cold-start cost (launcher boot, ~19,160 settings, plugin and dependency resolution) — about 12-15s on the CI runner. Combine the two lint steps into `sbt 'scalafmtCheckAll; scalafixAll --check'` and drop the standalone `clean package` step (test triggers compile via the standard sbt task graph; a fresh CI runner has no stale target/ to clean). Keep `sbt test` as its own step so the GitHub Actions UI still surfaces test runtime distinctly. Move the non-sbt setup steps (Create Databases, Set docker-java) above the sbt steps so they run before either invocation needs them. Closes apache#4507
Yicong-Huang
force-pushed
the
perf/ci-combine-sbt-steps
branch
from
a7289d2 to
691dc08
Compare
Yicong-Huang
changed the title
Yicong-Huang
added a commit
that referenced
this pull request
Apr 26, 2026
### What changes were proposed in this PR? Both the `scala` and `python` jobs in `.github/workflows/github-action-build.yml` install Python dependencies via `pip install -r requirements.txt`. That step alone takes: - `scala` job: **~1m 21s** - `python` matrix: **1m 12s – 4m 41s** (3.13 is slowest because some packages lack prebuilt wheels for it) An earlier attempt on this PR turned on the built-in pip wheel cache (`actions/setup-python` + `cache: 'pip'`) and saw only ~14s saved — the bottleneck is install (resolve + extract + write site-packages for ~230 packages), not download. This PR switches both jobs to `uv pip install --system`. uv is a Rust reimplementation of pip with no transitive deps, so installing it via `python -m pip install uv` adds only ~3s, and the same wheel set then installs in ~10s instead of ~70s on the same runner. No new third-party GitHub Action is added — uv is fetched as a regular pip package — so this stays within the ASF Infra GitHub Actions allowlist already used by this repo (`sbt/setup-sbt`, `coursier/cache-action`, `docker/*`, `amannn/*`, `apache/*`). ### Any related issues, documentation, discussions? Closes #4519. Companion to #4508 (which combined the two lint sbt invocations). ### How was this PR tested? CI on this PR — comparing `Install dependencies` step time before vs after on both `scala` and `python (3.10|3.11|3.12|3.13)` jobs. Earlier exploratory commits on this branch tried caching `target/scala-2.13/{classes,zinc,src_managed}` directly; that broke `scalafix` because zinc skipped the compile that produces the SemanticDB files scalafix needs, so the sbt-target cache was dropped. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.7) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this PR?
The
scalajob in.github/workflows/github-action-build.ymlran four separatesbtinvocations:sbt scalafmtCheckAllsbt clean packagesbt "scalafixAll --check"sbt testEach one paid the same cold-start cost (launcher boot, ~19,160 settings, plugin and dependency resolution) — about 12–15s on the CI runner. Across four invocations, that's ~36–60s of pure repeated startup.
This PR combines the two lint steps into
sbt 'scalafmtCheckAll; scalafixAll --check'and drops the standaloneclean package:testalready triggerscompilevia the sbt task graph, and a fresh CI runner has no staletarget/to clean.sbt teststays as its own step so the GitHub Actions UI still surfaces test runtime distinctly.Non-sbt setup steps (
Create Databases,Set docker-java API version) are moved above the sbt steps so they run before either invocation needs them.Any related issues, documentation, discussions?
Closes #4507
How was this PR tested?
Local measurement with sbt-launcher and
target/warm:sbt scalafmtCheckAllalonesbt 'scalafixAll --check'alonesbt 'scalafmtCheckAll; scalafixAll --check'combinedSaved ~9s locally per pair, which is one sbt cold-start. CI's slower startup means this PR collapses two invocations into one (lint) and drops a third (
clean package), saving ~24–36s on thescalajob. Will be confirmed by CI on this PR.Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.7)