[CORE][SPARK-49788] Add optional TTL based cleaning for blocks to better support notebooks - #49032

Draft

holdenk wants to merge 29 commits into

apache:masterfrom

holdenk:SPARK-49788-Restore-Cleaner-TTL-Functionality

Commits on Aug 11, 2026

start thinking about cleanup timers

holdenk
authored and
sfc-gh-hkarau
committed
f6eed2f
Start a bit on putting in block tracking.

holdenk
authored and
sfc-gh-hkarau
committed
10da40b
Test compiles

holdenk
authored and
sfc-gh-hkarau
committed
85dada7
Add the Block TTL Integration suite

holdenk
authored and
sfc-gh-hkarau
committed
40168ca
Increase logging a little bit.

holdenk
authored and
sfc-gh-hkarau
committed
75769fb
Make the TTL cleaners clean and add some simple tests

holdenk
authored and
sfc-gh-hkarau
committed
4742448
Switch to using MDC logging

holdenk
authored and
sfc-gh-hkarau
committed
c62dd9c
hmmm fails in CI maybe our TTL is too tight

holdenk
authored and
sfc-gh-hkarau
committed
4ba1b73
Bump TTL a bit

holdenk
authored and
sfc-gh-hkarau
committed
815420e
Use eventually instead of sleep

holdenk
authored and
sfc-gh-hkarau
committed
45949d2
Back out un-needed change to ShuffledRDD.scala
sfc-gh-hkarau
committed
93a67e3
Add a note about why JHashMap
sfc-gh-hkarau
committed
8d507b0
Add a version 4.1.0 to the new config options.
sfc-gh-hkarau
committed
3334f22
CR feedback
sfc-gh-hkarau
committed
0a17c61
Deal with the interruption excdeption from shutdownnow
sfc-gh-hkarau
committed
7d824ad
Reduce the TTL in the tests so they don't take so long, move the time printed out on failure inside of the eventual block, remove RDD/Shuffle from access time tracking regardless of if RDD/shuffle …
sfc-gh-hkarau
committed
9d76458
Increase TTL so the blocks don't get cleaned before the test has a chance to check for them (mostly needed for GHA where we run in rather resource constrained machines).
sfc-gh-hkarau
committed
0107e28
Bump up the blockTTL for testing since CI seems to cleanup the blocks faster than we can get to them
sfc-gh-hkarau
committed
2c2fe0c
hmmm
sfc-gh-hkarau
committed
f384753
Make TTL cleaner thread-safe: concurrent atime maps + route RDD removal through the endpoint loop

sfc-gh-hkarau
and
claude
committed
f7074f7

Commits on Aug 12, 2026

Address max code-review: TTL cleaner correctness/cleanup (Tier B+C)

sfc-gh-hkarau
and
claude
committed
68347fa

Commits on Aug 13, 2026

Shuffle TTL reclaims disk via the normal remove path; refresh shuffle atime on map output (F2/F3)

sfc-gh-hkarau
and
claude
committed
e0d4f44

Commits on Aug 14, 2026

Address max review round 2: fix X1 abort regression + dedup/cleanup

sfc-gh-hkarau
and
claude
committed
1e126c3
Fix scalastyle line-length failures and config version tag

sfc-gh-hkarau
and
claude
committed
878896c
Don't unregister the ShuffleStatus when reaping: it could kill the SparkContext

sfc-gh-hkarau
and
claude
committed
df7458b
Don't reap local checkpoints, reject non-positive TTLs, complete the reclamation path

sfc-gh-hkarau
and
claude
committed
075607e
Reap RDDs through ContextCleaner.doCleanupRDD instead of re-implementing it

sfc-gh-hkarau
and
claude
committed
e2c7f97
Fix the local-checkpoint gate defeating itself, plus phantom atimes and cleaner shutdown

sfc-gh-hkarau
and
claude
committed
d3530ae
[DOCS] Spell out the shuffle TTL's job-failure tail

sfc-gh-hkarau
and
claude
committed
87be8e6