Repository navigation

[SPARK-1103] [WIP] Automatic garbage collection of RDD, shuffle and broadcast data - #126

Closed

tdas wants to merge 51 commits into

apache:masterfrom

tdas:state-cleanup

Closed

[SPARK-1103] [WIP] Automatic garbage collection of RDD, shuffle and broadcast data#126
tdas wants to merge 51 commits into
apache:masterfrom
tdas:state-cleanup

Commits on Feb 5, 2014
- Added unpersist method to Broadcast.
  Roman Pastukhov
  committed
  1e752f1
Commits on Feb 6, 2014
- Fix for Broadcast unpersist patch.
  Roman Pastukhov
  committed
  80dd977
Commits on Feb 14, 2014
- Added ContextCleaner to automatically clean RDDs and shuffles when they fall out of scope. Also replaced TimeStampedHashMap to BoundedHashMaps and TimeStampedWeakValueHashMap for the necessary hash…
  tdas
  committed
  e427a9e
- Changed TimeStampedHashMap to use WrappedJavaHashMap.
  tdas
  committed
  8512612
Commits on Mar 11, 2014
- Merge remote-tracking branch 'apache/master' into state-cleanup
  tdas
  committed
  a24fefc
- Fixed docs and styles.
  tdas
  committed
  cb0a5a6
Commits on Mar 12, 2014
- Removed unncessary TimeStampedHashMap from DAGScheduler, added try-catches in finalize() methods, and replaced ArrayBlockingQueue to LinkedBlockingQueue to avoid blocking in Java's finalizing thread.
  tdas
  committed
  ae9da88
Commits on Mar 13, 2014
- Modifications based on the comments on PR 126.
  tdas
  committed
  e61daa0
Commits on Mar 17, 2014
- Added try-catch in context cleaner and null value cleaning in TimeStampedWeakValueHashMap.
  tdas
  committed
  a7260d3
Commits on Mar 18, 2014
- Removed use of BoundedHashMap, and made BlockManagerSlaveActor cleanup shuffle metadata in MapOutputTrackerWorker.
  tdas
  committed
  892b952
Commits on Mar 19, 2014
- Style fix
  tdas
  committed
  e1fba5f
Commits on Mar 25, 2014
- Changed ContextCleaner to use ReferenceQueue instead of finalizer
  tdas
  committed
  f2881fd
- Changes based on PR comments.
  tdas
  committed
  620eca3
- Merge remote-tracking branch 'apache/master' into state-cleanup
  tdas
  committed
  a007307
- Removed duplicate unpersistRDD.
  tdas
  committed
  d2f8b97
- Added missing Apache license
  tdas
  committed
  6c9dcf6
Commits on Mar 26, 2014
- Merge branch 'bc-unpersist-merge' of github.com:ignatich/incubator-spark into cleanup
  andrewor14
  committed
  c7ccef1
- Refactor broadcast classes
  andrewor14
  committed
  ba52e00
- Add framework for broadcast cleanup
  andrewor14
  committed
  d0edef3
- Clean up broadcast blocks through BlockManager*
  andrewor14
  committed
  544ac86
Commits on Mar 27, 2014
- Add tests for unpersisting broadcast
  andrewor14
  committed
  e95479c
- Test broadcast cleanup in ContextCleanerSuite + remove BoundedHashMap
  andrewor14
  committed
  f201a8d
- Merge github.com:apache/spark into cleanup
  andrewor14
  committed
  c92e4d9
Commits on Mar 28, 2014
- Import, comments, and style fixes (minor)
  andrewor14
  committed
  0d17060
- Generalize BroadcastBlockId to remove BroadcastHelperBlockId
  andrewor14
  committed
  34f436f
Commits on Mar 29, 2014
Commits on Mar 30, 2014
- Merge github.com:apache/spark into cleanup
  andrewor14
  committed
  8557c12
Commits on Mar 31, 2014
Commits on Apr 1, 2014
- Address TD's comments
  andrewor14
  committed
  5016375
Commits on Apr 2, 2014
- Correct semantics for TimeStampedWeakValueHashMap + add tests
  andrewor14
  committed
  f0aabb1
- Merge pull request #1 from andrewor14/cleanup
  tdas
  committed
  762a4d8
Commits on Apr 4, 2014
- Merge github.com:apache/spark into cleanup
  andrewor14
  committed
  a6460d4
- Address Patrick's comments
  andrewor14
  committed
  c5b1d98
- Merge remote-tracking branch 'apache/master' into state-cleanup
  tdas
  committed
  a2cc8bc
- Merge branch 'state-cleanup' of github.com:tdas/spark into cleanup
  andrewor14
  committed
  ada45f0
- Make automatic cleanup configurable (not documented)
  andrewor14
  committed
  cd72d19
- Merge pull request #3 from andrewor14/cleanup
  tdas
  committed
  b27f8e8
- Fixed compilation errors.
  tdas
  committed
  a430f06
- Fixed failing BroadcastSuite unit tests by introducing blocking for removeShuffle and removeBroadcast in BlockManager*
  tdas
  committed
  104a89a
- Fixed bug and adding unit test for removeBroadcast in BlockManagerSuite.
  tdas
  committed
  6222697
Commits on Apr 7, 2014
Commits on Apr 8, 2014
- Merge remote-tracking branch 'apache/master' into state-cleanup
  tdas
  committed
  f489fdc
- Fixed issue with Tachyon + new BlockManager methods.
  tdas
  committed
  61b8d6e

Commits

Commits on Feb 5, 2014

Commits on Feb 6, 2014

Commits on Feb 14, 2014

Commits on Mar 11, 2014

Commits on Mar 12, 2014

Commits on Mar 13, 2014

Commits on Mar 17, 2014

Commits on Mar 18, 2014

Commits on Mar 19, 2014

Commits on Mar 25, 2014

Commits on Mar 26, 2014

Commits on Mar 27, 2014

Commits on Mar 28, 2014

Commits on Mar 29, 2014

Commits on Mar 30, 2014

Commits on Mar 31, 2014

Commits on Apr 1, 2014

Commits on Apr 2, 2014

Commits on Apr 4, 2014

Commits on Apr 7, 2014

Commits on Apr 8, 2014