Repository navigation
[3.4] rpc: eth_getBlockReceipts(): fix OOM - bound goroutines and concurren… - #19725
Merged
Merged
Conversation
…cy (#19710) close #19720 Fix goroutine leak in GetReceipts loop: each tx spawned a goroutine waiting on ctx.Done() to cancel the EVM, but ctx was shared across the entire block execution, so all N goroutines stayed alive until GetReceipts returned. Under concurrent requests for different blocks this caused goroutines and EVMs to accumulate in memory, triggering OOM. Fixed by adding a txDone channel closed immediately after ApplyTransactionWithEVM - the goroutine now exits as soon as its tx completes, keeping at most 1 goroutine alive at a time. Add execSem semaphore (capacity max(1, GOMAXPROCS/2), env R_EXEC_CONCURRENCY) to limit the number of blocks executing concurrently in GetReceipts. Each parallel block execution holds its own IntraBlockState which can be hundreds of MB for busy mainnet blocks; without a bound, many concurrent requests for different blocks exhaust memory. Requests already served from the LRU cache bypass the semaphore entirely. ./run_perf_tests.py .... _eth_get_block_receipts_21M_20K.tar -t 500:10,5000:1 with current SW: [3. 1] daemon: executes test qps: 500 time: 10 -> [R=100.00% max=10.597s] [3. 2] daemon: executes test qps: 500 time: 10 -> [R=100.00% max=13.36s] [3. 3] daemon: executes test qps: 500 time: 10 -> [R=100.00% max=26.353s] [3. 4] daemon: executes test qps: 500 time: 10 -> [R=100.00% max=1m56s] [3. 5] daemon: executes test qps: 500 time: 10 -> [R=100.00% max=2m44s] [4. 1] daemon: executes test qps: 5000 time: 1 -> [R=100.00% max=4m11s] [4. 2] daemon: executes test qps: 5000 time: 1 -> test failed: server is Dead for OOM with new SW: [3. 1] daemon: executes test qps: 500 time: 10 -> [R=100.00% max=11.591s] [3. 2] daemon: executes test qps: 500 time: 10 -> [R=100.00% max=5.202s] [3. 3] daemon: executes test qps: 500 time: 10 -> [R=100.00% max=4.947s] [3. 4] daemon: executes test qps: 500 time: 10 -> [R=100.00% max=5.061s] [3. 5] daemon: executes test qps: 500 time: 10 -> [R=100.00% max=5.009s] [4. 1] daemon: executes test qps: 5000 time: 1 -> [R=100.00% max=13.789s] [4. 2] daemon: executes test qps: 5000 time: 1 -> [R=100.00% max=14.032s] [4. 3] daemon: executes test qps: 5000 time: 1 -> [R=100.00% max=14.02s] [4. 4] daemon: executes test qps: 5000 time: 1 -> [R=100.00% max=13.958s] [4. 5] daemon: executes test qps: 5000 time: 1 -> [R=100.00% max=13.924s] Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
AskAlexSharov
approved these changes
Mar 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cherry-pick of #19710
close #19720
Fix goroutine leak in GetReceipts loop: each tx spawned a goroutine waiting on ctx.Done() to cancel the EVM, but ctx was shared across the entire block execution, so all N goroutines stayed alive until GetReceipts returned. Under concurrent requests for different blocks this caused goroutines and EVMs to accumulate in memory, triggering OOM. Fixed by adding a txDone channel closed immediately after ApplyTransactionWithEVM - the goroutine now exits as soon as its tx completes, keeping at most 1 goroutine alive at a time.
Add execSem semaphore (capacity max(1, GOMAXPROCS/2), env R_EXEC_CONCURRENCY) to limit the number of blocks executing concurrently in GetReceipts. Each parallel block execution holds its own IntraBlockState which can be hundreds of MB for busy mainnet blocks; without a bound, many concurrent requests for different blocks exhaust memory. Requests already served from the LRU cache bypass the semaphore entirely.
./run_perf_tests.py .... _eth_get_block_receipts_21M_20K.tar -t 500:10,5000:1
with current SW:
[3. 1] daemon: executes test qps: 500 time: 10 -> [R=100.00% max=10.597s]
[3. 2] daemon: executes test qps: 500 time: 10 -> [R=100.00% max=13.36s] [3. 3] daemon: executes test qps: 500 time: 10 -> [R=100.00% max=26.353s]
[3. 4] daemon: executes test qps: 500 time: 10 -> [R=100.00% max=1m56s] [3. 5] daemon: executes test qps: 500 time: 10 -> [R=100.00% max=2m44s]
[4. 1] daemon: executes test qps: 5000 time: 1 -> [R=100.00% max=4m11s] [4. 2] daemon: executes test qps: 5000 time: 1 -> test failed: server is Dead for OOM
with new SW:
[3. 1] daemon: executes test qps: 500 time: 10 -> [R=100.00% max=11.591s]
[3. 2] daemon: executes test qps: 500 time: 10 -> [R=100.00% max=5.202s] [3. 3] daemon: executes test qps: 500 time: 10 -> [R=100.00% max=4.947s] [3. 4] daemon: executes test qps: 500 time: 10 -> [R=100.00% max=5.061s] [3. 5] daemon: executes test qps: 500 time: 10 -> [R=100.00% max=5.009s]
[4. 1] daemon: executes test qps: 5000 time: 1 -> [R=100.00% max=13.789s]
[4. 2] daemon: executes test qps: 5000 time: 1 -> [R=100.00% max=14.032s]
[4. 3] daemon: executes test qps: 5000 time: 1 -> [R=100.00% max=14.02s] [4. 4] daemon: executes test qps: 5000 time: 1 -> [R=100.00% max=13.958s]
[4. 5] daemon: executes test qps: 5000 time: 1 -> [R=100.00% max=13.924s]