[lupin012]

close #19720

Fix goroutine leak in GetReceipts loop: each tx spawned a goroutine waiting on ctx.Done() to cancel the EVM, but ctx was shared across the entire block execution, so all N goroutines stayed alive until GetReceipts returned. Under concurrent requests for different blocks this caused goroutines and EVMs to accumulate in memory, triggering OOM. Fixed by adding a txDone channel closed immediately after ApplyTransactionWithEVM - the goroutine now exits as soon as its tx completes, keeping at most 1 goroutine alive at a time.

Add execSem semaphore (capacity max(1, GOMAXPROCS/2), env R_EXEC_CONCURRENCY) to limit the number of blocks executing concurrently in GetReceipts. Each parallel block execution holds its own IntraBlockState which can be hundreds of MB for busy mainnet blocks; without a bound, many concurrent requests for different blocks exhaust memory. Requests already served from the LRU cache bypass the semaphore entirely.

./run_perf_tests.py .... _eth_get_block_receipts_21M_20K.tar -t 500:10,5000:1

with current SW:
[3. 1] daemon: executes test qps: 500 time: 10 -> [R=100.00% max=10.597s]
[3. 2] daemon: executes test qps: 500 time: 10 -> [R=100.00% max=13.36s]
[3. 3] daemon: executes test qps: 500 time: 10 -> [R=100.00% max=26.353s]
[3. 4] daemon: executes test qps: 500 time: 10 -> [R=100.00% max=1m56s]
[3. 5] daemon: executes test qps: 500 time: 10 -> [R=100.00% max=2m44s]

[4. 1] daemon: executes test qps: 5000 time: 1 -> [R=100.00% max=4m11s]
[4. 2] daemon: executes test qps: 5000 time: 1 -> test failed: server is Dead for OOM

with new SW:
[3. 1] daemon: executes test qps: 500 time: 10 -> [R=100.00% max=11.591s]
[3. 2] daemon: executes test qps: 500 time: 10 -> [R=100.00% max=5.202s]
[3. 3] daemon: executes test qps: 500 time: 10 -> [R=100.00% max=4.947s]
[3. 4] daemon: executes test qps: 500 time: 10 -> [R=100.00% max=5.061s]
[3. 5] daemon: executes test qps: 500 time: 10 -> [R=100.00% max=5.009s]

[4. 1] daemon: executes test qps: 5000 time: 1 -> [R=100.00% max=13.789s]
[4. 2] daemon: executes test qps: 5000 time: 1 -> [R=100.00% max=14.032s]
[4. 3] daemon: executes test qps: 5000 time: 1 -> [R=100.00% max=14.02s]
[4. 4] daemon: executes test qps: 5000 time: 1 -> [R=100.00% max=13.958s]
[4. 5] daemon: executes test qps: 5000 time: 1 -> [R=100.00% max=13.924s]

[AskAlexSharov]

yes. bounded concurrency for historical re-exec of blocks - is very helpful.

We also have other re-exec API's: debug_traceBlock, trace_block - which basically doing same thing (eating same resources).

Feel free to pick it to 3.4 - because in my head this PR greatly reducing impact of RPC to ChainTip - and "reducing things impact to ChainTip" - is primary feature of 3.4 (for example i did work much on reducing files-Merge impact to ChainTip).

[lupin012]

@AskAlexSharov After the merge in main, I will cherry-pick it to 3.4. I'll also investigate trace_block and debug_traceBlock; I did a quick check earlier and didn't see any parallel goroutines, but I'll double-check to be sure