Memory backpressure: pause ingress when system RAM is high

[liquidsec]

Summary

Despite the improvements to memory use added in 38db27e (refcount-based HTTP response body stripping), BBOT scans are still ultimately unbounded when it comes to memory, and CAN fill up the entire volume of system memory in some cases. To address that reality, this PR creates a memory backpressure system, which will pause ingress of new events once memory passes a threshold. This gives existing events time to finish processing and have their heavy payloads (HTTP response bodies, headers, etc.) stripped via _minimize(), freeing memory before more work piles on.

How it works

• The scanner's existing modules_status() tick (every 15s) already checked memory and logged a warning — but did nothing. Now it acts on it.
• An asyncio.Event (_memory_ok) is set by default (= proceed). When system memory exceeds max_mem_percent (default 90%), the event is cleared, which pauses ScanIngress.get_incoming_event() — the single chokepoint where new events enter the pipeline.
• Only ingress is paused. Module workers keep draining their queues and calling _minimize() on finished events, which is what actually frees memory.
• Once memory drops 5 points below the threshold (low watermark), ingress resumes.
• Safety valve: if memory hasn't dropped after 120 seconds of pausing, ingress force-resumes to prevent stuck scans. This is critical for headless/agent deployments where no user is watching.

Config

One new setting: max_mem_percent (default 90). Can be overridden via CLI:

bbot -t example.com -c max_mem_percent=50

Files changed

• bbot/scanner/scanner.py — _memory_ok event, pause/resume/force-resume logic in modules_status()
• bbot/scanner/manager.py — await _memory_ok in ScanIngress.get_incoming_event()
• bbot/defaults.yml + docs/scanning/configuration.md — new max_mem_percent setting
• bbot/test/test_step_1/test_scan.py — tests for all state transitions (pause, stay paused in gap, resume, force-resume)

[github-actions]

📊 Performance Benchmark Report

Comparing blasthttp-integration-clean (baseline) vs memory-backpressure (current)

📈 Detailed Results (All Benchmarks)

📋 Complete results for all benchmarks - includes both significant and insignificant changes

🧪 Test Name	📏 Base	📏 Current	📈 Change	🎯 Status
Bloom Filter Dns Mutation Tracking Performance	4.16ms	4.20ms	+1.0% ⚪	✅
Bloom Filter Large Scale Dns Brute Force	17.44ms	17.99ms	+3.2% ⚪	✅
Large Closest Match Lookup	356.05ms	356.34ms	+0.1% ⚪	✅
Realistic Closest Match Workload	190.94ms	188.17ms	-1.5% ⚪	✅
Event Memory Medium Scan	1782 B/event	1779 B/event	-0.2% ⚪	✅
Event Memory Large Scan	1768 B/event	1765 B/event	-0.2% ⚪	✅
Event Validation Full Scan Startup Small Batch	405.10ms	409.92ms	+1.2% ⚪	✅
Event Validation Full Scan Startup Large Batch	579.02ms	579.93ms	+0.2% ⚪	✅
Make Event Autodetection Small	31.08ms	30.91ms	-0.5% ⚪	✅
Make Event Autodetection Large	315.50ms	312.94ms	-0.8% ⚪	✅
Make Event Explicit Types	13.87ms	14.01ms	+1.0% ⚪	✅
Excavate Single Thread Small	3.886s	4.000s	+2.9% ⚪	✅
Excavate Single Thread Large	9.599s	9.717s	+1.2% ⚪	✅
Excavate Parallel Tasks Small	4.104s	4.332s	+5.6% ⚪	✅
Excavate Parallel Tasks Large	6.550s	6.738s	+2.9% ⚪	✅
Is Ip Performance	3.17ms	3.20ms	+0.8% ⚪	✅
Make Ip Type Performance	11.58ms	11.63ms	+0.5% ⚪	✅
Mixed Ip Operations	4.50ms	4.53ms	+0.7% ⚪	✅
Memory Use Web Crawl	159.3 MB	163.9 MB	+2.9% ⚪	✅
Memory Use Subdomain Enum	19.4 MB	19.4 MB	-0.1% ⚪	✅
Scan Throughput 100	4.522s	5.041s	+11.5% 🟡🟡	⚠️
Scan Throughput 1000	36.549s	40.255s	+10.1% 🟡🟡	⚠️
Typical Queue Shuffle	66.43µs	64.27µs	-3.2% ⚪	✅
Priority Queue Shuffle	746.22µs	719.26µs	-3.6% ⚪	✅

🎯 Performance Summary

! 2 regressions ⚠️
  22 unchanged ✅

🔍 Significant Changes (>10%)

• Scan Throughput 100: 11.5% 🐌 slower
• Scan Throughput 1000: 10.1% 🐌 slower

---

🐍 Python Version 3.11.15

[codecov]

Codecov Report

❌ Patch coverage is 96.51163% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 91%. Comparing base (0d68bcc) to head (d4d0e74).
⚠️ Report is 163 commits behind head on blasthttp-integration-clean.

Files with missing lines	Patch %	Lines
bbot/scanner/manager.py	50%	2 Missing ⚠️
bbot/scanner/scanner.py	98%	1 Missing ⚠️

Additional details and impacted files

@@                     Coverage Diff                      @@
##           blasthttp-integration-clean   #3040    +/-   ##
============================================================
- Coverage                           91%     91%    -0%     
============================================================
  Files                              443     443            
  Lines                            37835   38368   +533     
============================================================
+ Hits                             34237   34691   +454     
- Misses                            3598    3677    +79     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[liquidsec]

still testing this out

[liquidsec]

closing in favor of a scaled down version of this implemented in #3085