Reduce memory allocation pressure in HttpCompare, wayback, and paramminer

[liquidsec]

Summary

Addresses three of the measured memory hotspots from the memray profiling report (excluding baddns, which is being handled separately):

• HttpCompare baseline spill: After establishing the baseline, the heavy blasthttp Response is replaced with a lightweight _BaselineSnapshot that spills body bytes to the scan's BodySpillStore (disk-backed LRU cache). External code (lightfuzz, webbrute, sqli) sees the same .text, .content, .status_code, .headers API. Frees the Rust-side body allocation for the lifetime of each HttpCompare instance.

• Batch wayback YARA matching: Instead of calling rules.match(data=url) per-URL (100K+ CFFI boundary crossings on large CDX responses), all candidate URLs are concatenated into a newline-delimited blob and matched once. String instance offsets are mapped back to individual URLs via bisect and counted per-URL against the threshold. Measured: ~22 GB cumulative allocation pressure reduced to a single match invocation.

• Paramminer per-URL scoping: already_checked changes from a flat set(hash(word+url)) to dict[url, set(hash(word))]. After finish() processes each URL, its word set is evicted. Frees ~30-100 MB on heavy paramminer scans instead of retaining all word hashes until scan end.

Also audited _module_consumers accounting -- all increment/decrement paths are correctly paired, no actionable leak found.

[github-actions]

📊 Performance Benchmark Report

Comparing dev (baseline) vs more-memory-improvements (current)

📈 Detailed Results (All Benchmarks)

📋 Complete results for all benchmarks - includes both significant and insignificant changes

🧪 Test Name	📏 Base	📏 Current	📈 Change	🎯 Status
Bloom Filter Dns Mutation Tracking Performance	5.10ms	4.33ms	-15.0% 🟢🟢	🚀
Bloom Filter Large Scale Dns Brute Force	18.35ms	18.13ms	-1.2% ⚪	✅
Large Closest Match Lookup	362.78ms	347.14ms	-4.3% ⚪	✅
Realistic Closest Match Workload	194.10ms	198.91ms	+2.5% ⚪	✅
Event Memory Medium Scan	1402 B/event	1402 B/event	+0.0% ⚪	✅
Event Memory Large Scan	1527 B/event	1527 B/event	+0.0% ⚪	✅
Event Validation Full Scan Startup Small Batch	434.32ms	428.68ms	-1.3% ⚪	✅
Event Validation Full Scan Startup Large Batch	594.48ms	564.77ms	-5.0% ⚪	✅
Make Event Autodetection Small	26.27ms	25.32ms	-3.6% ⚪	✅
Make Event Autodetection Large	263.23ms	258.33ms	-1.9% ⚪	✅
Make Event Explicit Types	11.49ms	11.31ms	-1.6% ⚪	✅
Excavate Single Thread Small	4.290s	4.165s	-2.9% ⚪	✅
Excavate Single Thread Large	10.243s	10.056s	-1.8% ⚪	✅
Excavate Parallel Tasks Small	4.475s	4.412s	-1.4% ⚪	✅
Excavate Parallel Tasks Large	6.690s	6.368s	-4.8% ⚪	✅
Intercept Throughput Small	916.61ms	899.06ms	-1.9% ⚪	✅
Intercept Throughput Medium	905.99ms	902.61ms	-0.4% ⚪	✅
Dns Throughput Quiet	2.843s	2.951s	+3.8% ⚪	✅
Dns Throughput Loaded	1.886s	1.876s	-0.5% ⚪	✅
Dns Throughput Inherited	2.479s	2.380s	-4.0% ⚪	✅
Is Ip Performance	2.40ms	2.35ms	-2.3% ⚪	✅
Make Ip Type Performance	255.84µs	255.79µs	-0.0% ⚪	✅
Mixed Ip Operations	2.48ms	2.42ms	-2.5% ⚪	✅
Memory Use Web Crawl	1.0 MB	1.0 MB	+0.0% ⚪	✅
Memory Use Subdomain Enum	28.3 MB	28.3 MB	+0.0% ⚪	✅
Memory Use Deep Chain	5.0 MB	5.0 MB	+0.0% ⚪	✅
Memory Use Parallel Chains	10.3 MB	11.4 MB	+11.1% 🟡🟡	⚠️
Scan Throughput 100	2.974s	2.935s	-1.3% ⚪	✅
Scan Throughput 1000	21.921s	21.411s	-2.3% ⚪	✅
Typical Queue Shuffle	5.56µs	5.46µs	-1.8% ⚪	✅
Priority Queue Shuffle	26.62µs	25.90µs	-2.7% ⚪	✅

🎯 Performance Summary

+ 1 improvement 🚀
! 1 regression ⚠️
  29 unchanged ✅

🔍 Significant Changes (>10%)

• Bloom Filter Dns Mutation Tracking Performance: 15.0% 🚀 faster
• Memory Use Parallel Chains: 11.1% 🐌 more memory

---

🐍 Python Version 3.11.15

[codecov]

Codecov Report

❌ Patch coverage is 96.89922% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 90%. Comparing base (3bc8e9c) to head (07eec1b).

Files with missing lines	Patch %	Lines
bbot/core/helpers/diff.py	91%	4 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff           @@
##             dev   #3213    +/-   ##
======================================
+ Coverage     90%     90%    +1%     
======================================
  Files        453     453            
  Lines      46304   46422   +118     
======================================
+ Hits       41411   41529   +118     
  Misses      4893    4893            

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.