Memory Optimizations

[liquidsec]

Summary

• Intern tag and event type strings with sys.intern() so identical strings across events share one object
• Move _discovery_context_regex from per-instance (1440 bytes each) to class variable
• Add envelopes to __slots__ to prevent __dict__ creation on WEB_PARAMETER events
• Pre-lowercase excavate blacklist prefixes in setup() instead of per-call
• Move redundant re.compile() calls to class level in excavate and httpx
• Add memory benchmark tests and update benchmark report to display memory metrics

[github-actions]

📊 Performance Benchmark Report

Comparing 3.0 (baseline) vs memory-optimizations (current)

📈 Detailed Results (All Benchmarks)

📋 Complete results for all benchmarks - includes both significant and insignificant changes

🧪 Test Name	📏 Base	📏 Current	📈 Change	🎯 Status
Bloom Filter Dns Mutation Tracking Performance	4.37ms	4.26ms	-2.4% ⚪	✅
Bloom Filter Large Scale Dns Brute Force	17.74ms	18.33ms	+3.3% ⚪	✅
Large Closest Match Lookup	368.33ms	356.97ms	-3.1% ⚪	✅
Realistic Closest Match Workload	195.48ms	192.57ms	-1.5% ⚪	✅
Event Memory Medium Scan	-	2.093s	New 🆕	🆕
Event Memory Large Scan	-	10.628s	New 🆕	🆕
Event Validation Full Scan Startup Small Batch	496.12ms	486.76ms	-1.9% ⚪	✅
Event Validation Full Scan Startup Large Batch	770.59ms	757.16ms	-1.7% ⚪	✅
Make Event Autodetection Small	31.40ms	30.74ms	-2.1% ⚪	✅
Make Event Autodetection Large	316.03ms	316.34ms	+0.1% ⚪	✅
Make Event Explicit Types	13.90ms	13.92ms	+0.1% ⚪	✅
Excavate Single Thread Small	4.098s	4.151s	+1.3% ⚪	✅
Excavate Single Thread Large	9.815s	9.824s	+0.1% ⚪	✅
Excavate Parallel Tasks Small	4.287s	4.281s	-0.1% ⚪	✅
Excavate Parallel Tasks Large	7.407s	7.387s	-0.3% ⚪	✅
Is Ip Performance	3.14ms	3.13ms	-0.4% ⚪	✅
Make Ip Type Performance	11.24ms	11.28ms	+0.4% ⚪	✅
Mixed Ip Operations	4.41ms	4.40ms	-0.2% ⚪	✅
Typical Queue Shuffle	62.61µs	60.88µs	-2.8% ⚪	✅
Priority Queue Shuffle	712.52µs	701.84µs	-1.5% ⚪	✅

🎯 Performance Summary

✅ No significant performance changes detected (all changes <10%)

🔍 Significant Changes (>10%)

• Event Memory Medium Scan: New test 🆕 (2.093s, 4.8K ops/sec)
• Event Memory Large Scan: New test 🆕 (10.628s, 4.7K ops/sec)

---

🐍 Python Version 3.11.14

[codecov]

Codecov Report

❌ Patch coverage is 28.76712% with 52 lines in your changes missing coverage. Please review.
✅ Project coverage is 92%. Comparing base (389b8aa) to head (4aa554c).
⚠️ Report is 5 commits behind head on 3.0.

Files with missing lines	Patch %	Lines
...ot/test/benchmarks/test_event_memory_benchmarks.py	19%	52 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff          @@
##             3.0   #2953   +/-   ##
=====================================
- Coverage     92%     92%   -0%     
=====================================
  Files        436     437    +1     
  Lines      35642   35710   +68     
=====================================
+ Hits       32504   32537   +33     
- Misses      3138    3173   +35     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.