gowitness: ephemeral per-batch database

[liquidsec]

Problem

On large scans the gowitness module OOMs, and floods the log with
no such table: results. Two root causes:

1. Unbounded memory. The module used one shared gowitness.sqlite3 for the
   entire scan and, every batch, re-ran SELECT * over the whole (ever-growing)
   results / network_logs / technologies tables, deduping in per-scan
   Python sets. network_logs explodes on big scans (a row per sub-request per
   page), so memory and per-batch work grew with cumulative scan size. Only
   happens with gowitness enabled.

2. no such table: results flood. The readers ran SELECT * unguarded.
   When chrome fails to launch, gowitness exits before creating its schema, so
   the query raised OperationalError: no such table: results, surfaced as
   Error in gowitness.handle_batch(...) for every affected batch.

Fix

• Each batch gets its own throwaway database in scan.temp_dir, deleted in a
  finally once events are emitted. Memory stays flat regardless of scan size.
• A batch whose database is missing or unreadable -- chrome failed to launch, so
  gowitness exited before writing its schema -- now logs a single warning
  (Gowitness produced no results for N URLs; chrome may have failed to launch)
  and the scan continues, instead of flooding the log per row.
• Drops the persistent DB, the gowitness report server gallery hint, and the
  per-scan binary copy -- intentional; the gallery depended on the persistent DB.

Screenshots still persist on disk and as WEBSCREENSHOT events.

[github-actions]

📊 Performance Benchmark Report

Comparing dev (baseline) vs gowitness-per-batch-db (current)

📈 Detailed Results (All Benchmarks)

📋 Complete results for all benchmarks - includes both significant and insignificant changes

🧪 Test Name	📏 Base	📏 Current	📈 Change	🎯 Status
Bloom Filter Dns Mutation Tracking Performance	3.97ms	3.90ms	-1.8% ⚪	✅
Bloom Filter Large Scale Dns Brute Force	17.89ms	18.08ms	+1.0% ⚪	✅
Large Closest Match Lookup	335.43ms	317.92ms	-5.2% ⚪	✅
Realistic Closest Match Workload	176.14ms	179.11ms	+1.7% ⚪	✅
Event Memory Medium Scan	1402 B/event	1402 B/event	+0.0% ⚪	✅
Event Memory Large Scan	1527 B/event	1527 B/event	+0.0% ⚪	✅
Event Validation Full Scan Startup Small Batch	399.23ms	399.23ms	+0.0% ⚪	✅
Event Validation Full Scan Startup Large Batch	511.29ms	514.11ms	+0.6% ⚪	✅
Make Event Autodetection Small	20.64ms	21.34ms	+3.4% ⚪	✅
Make Event Autodetection Large	210.35ms	217.52ms	+3.4% ⚪	✅
Make Event Explicit Types	9.01ms	9.27ms	+2.8% ⚪	✅
Excavate Single Thread Small	3.692s	3.690s	-0.0% ⚪	✅
Excavate Single Thread Large	9.823s	9.723s	-1.0% ⚪	✅
Excavate Parallel Tasks Small	3.904s	3.834s	-1.8% ⚪	✅
Excavate Parallel Tasks Large	6.390s	6.250s	-2.2% ⚪	✅
Intercept Throughput Small	865.68ms	863.09ms	-0.3% ⚪	✅
Intercept Throughput Medium	863.18ms	909.92ms	+5.4% ⚪	✅
Dns Throughput Quiet	2.370s	2.364s	-0.2% ⚪	✅
Dns Throughput Loaded	1.621s	1.728s	+6.6% ⚪	✅
Dns Throughput Inherited	2.236s	2.138s	-4.4% ⚪	✅
Is Ip Performance	1.98ms	1.99ms	+0.9% ⚪	✅
Make Ip Type Performance	165.72µs	171.13µs	+3.3% ⚪	✅
Mixed Ip Operations	2.06ms	2.07ms	+0.4% ⚪	✅
Memory Use Web Crawl	1.0 MB	1.0 MB	+0.0% ⚪	✅
Memory Use Subdomain Enum	28.3 MB	28.3 MB	+0.0% ⚪	✅
Memory Use Deep Chain	5.0 MB	5.0 MB	+0.0% ⚪	✅
Memory Use Parallel Chains	12.2 MB	11.5 MB	-5.7% ⚪	✅
Scan Throughput 100	2.547s	2.681s	+5.3% ⚪	✅
Scan Throughput 1000	18.009s	18.304s	+1.6% ⚪	✅
Typical Queue Shuffle	4.90µs	4.91µs	+0.2% ⚪	✅
Priority Queue Shuffle	23.27µs	23.14µs	-0.6% ⚪	✅

🎯 Performance Summary

✅ No significant performance changes detected (all changes <10%)

---

🐍 Python Version 3.11.15

[codecov]

Codecov Report

❌ Patch coverage is 78.66667% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 90%. Comparing base (3bc8e9c) to head (e381ee5).
⚠️ Report is 9 commits behind head on dev.

Files with missing lines	Patch %	Lines
bbot/modules/gowitness.py	79%	16 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff          @@
##             dev   #3241   +/-   ##
=====================================
+ Coverage     90%     90%   +1%     
=====================================
  Files        453     453           
  Lines      46304   46258   -46     
=====================================
- Hits       41411   41371   -40     
+ Misses      4893    4887    -6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.