Skip to content

Fix gowitness bug#2974

Merged
TheTechromancer merged 2 commits into
3.0from
fix-gowitness-bug
Mar 17, 2026
Merged

Fix gowitness bug#2974
TheTechromancer merged 2 commits into
3.0from
fix-gowitness-bug

Conversation

@TheTechromancer

Copy link
Copy Markdown
Contributor

No description provided.

@github-actions

Copy link
Copy Markdown
Contributor

📊 Performance Benchmark Report

Comparing 3.0 (baseline) vs fix-gowitness-bug (current)

📈 Detailed Results (All Benchmarks)

📋 Complete results for all benchmarks - includes both significant and insignificant changes

🧪 Test Name 📏 Base 📏 Current 📈 Change 🎯 Status
Bloom Filter Dns Mutation Tracking Performance 4.28ms 4.19ms -2.1%
Bloom Filter Large Scale Dns Brute Force 17.79ms 17.55ms -1.4%
Large Closest Match Lookup 356.42ms 369.64ms +3.7%
Realistic Closest Match Workload 188.04ms 198.45ms +5.5%
Event Memory Medium Scan 1769 B/event 1768 B/event -0.1%
Event Memory Large Scan 1757 B/event 1755 B/event -0.1%
Event Validation Full Scan Startup Small Batch 490.26ms 504.62ms +2.9%
Event Validation Full Scan Startup Large Batch 786.27ms 804.30ms +2.3%
Make Event Autodetection Small 30.81ms 30.64ms -0.6%
Make Event Autodetection Large 315.12ms 316.22ms +0.3%
Make Event Explicit Types 13.57ms 13.94ms +2.7%
Excavate Single Thread Small 4.160s 4.113s -1.1%
Excavate Single Thread Large 9.751s 9.699s -0.5%
Excavate Parallel Tasks Small 4.388s 4.292s -2.2%
Excavate Parallel Tasks Large 7.487s 7.398s -1.2%
Is Ip Performance 3.22ms 3.22ms -0.2%
Make Ip Type Performance 11.49ms 11.66ms +1.5%
Mixed Ip Operations 4.54ms 4.60ms +1.3%
Typical Queue Shuffle 63.89µs 63.30µs -0.9%
Priority Queue Shuffle 711.79µs 701.63µs -1.4%

🎯 Performance Summary

No significant performance changes detected (all changes <10%)


🐍 Python Version 3.11.15

@codecov

codecov Bot commented Mar 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 60.00000% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 91%. Comparing base (3f7a2fa) to head (8d56a8b).
⚠️ Report is 5 commits behind head on 3.0.

Files with missing lines Patch % Lines
bbot/modules/gowitness.py 60% 4 Missing ⚠️
Additional details and impacted files
@@          Coverage Diff          @@
##             3.0   #2974   +/-   ##
=====================================
- Coverage     91%     91%   -0%     
=====================================
  Files        436     436           
  Lines      36333   36339    +6     
=====================================
- Hits       33057   33052    -5     
- Misses      3276    3287   +11     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@TheTechromancer TheTechromancer merged commit 360ec9a into 3.0 Mar 17, 2026
14 of 16 checks passed
aconite33 added a commit that referenced this pull request Mar 23, 2026
…2974. This prevents the crash and logs a warning instead of aborting the entire batch.
aconite33 added a commit that referenced this pull request Mar 23, 2026
The radixtarget 4.x migration introduced a Rust-backed PyRadixTarget
type that cannot be pickled. Since the web engine passes BBOTTarget
(which contains RadixTarget) to a subprocess via SpawnProcess, every
module that makes HTTP requests was failing with:
  "cannot pickle 'builtins.PyRadixTarget' object"

This affected telerik, reflected_parameters, azure_tenant, emailformat,
dnsbrute_mutations, and many others.

Fix: add __getstate__/__setstate__ to BaseTarget so the RadixTarget is
excluded from pickling and reconstructed from event_seeds on the other
side.

Additionally, fix gowitness handle_batch URL correlation:
- Normalize event_dict keys with clean_url() so they match the
  normalized DB URLs during lookup (fixes the root cause of KeyError
  crashes like the one partially addressed by PR #2974)
- Use .get() instead of bare dict access for the screenshot section,
  which PR #2974 missed (it only fixed network_logs and technologies)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
aconite33 added a commit that referenced this pull request Mar 23, 2026
The radixtarget 4.x migration introduced a Rust-backed PyRadixTarget
type that cannot be pickled. Since the web engine passes BBOTTarget
(which contains RadixTarget) to a subprocess via SpawnProcess, every
module that makes HTTP requests was failing with:
  "cannot pickle 'builtins.PyRadixTarget' object"

This affected telerik, reflected_parameters, azure_tenant, emailformat,
dnsbrute_mutations, and many others.

Fix: add __getstate__/__setstate__ to BaseTarget so the RadixTarget is
excluded from pickling and reconstructed from event_seeds on the other
side.

Additionally, fix gowitness handle_batch URL correlation:
- Normalize event_dict keys with clean_url() so they match the
  normalized DB URLs during lookup (fixes the root cause of KeyError
  crashes like the one partially addressed by PR #2974)
- Use .get() instead of bare dict access for the screenshot section,
  which PR #2974 missed (it only fixed network_logs and technologies)
@liquidsec liquidsec mentioned this pull request Jun 9, 2026
@ausmaster ausmaster deleted the fix-gowitness-bug branch June 11, 2026 01:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant