Don't recursively mutate mutations (by default)

[liquidsec]

Background

Long-running BBOT scans frequently exhibit a "cliff + tail" throughput pattern: high steady-state throughput for several hours, a sharp drop (often >99% in a single heartbeat), then a long slow drain that can last 10+ hours.

Tracing a representative scan, throughput dropped from ~6,000 events/15s to <100 events/15s in a single tick. At the cliff, four modules carried the remaining queue depth, all with processing=1 (single-threaded handlers):

Module	Queued at cliff	Drain rate	Per-event cost
dnscommonsrv	34,897	~3.6/min	massdns SRV brute per host
dnsbrute	33,930	~7.5/min	massdns 5,000-word brute per host
wayback	25,050	~17/min	CDX API call per host
pgp	14,686	~52/min	keyserver lookup per host

At those rates, dnscommonsrv alone projected to ~158 hours to fully drain. The other modules' arithmetic was similar.

Root cause

dnsbrute_mutations runs in finish() and can emit tens of thousands of new in-scope subdomains. Every one of those then enters dnsbrute / dnscommonsrv (both dedup_strategy="lowest_parent", so every unique subdomain triggers an independent massdns run). The 4-module set above is now doing per-host bruteforce against the entire mutation output.

The pathological part: this work is mostly redundant. The SecLists subdomains-top1million-5000.txt wordlist is dominated by ~50 universally common labels (www, mail, admin, api, dev, staging, vpn, ftp, ns1, cpanel, autodiscover, etc.), which are exactly the labels dnsbrute_mutations derives from its own corpus. The module's mutation Source 1 explicitly does:

for _domain, _subdomains in found:
    for s in _subdomains:
        first_segment = s.split(".")[0]
        add_mutation(first_segment)

So the first segment of every discovered subdomain anywhere in the scan becomes a mutation candidate. By the time mutations runs, the wordcloud already contains nearly the entire useful prefix of the SecLists list. Re-running that static list against each mutation output is ~150M massdns queries for, generously, single-digit-percent unique discoveries.

What this PR does

Adds a recursive_mutations option to dnsbrute and dnscommonsrv (default False). When False, events carrying any mutation-* tag (emitted by dnsbrute_mutations) are rejected by the module's filter_event. The events still queue but drain at thousands/sec because no massdns work runs; the expensive per-host brute is skipped.

The filter handles arbitrary mutation depth (mutation-1, mutation-2, mutation-3, ...) because it matches the tag prefix. dnsbrute_mutations is untouched - it can still iterate internally on its own output. We're cutting the amplification loop where each mutation tier triggers a full SecLists brute against every host.

Defaults

• Both modules default to recursive_mutations: false (skip the redundant work).
• kitchen-sink preset overrides to true for users explicitly opting into maximum coverage.

Tradeoff

The skipped work occasionally catches names that mutations can't generate - the tail of the SecLists wordlist contains software-specific labels (roundcube, phpmyadmin, webmin, etc.) that only end up in the wordcloud if some target host already runs that software. Coverage loss is small, and the option exists for users who want it back.

[github-actions]

🚀 Performance Benchmark Report

⚠️ No current benchmark data available

This might be because:

• Benchmarks failed to run
• No benchmark tests found
• Dependencies missing

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90%. Comparing base (feacaba) to head (eb12191).
⚠️ Report is 21 commits behind head on dev.

Additional details and impacted files

@@          Coverage Diff          @@
##             dev   #3131   +/-   ##
=====================================
+ Coverage     90%     90%   +1%     
=====================================
  Files        441     441           
  Lines      38743   38788   +45     
=====================================
+ Hits       34663   34707   +44     
- Misses      4080    4081    +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ausmaster]

Approved with one optional:
The if not self.config... return False, ... could be pulled out into a separate function for DRY but its only called twice and its not really substantial. Up to you if you wanna do that, not required for merge.