Allow dnsbrute to accept a list of wordlists

[SpamFaux]

Summary

• The wordlist option on dnsbrute now accepts either a single URL/path (existing behavior) or a list of URLs/paths
• Multiple wordlists are fetched, merged, and deduplicated into a single set before brute-forcing begins
• Fully backward-compatible — single-string configs are unchanged

Motivation

Users with custom subdomain wordlists previously had to choose between the default SecLists wordlist and their own. There was no way to supplement the default without replacing it entirely. With this change, a preset can specify both:

config:
  modules:
    dnsbrute:
      wordlist:
        - https://raw.githubusercontent.com/danielmiessler/SecLists/master/Discovery/DNS/subdomains-top1million-5000.txt
        - /path/to/custom-wordlist.txt

Test plan

• Existing TestDnsbrute and TestDnsbruteCanaryCheck tests pass (no regression)
• New TestDnsbruteMultiWordlist test verifies merged wordlist size, correct deduplication of overlapping entries, and end-to-end discovery of a subdomain sourced from the second wordlist

[github-actions]

🚀 Performance Benchmark Report

⚠️ No current benchmark data available

This might be because:

• Benchmarks failed to run
• No benchmark tests found
• Dependencies missing

[liquidsec]

@SpamFaux good feature, I like it. My main feedback is just that if we're going to have it , it shouldn't be confined to one module, since we have other modules that define wordlists in configs. The functionality should be generalized as a helper, probably an extension of the wordlist() helper, so things like webbrute, legba, etc can all utilize it.

[SpamFaux]

Good notes. I made the recommended changes. Moved it up to be more general. Let me know if you have other feedback.

[liquidsec]

@SpamFaux

Some feedback:

Functionally, I think this will work, but there is some redundancy here:

new merge branch (lines 550-560)

  if lines is None:
      return merged_filename
  lines = int(lines)
  with open(merged_filename) as f:
      read_lines = f.readlines()
  trunc_key = f"{merged_filename}:{lines}"
  truncated_filename = self.parent_helper.cache_filename(trunc_key)
  with open(truncated_filename, "w") as f:
      for line in read_lines[:lines]:
          f.write(line)
  return truncated_filename

existing single-path branch (lines 588-599)

  if lines is None:
      return filename
  else:
      lines = int(lines)
      with open(filename) as f:
          read_lines = f.readlines()
      cache_key = f"{filename}:{lines}"
      truncated_filename = self.parent_helper.cache_filename(cache_key)
      with open(truncated_filename, "w") as f:
          for line in read_lines[:lines]:
              f.write(line)
      return truncated_filename

These are nearly identical with the only difference being trunc_key vs cache_key. We should be able to merge these and remove the redundant code and clean it up a lot.

The other problem is that you are putting these into a set(). Set() is non-deterministic, which means the order will essentially be random each time. That breaks the "lines=n" option, in terms of producing a deterministic predictable truncation of lines. The fix is to use dict.fromkeys(), deduplicates while preserving insertion order.

and, apparently you need to lint again

[liquidsec]

@SpamFaux I added options documentation for other modules which can now take advantage as well. Looks good to me though.

[liquidsec]

@ausmaster, you need to approve since we both edited

[codecov]

Codecov Report

❌ Patch coverage is 88.33333% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 90%. Comparing base (361c0cc) to head (402cff1).
⚠️ Report is 20 commits behind head on dev.

Files with missing lines	Patch %	Lines
bbot/core/helpers/web/web.py	88%	5 Missing ⚠️
...t/test_step_2/module_tests/test_module_dnsbrute.py	90%	2 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff          @@
##             dev   #3094   +/-   ##
=====================================
- Coverage     90%     90%   -0%     
=====================================
  Files        445     445           
  Lines      38792   38823   +31     
=====================================
+ Hits       34715   34736   +21     
- Misses      4077    4087   +10     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.