Extract IPs and CIDRs from SPF TXT records

[thunderstornX]

Summary

Adds a passive dnsspf module that parses SPF (Sender Policy Framework) DNS TXT records and emits the infrastructure they disclose. Closes #2545.

SPF records frequently expose target-owned infrastructure — individual IPs, whole subnets, and related domains — which is useful for attack-surface mapping. The subnets in particular (the original ask in #2545) are often owned by the target.

Implementation

The issue suggested doing this in excavate, but excavate only watches HTTP_RESPONSE/RAW_TEXT and never sees DNS records. bbot already has a clear pattern of per-record DNS-TXT parser modules — dnscaa, dnstlsrpt, dnsbimi — so I implemented this as a dedicated dnsspf module following that same pattern. Happy to fold it into excavate instead if you'd prefer.

The module watches DNS_NAME, resolves the host's SPF record, and emits:

• IP_NETWORK for ip4:/ip6: CIDR subnets
• IP_ADDRESS for individual ip4:/ip6: hosts
• DNS_NAME for domains referenced via include:/a:/mx:/redirect=/exists:
• RAW_DNS_RECORD (optional, off by default)

Each output type is individually toggleable via module options. SPF macros (%{...}) are skipped, and malformed addresses are ignored.

Testing

• New bbot/test/test_step_2/module_tests/test_module_dnsspf.py — covers a single IPv4, an IPv4 subnet, an IPv6 subnet, and the include:/a: domains. Passing.
• ruff check / ruff format --check clean.

[github-actions]

Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.

---

I have read the CLA Document and I hereby sign the CLA

---

1 out of 2 committers have signed the CLA.
✅ (thunderstornX)[https://github.com/thunderstornX]
❌ @liquidsec
_{You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot.}

[thunderstornX]

I have read the CLA Document and I hereby sign the CLA

[thunderstornX]

recheck

[liquidsec]

hey @thunderstornX, thanks for the PR.

There are some issues with your module as-written now we'd need to address though:

Redundant Resolutions

The biggest thing is you are doing redundant DNS resolution. BBOT will have already resolved every single DNS_NAME including all the rdtypes (by default), to include txt. So the:

response = await self.helpers.dns.resolve_full(hostname, rdtype)

line is doing completely redundant work.

We have some modules that re-resolve, but they are all doing that to some modified version of the domain, not the same domain again.

You should be able to read TXT data directly from `event.raw_dns_records["TXT"]

Feature (mostly) already implemented

DNS_NAMEs are already being extracted from txt records. this happens in extract_targets() in dns/helpers.py.

It is not getting IP addresses or CIDRs, though - which is what that issue you referenced was talking about. I am not sure of the utility of those, we can certain emit them as IPs, but if they aren't in scope, nothing will happen to them anyway. I think the value is probably just the added context they provide, more as an end product than something to feed back into the scan.

There are a few other minor things, but given the other ones I pointed out, they will probably not remain relevant.

I think ultimately this probably needs to be a much smaller change, modifying extract_targets so it also uses our pre-existing cidr/ip regexes, etc.

[thunderstornX]

Thanks for the thorough review @liquidsec — all fair points. Reworked per your suggestion:

• Dropped the standalone dnsspf module (and the redundant re-resolution with it).
• Extended extract_targets() in dns/helpers.py to also pull IPs and CIDRs out of TXT using bbot's existing ip_range_regexes/ipv4_regex/ipv6_regex — operating on the already-resolved TXT, no extra resolution.
• It tokenises and strips the ip4:/ip6: prefix before matching (otherwise ip6: pollutes the IPv6 match → 6:2001:db8::/48), and a CIDR's bare network address isn't emitted alongside it.
• Added a test_dns.py case for an ip4: IP, ip4:/ip6: CIDRs, and include:.

Happy to gate it to v=spf1 records specifically rather than all TXT if you'd prefer.

[liquidsec]

Alright, theres some more things to go over:

Don't use stdlib re, use import regex as re

BBOT uses the third-party regex package everywhere, not stdlib re. It's a drop-in replacement but releases the GIL during matching, which matters for BBOT's async/threaded architecture. All existing code (regexes.py, misc.py, regex.py) uses import regex as re. Any new regex patterns should either go in regexes.py and be imported, or use regex if compiled locally.

CIDRs are silently dropped currently

extract_targets would now return CIDRs, but its only consumer, which turns it into an event hardcodes to DNS_NAME as event_type. This will mean it will fail validation and just do nothing. Instead, we should be abel to set None as the event type, and let the event auto detection handle assign it a type.

Gate on v=spf1

We should do a cheap check for v=spf1 before we do any regexes.

bbot/core/helpers/regexes.py

we generally centralize our regexes here. If theres even a remote chance it will be used again, put it there and import it over

Tests

Lets make a test that validates all the way from coming in via a txt record to being emitted as a an IP_ADDRESS or IP_RANGE. Such a test would have caught the second issue i mentioned, i think.

Also a test to verify the macros are skipped

[thunderstornX]

Addressed all of it:

• The SPF mechanism regex now lives in regexes.py, compiled with regex like the rest. No stdlib re left in the change.
• You were right about the CIDRs: an IP with the hardcoded DNS_NAME type was getting coerced, but a CIDR failed validation and was silently swallowed in emit_dns_children. The child event type is now None so auto-detection assigns it, and IP_RANGE was added to dnsresolve's produced_events.
• IP/CIDR extraction is gated on a cheap v=spf1 check before any regex work. Non-SPF TXT keeps the original hostname-only extraction.
• SPF macro tokens are skipped.
• Added an end-to-end test from a mocked SPF TXT record to emitted IP_ADDRESS/IP_RANGE events (run with report_distance: 1 so the distance-1 children show up in output), plus macro and gating assertions in the unit test.

Unrelated heads-up from running the suite: test_events::test_events fails on a clean dev checkout on non-UTC machines, the timestamp asserts are off by exactly the local UTC offset (mine is +5). Happy to open an issue for that separately if useful.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91%. Comparing base (dfd4a38) to head (aaae002).
⚠️ Report is 7 commits behind head on dev.

Additional details and impacted files

@@          Coverage Diff           @@
##             dev   #3144    +/-   ##
======================================
+ Coverage     90%     91%    +1%     
======================================
  Files        448     448            
  Lines      42614   42732   +118     
======================================
+ Hits       38340   38478   +138     
+ Misses      4274    4254    -20     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[thunderstornX]

CodeQL's three alerts were false positives on exact set-membership asserts in the new tests; switched those to subset comparisons. The benchmark job's failure is only the comment-on-PR step (fork token), and ubuntu:24.04 died on Docker Hub registry timeouts before running anything.

[liquidsec]

looking good! little swamped at the moment but will give this a final check and hopefully merge soon

[liquidsec]

Tested against some records i threw up onto one of my domains. I think everything is working perfectly. We can merge pending @ausmaster final check.

One thing to keep in mind: with the default report distance of 0 - these usually wont be visible at all, unless they happen to land in scope. They will still be going to modules regardless. This is fine and expected, just wanted to throw that out there. There are definitely some edge cases where they will feed discovery of new in scope assets, particularly when the scope is a giant ip range.

[liquidsec]

Hey, we discovered a pre-existing potential issue that this was kindof adjacent to, so we went ahead and tacked the fix onto this PR to make sure all tests pass together. Will be merging once tests pass.

[ausmaster]

A great PR, thanks very much @thunderstornX for your contribution!

[thunderstornX]

Hey @liquidsec, thank you for this, and for the patience across both rounds. The depth of the review genuinely helped: the redundant resolution, the CIDR event type that was getting silently dropped, and centralizing the regex all made the change tighter and taught me a lot about how BBOT fits together.

Good point on the default report distance of 0. The IPs and ranges still reach the modules even when they aren't surfaced, and the giant-IP-range scope is a neat case where they could pull in new in-scope assets. And no problem at all tacking the adjacent fix onto the PR, glad if it helps everything land together.

Really appreciate it, this has been a great first experience contributing here.

[thunderstornX]

Thank you @ausmaster, that means a lot. Really glad it came together well.

[liquidsec]

Hey @liquidsec, thank you for this, and for the patience across both rounds. The depth of the review genuinely helped: the redundant resolution, the CIDR event type that was getting silently dropped, and centralizing the regex all made the change tighter and taught me a lot about how BBOT fits together.

Good point on the default report distance of 0. The IPs and ranges still reach the modules even when they aren't surfaced, and the giant-IP-range scope is a neat case where they could pull in new in-scope assets. And no problem at all tacking the adjacent fix onto the PR, glad if it helps everything land together.

Really appreciate it, this has been a great first experience contributing here.

thanks for sticking with it! Hope we see more from you.

[liquidsec]

recheck

[liquidsec]

recheck

[aconite33]

recheck