make_ip_type(): Invalid hostname

[liquidsec]

lightfuzz branch, to reproduce:

poetry run bbot -m httpx hunt -t https://hv-impact.edc.org/ hunt is to enable parameter extraction

2024-12-19 05:21:27,194 [ERROR] bbot.scanner scanner.py:1319 Error in excavate.handle_event((HTTP_RESPONSE("{'url': 'https://hv-impact.edc.org/admin/', 'timestamp': '2024-12-19T05:21:26.73...", module=httpx, tags={'in-scope', 'ip-34-198-65-201', 'dir', 'cloud-ip', 'http-title-access-denied-miechv-tarc', 'cloud-amazon', 'spider-danger', 'status-403'}), {})): /home/redacted/bbot/bbot/core/helpers/misc.py:685:make_ip_type(): Invalid hostname: "None"
2024-12-19 05:21:27,194 [TRACE] bbot.scanner logger.py:162 Traceback (most recent call last):
  File "/home/redacted/bbot/bbot/scanner/scanner.py", line 1296, in _acatch
    yield
  File "/home/redacted/bbot/bbot/modules/base.py", line 1627, in _worker
    forward_event = await self.handle_event(event, **kwargs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/redacted/bbot/bbot/modules/internal/excavate.py", line 1216, in handle_event
    await self.search(
  File "/home/redacted/bbot/bbot/modules/internal/excavate.py", line 1061, in search
    await self.yara_preprocess_dict[rule_name](result, event, discovery_context)
  File "/home/redacted/bbot/bbot/modules/internal/excavate.py", line 158, in preprocess
    await self.process(yara_results, event, yara_rule_settings, discovery_context)
  File "/home/redacted/bbot/bbot/modules/internal/excavate.py", line 626, in process
    await self.report(
  File "/home/redacted/bbot/bbot/modules/internal/excavate.py", line 258, in report
    event_draft = await self.report_prep(event_data, event_type, event, tags, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/redacted/bbot/bbot/modules/internal/excavate.py", line 212, in report_prep
    event_draft = self.excavate.make_event(event_data, event_type, parent=event)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/redacted/bbot/bbot/modules/base.py", line 487, in make_event
    event = self.scan.make_event(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/redacted/bbot/bbot/scanner/scanner.py", line 987, in make_event
    event = make_event(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/redacted/bbot/bbot/core/event/base.py", line 1774, in make_event
    return event_class(
           ^^^^^^^^^^^^
  File "/home/redacted/bbot/bbot/core/event/base.py", line 224, in __init__
    self.parent = parent
    ^^^^^^^^^^^
  File "/home/redacted/bbot/bbot/core/event/base.py", line 569, in parent
    hosts_are_same = (self.host and parent.host) and (self.host == parent.host)
                      ^^^^^^^^^
  File "/home/redacted/bbot/bbot/core/event/base.py", line 327, in host
    self.host = self._host()
                ^^^^^^^^^^^^
  File "/home/redacted/bbot/bbot/core/event/base.py", line 1019, in _host
    return make_ip_type(self.data["host"])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/redacted/bbot/bbot/core/helpers/misc.py", line 685, in make_ip_type
    raise ValueError(f'Invalid hostname: "{s}"')
ValueError: Invalid hostname: "None"

[TheTechromancer]

This is coming from the ParameterExtractor, specifically _GenericForm, which is trying to create a WEB_PARAMETER with a host of None.

EDIT: Mmm on second thought I don't think we know which extractor class is causing this. But the host is coming directly from the parsed_url; so it seems like an invalid URL is somehow sneaking in there.

[TheTechromancer]

Dev also produces some errors which are a direct result of discrepancies between our URL extraction and validation:

The first step to fixing these bugs will be to fix those discrepancies (#1860).

[liquidsec]

I've seen the mailto one come up before. I'll see if i can run that down.

[liquidsec]

Actually, I probably already fixed the mailto issue in the process of making the lightfuzz branch. There are, admittedly, a few bug fixes that ended up in the lightfuzz branch when they probably should have just gone to dev.

[liquidsec]

Its a regex problem, i've got it isolated to the specific one now

[liquidsec]

Ok, #2104 should take care of it.

Its a bit crude, and only addresses mailto links, but that has easily been 90% of this type of problem.

I also made a change on lightfuzz side which will make identifying these type of issues much easier and preventing an unhandled error:

8808fd3

[liquidsec]

fixed in #2104