Lightfuzz Overhaul (False-positive reductions, False-negative reductions, many new techniques)#2967
Merged
Merged
Conversation
Contributor
🚀 Performance Benchmark Report
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## dev #2967 +/- ##
=======================================
+ Coverage 90% 91% +1%
=======================================
Files 447 448 +1
Lines 40462 42504 +2042
=======================================
+ Hits 36320 38256 +1936
- Misses 4142 4248 +106 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
liquidsec
changed the title
…dule - detect_ecb(): passive repeated-block analysis, zero HTTP requests - cbc_bitflip(): active test mutating penultimate block positions, 2 HTTP requests - Fix modify_string() treating position=0 as falsy
… CSRF tokens, remove generic error string - Add get_waf_strings() helper to misc.py with 7 common WAF signatures - Use centralized WAF list in path.py and serial.py instead of hardcoded strings - Blacklist CSRF tokens and ASP.NET session cookies in defaults.yml - Remove overly generic "access denied" from crypto error strings
…erification, SQLi WAF detection - Add endpoint stability pre-check to padding oracle and CBC bitflip tests - Verify XSS probe matches appear in the correct HTML context - Suppress SQLi findings when single-quote probe triggers WAF 403 - Blacklist PKCE and Akamai Bot Manager parameters
Prevents false positive deserialization findings against endpoints like GlobalProtect that use non-standard status codes (e.g. 512).
URL events are now DictHostEvent so e.data is a dict, not a string.
liquidsec
force-pushed
the
lightfuzz-improvements-mar-26
branch
from
3c54eca to
ab42ab3
Compare
…rmation loop, better finding descriptions - Hoist url_extension from URL_UNVERIFIED to DictEvent so WEB_PARAMETER events on static/special URLs get filtered - Extend accept_url_special gate to all events with URLs, not just URL types - Add lightfuzz filter_event rejection of WEB_PARAMETERs on url_extension_static URLs - Add sqli code-change confirmation loop (3 rounds with fresh baselines) to suppress transient flaps - Surface additional_params and probe request bodies in sqli finding descriptions - Fix baseline_probe() "eventtype" -> "type" typo, fix falsy check in incoming_probe_value() - Update paramminer_headers to use url_extension attribute
sqli: fix missing-comma bug that concatenated Oracle+MSSQL probes into one malformed string. Add one-shot row-independent SLEEP payloads (`' OR SLEEP(N) IS NOT NULL LIMIT 1-- -`) so blind sqli against non-matching rows still delays predictably. path: add simple `./X` / `../X` probes for stacks that walk paths via the OS (Python open(), Go os.Open, Rust File::open) rather than string-normalizing first. The existing `./a/../X` variants require the intermediate `a/` to exist on disk; the new variants don't. cmdi: three-stage cascade per delimiter — generic echo canary, then POSIX arith `echo \$((A*B))`, then Windows `set /A A*B`. When an arith probe confirms execution, upgrade to HIGH confidence. When only the generic probe fires (no shell confirmation), emit a separate "Possible Parameter Reflection" finding rather than overclaiming cmdi. Preserves the adjacent-vuln reflection signal without mislabeling it. tests: add save-point tests for each new detection path, and bring the rand_string mocks into compliance with numeric_only=True so the new arith code doesn't crash on mocked values.
Previously `determine_context` recognized only three contexts: between
tags, double-quoted attribute, and plain JS string. Real-world XSS
reflections into single-quoted attributes, non-`action` URL attributes
(href, src, formaction, etc.), HTML comments, and JS template literals
were all missed.
Added:
- Single-quoted attribute detection + quote-aware breakout probes.
Tracking the wrapping quote type prevents false positives from the
wrong-quote char reflecting harmlessly inside.
- URL-scheme probe generalized from `action="javascript:..."` to match
any URL-bearing attribute.
- HTML comment context: detected via `<!-- ... {refl} ... -->` and
probed by testing whether `-->` survives reflection.
- JS template literal (backtick) context: detected via backtick-wrapped
reflection inside <script>, probed via `${...}` interpolation survival.
Also fixed an early `return` in the existing in-javascript path that
aborted fuzz() entirely when the quote context was "outside" — now it
just skips the escape-the-escape probe and falls through to the new
template-literal check.
Tests: 4 new save-point classes covering single-quote, URL-scheme-href,
HTML comment, and template literal. Existing xss test assertions updated
for the new `Tag Attribute ({q} quoted)` / `URL-scheme Injection` context
format.
Two changes to the ssti submodule:
1. Add a direct `{{1337*1337}}` probe and a single-brace Smarty variant
`{1337*1337}`. The existing `1,787{{z}},569` trick relies on
`{{z}}` silently rendering as empty, which fails in StrictUndefined
Jinja2 environments and other engines that raise on undefined vars.
A direct arithmetic probe still works in those contexts.
2. Add a baseline-response check: before running any probes, fetch
the response with the original value and confirm the detection
canaries (1787569 / 1,787,569) aren't already present. If they
are, abort — the number is part of the page, not a template-eval
result. This eliminates the rare coincidental-number false positive.
Tests: 2 new save-points. One exercises the direct {{A*B}} path with
a mock template engine that raises on {{z}} (simulating StrictUndefined).
The other confirms the baseline check suppresses findings on pages
that already contain 1,787,569 in static content.
Add Apache Velocity `#set($x=A*B)$x` probe (URL-encoded since `#` would otherwise be interpreted as a URL fragment and truncate the payload). Save-point test with a mock that only recognizes the Velocity syntax, proving the new probe fires on engines the existing probes miss.
Two changes:
1. Python pickle coverage for the serial submodule.
- Static payloads: benign `pickle.dumps('test')` in both base64 and
hex dicts, plus `unpicklingerror` added to the error-string list
for the Differential Error Analysis path.
- Blind RCE: generates a fresh pickle payload per scan whose
`__reduce__` calls `socket.gethostbyname(interactsh_subdomain)`.
Because pickle is Python-native, the payload can be rebuilt per
scan with a unique OOB callback — no pre-generated template or
out-of-process gadget tooling required.
- `uses_interactsh = True` so lightfuzz spins up interactsh when
serial is the only enabled submodule.
2. `skip_envelopes` opt-in on BaseLightfuzz.
- When a submodule's payloads ARE the wire format (base64-encoded
serialized objects, hex blobs, etc.), the envelope system must
not re-pack them based on what the parameter's original value
looked like. Without this, a base64-wrapped plain-text parameter
would cause serial's already-base64 pickle payload to be
double-encoded before transit.
- Class-level flag: submodules set `skip_envelopes = True` and the
base-class helpers `incoming_probe_value` / `outgoing_probe_value`
skip unwrap/pack accordingly. Default stays False for every other
submodule.
Tests: 2 new serial save-points (pickle Error Resolution + pickle OOB
interactsh). The OOB test mocks interactsh, decodes the incoming
payload, extracts the embedded subdomain, and triggers a mock
interaction to confirm the detection path fires. Full lightfuzz suite
75/75 (was 73).
baseline_probe was firing a bare request with no body — for POST baselines the server received an empty body and rendered a default/error view, not the actual form submission. The resulting HTTP_RESPONSE was useless for excavate to re-mine, defeating the recon→DAST loop on real targets. Now uses prepare_request() (the same builder compare_baseline uses) so the baseline request carries the parameter's original_value plus every sibling field from additional_params. GET baselines get them in the querystring; POST/BODYJSON in the body; HEADER/COOKIE on the wire. Regression test: Test_Lightfuzz_baseline_probe_form_submission stands up a server that only reveals /crypto-baseline-secret when the POST body includes the form's token field. Verified the test catches the prior bug (reverted the fix in-stash, the assertion fails on the missing URL_UNVERIFIED for /crypto-baseline-secret). Also fixes Test_Lightfuzz_crypto_error fixture: the server returned the crypto error for any non-empty secret value, including the canonical one. With baseline_probe correctly sending the canonical value, the baseline itself contained the error and error_string_search had nothing to diff against. The fixture now mirrors a real crypto-vulnerable backend (canonical value decrypts cleanly; any other value errors).
…y slice YARA's regex engine has a hardcoded ceiling (~4 KB) on `.*` quantifiers between anchored points, so the form-discovery rules of shape `<form ...>.*</form>/s` silently failed to match any form whose body exceeded that bound — a single <select> over a real-world dataset (hundreds of options) is enough. Downstream consumers (lightfuzz, crypto's cross-value detectors) never saw WEB_PARAMETER events for those forms. Fix: - Narrow each form-rule discovery_regex to match only the opening <form ...> tag (GetForm, GetForm2, PostForm, PostForm2, PostForm_NoAction, _GenericForm). - Override ParameterExtractor.preprocess to retain per-instance YARA match offsets (the base class flattens to a set, losing them). - ParameterExtractorRule.__init__ now accepts the response body and the match offset; form_body_slice() returns a bounded slice of the body starting at the YARA offset. - The existing Python extraction_regex (non-greedy `<form...>...</form>` with no cliff) runs on that slice and finds the form normally. - New excavate config: max_form_bytes (default 256 KB) caps worst-case per-form extractor work. Forms whose body exceeds the cap are skipped — defensive cap, not best-effort partial extraction. Tests: - TestExcavateGiantForm: 5000-option <select> (~210 KB form), asserts all fields are extracted. Verified to fail (no fields emitted) before this fix. - TestExcavateGiantFormExceedsMaxBytes: ~240 KB form with max_form_bytes=32 KB, asserts no hang/OOM and no form fields leak through under the bound. - TestExcavateFormAttributeOrder: covers both `<form action=X method=post>` and `<form method=post action=X>`.
…ection New Stage 0 in the crypto section covering the cross-value pairwise-XOR check, its severity/confidence ladder, and why it bypasses the entropy gate.
…ization Excavate stores original_value=None for <input> elements without a value= attribute (distinct from value="" in event metadata). Browsers submit both cases identically as name= (empty value), but Python's urlencode renders None as the literal text 'None' — the target server then sees name=None and typically rejects the request as malformed. Coerce None → '' at the prepare_request wire boundary so the request matches browser-equivalent semantics. Affects the populate_empty=False path (baseline_probe and any explicit caller); the populate_empty=True path already replaces None with random strings in additional_params_process. Regression test Test_Lightfuzz_none_in_additional_params: server reveals the secret only when the no-value-attr field arrives as '' (empty) rather than the literal 'None'. Verified the test fails on the pre-fix codebase and passes after.
…'a') baseline_probe now fires up to two requests per WEB_PARAMETER: * Probe A — form as captured (existing behavior). Filter-style forms whose 'show all results' state is the no-narrowing default produce content here. * Probe B — substitute 'a' for the fuzzed field's value, only when its original_value is None or ''. Catches search-style forms whose useful content appears only when the primary text input is non-empty. Both responses go through emit_baseline_response so excavate can mine whichever returned populated content. No classifier picks between them — they both fire and the response is the arbiter. The baseline submission is hoisted to lightfuzz.handle_event so it runs once per WEB_PARAMETER regardless of which submodules are enabled (it's a module-level recon→DAST loop feature, not a crypto-specific one). Crypto still aborts early on empty probe_value as before — that's the correct behavior for crypto's per-value analyses, separate concern. Per-signature LRU cache (_baseline_probe_response_cache, bounded at 200 entries) ensures identical baselines across siblings of the same form collapse to one network request and one emission. Tests cover the two functional cases: * Search-style form with empty input → Probe B reveals secret link. * <select> with a selected option → Probe B skipped (real default preserved), no extra POST.
…kill PostForm_NoAction phantom - excavate stamps host_url on form-derived WEB_PARAMETERs when the form's action URL differs from the discovery page; lightfuzz primes its connectivity GET against host_url and stamps Referer on probe requests - lightfuzz coalesces concurrent connectivity_test GETs per prime_url via NamedLock so the 4-5 WEB_PARAMETERs excavate emits per page share one GET instead of racing parallel ones under _module_threads=4 - lightfuzz dedup hash includes body_md5 for non-FINDING events so dual baseline_probe (Probe A vs Probe B) surfaces distinct bodies to excavate - regexes: tighten post_form_regex_noaction with a negative-lookahead so it stops over-matching forms with action — was emitting a phantom WEB_PARAMETER whose url fell back to event.url - test: rewrite cookie_refresh server to bootstrap-then-validate (matches real CSRF protections); add host_url priming end-to-end test
…t-PostForm_NoAction-fix counts
…-mar-26 # Conflicts: # bbot/modules/lightfuzz/submodules/serial.py
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Both build_query_string and send_probe_with_canary appended ?name=value to event.url with an f-string, producing URLs with multiple ? segments when the source WEB_PARAMETER already carried a querystring (common from wayback when url_querystring_remove=False). Route through add_get_params so the probe merges into the existing querystring instead.
…cess ParameterExtractor.process read response_body from event.data['body'], which body-spill pops at HTTP_RESPONSE construction. Form-field extraction silently emitted no WEB_PARAMETERs whenever spill was on (default). Switch to event.body so the spill store is consulted.
sqli: two-stage scaling-delay confirmation for time-based blind detection
…ity cache poisoning - crypto: filter structured hex IDs (Mongo ObjectIds, hex timestamps, sequential counters) from keystream-reuse detector via tail analysis - cmdi: reject leading-zero arithmetic multiplicands (bash octal divergence) - lightfuzz: restore event type/flags after try_get_as_post conversion passes - lightfuzz: don't cache None connectivity responses (transient failure) - Add 16 regression tests covering all four fixes
ausmaster
approved these changes
Jun 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Systematic reduction of both false positives and false negatives in lightfuzz, plus new detection techniques across multiple submodules. FN work driven by a structured "false-negative hunt" — auditing each submodule, building gap-filler apps and running against DVWA + PortSwigger web security labs to expose missed vulnerabilities, then fixing.
Also frames lightfuzz as BBOT's DAST module and closes the recon→DAST loop by feeding canonical baseline responses back into excavate so pages that only render after a real submission get mined for new params/URLs.
Form Extraction & DAST Loop
Excavate parameter extraction
<a href="?SortBy=<hex>">links collapses to a single event becauseurl_unparsestrips the query string for GETPARAMs). The parameter extractor now groups yields by(type, name, url)and emits one WEB_PARAMETER per group with a newsame_param_valuesfield carrying the other observed values — surviving dedup so cross-value detectors can see them.<select>picker: prefer the option carryingselected, else the first option's value (blank or not). Filter-style forms often use a blank default that matches all rows; substituting a specific choice could narrow results to nothing. Emptyoriginal_valueis no longer coerced toNoneby the form-extractor's falsy check.Lightfuzz → excavate baseline emission
compare_baseline()call as anHTTP_RESPONSEevent, so excavate re-mines pages only rendered after a real submission (confirmation pages, redirect targets, forms behind a submit).HttpComparecache keyed on the full prepared request signature: identical baselines across submodules (sqli + cmdi + path on a POSTPARAM with no envelopes were firing 3× the same baseline pair) collapse to one HttpCompare, one network pair, one emission. Cache cleared per-event athandle_eventexit.HttpCompare.on_baseline_readyasync callback fires once afterbaseline_1is established.emit_http_responsekwarg oncompare_baseline()/baseline_probe()(defaultTrue); set False at sqli's FP-killer confirmation rounds.bbot/core/helpers/web/response_event.pybuilds HTTP_RESPONSE dicts from blasthttp 0.5.xResponseobjects — used by bothhttp.py(DRY refactor) and lightfuzz.emit_baseline_responses(defaulttrue).from-lightfuzzprovenance tagfrom-lightfuzztag at emit time, distinguishing baseline-emission events from primary recon HTTP_RESPONSEs.event/base.py(parallel toaffiliate/from-wayback), so URL_UNVERIFIEDs and WEB_PARAMETERs excavate emits off a lightfuzz baseline inherit the provenance.assigned_cookiesrefreshhandle_eventjust hit the server; its Set-Cookie response is the freshest state we can get without an extra request. Fresh values are merged intoevent.data["assigned_cookies"]so subsequent baseline POSTs go out with the current session. Fresh wins on conflict; cookies the GET didn't re-issue (e.g. an auth cookie originally seen elsewhere) are preserved.Keystream-Reuse (Many-Time-Pad) Detection — new
Many homebrew/legacy stream ciphers (e.g. ColdFusion
CFMX_COMPAT) reuse one keystream across every encryption; XOR-ing any two such ciphertexts yields the XOR of their plaintexts. For natural-language / identifier plaintexts sharing a prefix this shows up as a leading run of zero bytes — mathematically impossible under correct encryption.Per-value detectors (
detect_ecb, padding-oracle, etc.) can't see this — it's only visible across multiple ciphertexts. Newdetect_keystream_reuse()incrypto.py:event.original_value+additional_params+same_param_valuesCatches the canonical real-world fixture (
4E4CDA8A93F87AXOR4E4CDA8A93FF7B85…= 5-byte zero run) at HIGH/CONFIRMED.Docs updated to frame lightfuzz as BBOT's DAST module.
False Negative Reductions & New Techniques
Serial (
serial.py) — newunpicklingerrorfingerprinting__reduce__chains tosocket.gethostbynameagainst a fresh interactsh subdomain. No external tooling; payload is built at scan time.java.util.HashMap+java.net.URL(in every JVM since 1.1). Validated end-to-end against a real PortSwigger Apache Commons deserialization lab via oastify Collaborator callback.skip_envelopesclass flag — submodules whose payloads ARE the wire format (serial) opt out of envelope packing, so base64/hex/PHP-raw probes aren't re-encoded by the envelope system.ESI (
esi.py) — new<esi:include src="http://{interactsh}/"/>alongside the original tag-strip probe. Tag-strip proves ESI comment processing; remote-fetch proves exploitable processing with side-effects. CRITICAL / CONFIRMED.Path traversal (
path.py)a/intermediate) variants for strict path resolvers that reject paths whose intermediate components don't exist:....//X,/....//X..%252f{X},%252f..%252f{X}CMDi (
cmdi.py)echo $((A*B))(POSIX) thenset /A A*B(cmd.exe). The productA*Bnever appears in the probe's literal bytes, so a response containing it proves real shell arithmetic expansion.XSS (
xss.py)<!-- ... {x} ... -->with a-->zbreakout probe.`...${x}...`with a${...}interpolation probe."vs'as the wrapping quote rather than assuming double-quotes, so single-quoted-attribute XSS is no longer missed. Per-quote Tag Attribute, autoquote, and URL-scheme (javascript:) injection probes.SSTI (
ssti.py){{1337*1337}}, Smarty{1337*1337}, Apache Velocity#set($x=A*B)$x(URL-encoded).Crypto (
crypto.py)flipbranch frommodify_string.Core (
lightfuzz.py,submodules/base.py)interactsh_callbackwhen a subdomain-tag details entry is partially populated.skip_envelopesclass-level flag onBaseLightfuzz.register_interactsh_tag()helper onBaseLightfuzz— consolidates the tag → FQDN → subdomain_tags dict entry boilerplate previously duplicated across 5 call sites (ssrf, cmdi, esi, serial × 2).False Positive Reductions — Structural
Event system (
base.py)url_extensionfrom URL_UNVERIFIED up to DictEvent so WEB_PARAMETER events on.css,.js,.pdfURLs carry the extension attribute and getextension-*tags.Module filtering (
modules/base.py)accept_url_specialgate to apply to any event with a special URL extension (e.g..js), not just URL-type events — lightfuzz no longer receives WEB_PARAMETER events for JavaScript URLs.Lightfuzz (
lightfuzz.py)filter_eventrejects WEB_PARAMETER events whoseurl_extensionis inurl_extension_static(pdf, doc, xml, etc.) — eliminates ~25 static-asset FPs from the reference scan.SQLi confirmation loop (
sqli.py)SQLi finding descriptions (
base.py,sqli.py)metadata()now surfacesadditional_paramsfor POSTPARAM/BODYJSON/HEADER/COOKIE findings (truncated).False Positive Reductions — Prior
Serialization (
serial.py) — centralize WAF detection via sharedget_waf_strings(), skip Error Resolution when baseline status code is non-standard (>511).XSS (
xss.py) — verify XSS probe matches appear in the correct HTML context, not just anywhere in the response body.SQLi (
sqli.py) — suppress findings when a single-quote probe triggers a WAF 403 response.Crypto (
crypto.py) — endpoint stability pre-check for padding oracle and CBC bitflip; removed overly generic "access denied" error string.Path Traversal (
path.py) — centralized WAF detection list instead of hardcoded strings.Excavate (
excavate.py) — skip parameter extraction from out-of-scope redirect targets.Global / Config
get_waf_strings()helper with 7 common WAF signatures.Bug Fixes
baseline_probe()using nonexistenteventtypekey instead oftype— was silently falling through to GET for all POSTPARAM/BODYJSON parameters.incoming_probe_value().paramminer_headers: useurl_extensionattribute instead of string endswith for static URL filtering.modify_string()treatingposition=0as falsy.get_waf_strings()definition.Crypto — new features (prior)