feat(scanner): scan joblib, dill and numpy pickle variants - #91
Conversation
The everyday serialization formats of scikit-learn and scientific Python
are all pickle underneath, carrying exactly the arbitrary-code-execution
risk of a .pt — and until now a directory of malicious .pkl, .joblib,
.dill, .npy and .npz files reported "No AI models found" and exited 0.
.pkl is the sharpest case: the README has always documented
`aisbom scan model.pkl --strict`, and that command scanned nothing.
Three defects were in the way of this working honestly:
* A raw array block ended the scan. joblib writes its pickle up to an
array, dumps the buffer inline, then resumes — killing a structural
disassembly 226 bytes into a 552-byte file while the payload sat at
byte 516. Every real model has weights, so "after the first array"
is where a payload naturally goes. Bytes the walk cannot reach now
get a bounded pass that recovers globals directly.
* STACK_GLOBAL operands arriving from the memo via BINGET misresolved.
numpy.dtype read as dtype.dtype — a strict-mode false positive on
every ordinary joblib model, and a false negative on other shapes.
* dill's code reconstruction was invisible in blocklist mode. A dill'd
function is a marshalled code object rebuilt on load, the same
construct that makes a Keras Lambda layer an execution vector.
No runtime dependency is added: compression is stdlib, .npy headers are
parsed by hand, and nothing is ever unpickled. lz4 and zstd have no
stdlib decompressor and are named rather than opened, on the same
reasoning as 7z containers. joblib, dill and numpy are dev-group only,
for fixtures and for cross-validating the parsers in both directions.
Verified: 775 passed, 91.26% coverage. Scorecard holds at 8/11 with
floor.json byte-identical, and an old-vs-new verdict differential over
134 fixture verdicts in both modes is empty — nothing already scanned
changed verdict.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1edc4b843e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| elif ext in PICKLE_VARIANT_EXTENSIONS: | ||
| with RemoteStream(url) as stream: | ||
| self.artifacts.append( | ||
| self._inspect_pickle_variant(stream, Path(url).name, is_remote=True) | ||
| ) |
There was a problem hiding this comment.
For hf:// targets, this dispatch arm is never reached because resolve_huggingface_repo() still limits supported_exts in aisbom/remote.py:149-153 to the older formats. Consequently, repositories containing .pkl, .pickle, .joblib, .dill, .npy, or .npz artifacts return no target for those files and can complete without an artifact or error; add the new extensions to the resolver as well.
Useful? React with 👍 / 👎.
| stream.seek(0) | ||
| blob = stream.read(budget) | ||
| if not isinstance(blob, bytes): | ||
| blob = bytes(blob) | ||
| details["truncated"] = len(blob) >= budget |
There was a problem hiding this comment.
When a local artifact exceeds 16 MiB or a remote artifact exceeds 2 MiB, only the prefix is scanned, but details["truncated"] never affects the risk result. A valid pickle can place a large BINBYTES value before an os.system reduction, causing the prefix scan to find no threat and the CLI to return success with only MEDIUM (Pickle Present) even though loading the complete artifact executes the omitted opcode; capped reads need to produce an incomplete-scan result rather than being treated as fully inspected.
Useful? React with 👍 / 👎.
| members = z.namelist()[:NPZ_MAX_MEMBERS] | ||
| details["internal_files"] = len(z.namelist()) |
There was a problem hiding this comment.
For an NPZ containing more than 512 entries, every member after the slice is silently ignored even though internal_files records that they exist. An attacker can put 512 benign numeric arrays first and a malicious object-array pickle afterward, causing carries_pickle to remain false and the artifact to be reported LOW; reaching this member cap must mark the scan incomplete instead of allowing a clean result.
Useful? React with 👍 / 👎.
| if label == _ZLIB: | ||
| return zlib.decompressobj().decompress(data, limit) or None | ||
| if label == _GZIP: | ||
| # 16 + MAX_WBITS selects the gzip wrapper. | ||
| return zlib.decompressobj(16 + zlib.MAX_WBITS).decompress(data, limit) or None |
There was a problem hiding this comment.
For a zlib-compressed joblib whose expanded pickle exceeds the 16 MiB limit, this returns the prefix without indicating that the decompressor stopped because of max_length. A dangerous global placed after a large compressible pickle value is therefore never scanned, while _inspect_pickle_variant() treats the returned payload as readable and reports only MEDIUM (Pickle Present); the decompression result must preserve whether the stream reached EOF so callers can mark capped output incomplete. The gzip, bz2, lzma, and xz branches have the same issue.
Useful? React with 👍 / 👎.
| if isinstance(descr, str): | ||
| return "O" in descr.lstrip("|<>=") | ||
| if isinstance(descr, (list, tuple)): | ||
| return any(_is_object_dtype(part) for part in descr) |
There was a problem hiding this comment.
Recursing through every tuple element also interprets structured-array field names as dtype strings. For example, the valid non-object descriptor [('FOO', '<i4')] returns true solely because the field name contains O, so an ordinary numeric .npy is incorrectly reported as MEDIUM (Pickle Present); structured descriptors should inspect the field dtype element (and nested/subarray dtype positions), not names and other metadata.
Useful? React with 👍 / 👎.
Addresses the Codex review on this branch. Four findings were the same
mistake in four places — a bound that stops the scan without saying so —
which is the lesson the concatenated-stream cap already taught.
* A capped file read reported MEDIUM (Pickle Present). A valid pickle
can carry a large BINBYTES value ahead of its payload, so the prefix
disassembles clean while the part that matters was never read.
* An .npz with more than 512 members silently ignored the rest. Benign
numeric arrays first and an object array after the cap read as LOW.
* Bounded decompression discarded the fact that it had stopped early,
so a global after a large compressible value was never scanned while
the payload was treated as fully read.
* The Hugging Face resolver filters the file list before dispatch ever
runs, so the new extensions were skipped for hf:// scans no matter
how the dispatch arm was wired. Its list is now derived from the
scanner's dispatch sets rather than restated beside them; the test
that missed this patched the resolver out, which is why it passed.
All four now report MEDIUM (Pickle Scan Incomplete), and a real finding
still outranks the marker.
Two more, found while fixing the above:
* Structured dtypes read field *names* as type codes, so the ordinary
[('FOO', '<i4')] came back as an object array on the strength of the
letter O. Only the format element of an entry is a dtype.
* The decompression limit came from a default argument, which binds
once at import — so a remote scan that fetched 2MB could expand 16,
and the remote budget was defined but unreachable. The scanner now
passes the same figure it used to read the file.
Verified: 790 passed, 91.43% coverage. Scorecard holds at 8/11 and the
old-vs-new differential over 134 fixture verdicts is still empty.
1 participant
Extends pickle discovery and disassembly to the non-torch serialization wrappers that also embed pickle streams: joblib (including its compressed containers), dill, and numpy object arrays. These are the everyday formats of scikit-learn and scientific Python, they carry exactly the arbitrary-code-execution risk of a
.pt, and generic SBOM tooling opens none of them.Until this, a directory holding malicious
.pkl,.pickle,.joblib,.dill,.npyand.npzfiles reported "No AI models found" and exited 0..pklis the sharpest case — the README has always documentedaisbom scan model.pkl --strict, and that command scanned nothing.Three defects were in the way of this working honestly
A raw array block ended the scan. joblib writes its pickle up to an array wrapper, dumps the raw buffer inline, then resumes pickling.
pickletools.genopscannot cross those bytes, so a structural walk died at byte 226 of a 552-byte file while the payload sat at byte 516 — and returned[]with no incompleteness marker, so the file read as clean. Since every real model carries weights, "after the first array" is the natural place for a payload.Bytes the structural walk cannot reach now get a second, non-structural pass that recovers globals directly. It reads bytes rather than structure, so it recognises known sinks only — it contributes no unrecognized-import findings in strict mode, and it cannot judge a dual-use constructor like
operator.methodcaller("system"), because the argument that decides is a stack relationship it has no stack for. Both limits are locked in by tests rather than left in a comment.STACK_GLOBALoperands arriving from the memo misresolved. Either operand can reach the stack viaBINGETrather than a literal, which a real joblib file does routinely:Watching only the literals resolved that as
dtype.dtype— a strict-mode false positive on every ordinary joblib model, and on other shapes a dangerous global resolving to a module name matching nothing. Tracking the memo table needs no stack simulation: a memo entry is always the value just pushed.dill's code reconstruction was invisible in blocklist mode. A dill'd function or lambda is a marshalled code object rebuilt on load — the same construct that makes a Keras
Lambdalayer an execution vector._create_function,_create_code,_import_moduleand_get_attrare now CRITICAL._create_type,_load_typeand_create_arrayare deliberately not flagged, with a test proving it: they name or rebuild a value and execute nothing, and flagging them would make every.dillholding a custom class CRITICAL.Design decisions
ZF),.npyheaders are parsed by hand, and nothing is ever unpickled.joblib,dillandnumpyare dev-group only — for building realistic fixtures and cross-validating the parsers in both directions.lz4andzstdare named, not opened —MEDIUM (Unscanned Container: lz4). Same reasoning as 7z containers: a native decompressor would land in every install and every standalone binary. An honest "we did not read these bytes" is a different answer from a clean scan..npyheader is attacker-supplied, so a pickle behind a header claiming'<f8'would be a one-line evasion. The data section is disassembled whatever the header says. What the header does affect is the reported risk: an array of ordinary numbers reportsLOW, not "pickle present"..npzmembers go through the existing raw local-header read, so a tampered CRC or filename does not buy silence — numpy's reader does not verify whatZipFile.openverifies.joblibandnumpyjoinpickleinproperties._format_for, and all three emit the sameaisbom:pickle:opcodeproperties — a consumer that reads a threat off a.ptreads one off a.joblibunchanged.RemoteStreamarm reports everyhf://model clean.Verification
poetry run pytest --cov=aisbom --cov-fail-under=85aisbom bypass-scorecard --checkfloor.jsonbyte-identicalThe differential scanned 134 verdicts of identical bytes with shipped
mainand with this branch: all 13 documented bypass-corpus cases, 13 pickle shapes across.pt/.pth/.bin, checkpoint-shaped pickles at protocols 2/4/5, and safetensors/gguf/keras/onnx — in both scan modes.diffexits 0. Nothing already scanned changed verdict.A false-positive sweep covers 10 ordinary object shapes across protocols 2/4/5 in both modes, plus benign fixtures in every new format and codec. Real joblib and numpy files are read by our parsers, and files our generator assembles are loaded back by the real libraries — both directions, so the tests cannot be self-consistent and both wrong.
User-visible
Five extensions that were skipped are now scanned, so a repo containing them produces more SBOM components than before and may newly exit
2. That is the intended effect; nothing that was already scanned changes.cve-2025-1889-nonstandard-extensionstaysmissed— it usesconfig.p, and the honest fix for that class is content-sniffing unknown files, which is its own design decision rather than another extension on the list.🤖 Generated with Claude Code