Support bzip2 and xz compressed inputs in LLM file analysis - #70302
Support bzip2 and xz compressed inputs in LLM file analysis#70302guan404ming wants to merge 4 commits into
Conversation
45bb2b6 to
3ed4e00
Compare
eee5e4e to
11472ad
Compare
| return _read_limited_bytes(handle, path=path, max_bytes=max_bytes) | ||
| if compression is None: | ||
| return _read_limited_bytes(handle, path=path, max_bytes=max_bytes) | ||
| with _DECOMPRESSORS[compression](handle) as decompressed: |
There was a problem hiding this comment.
One asymmetry inside this dispatch: gzip.open raises BadGzipFile if anything follows a complete stream, but bz2.open and lzma.open silently return just the first stream, so _read_limited_bytes sees a normal EOF and nothing marks the content as incomplete. The spec-legal case is the interesting one: with 4 null bytes of xz Stream Padding between two streams, xz -t passes and xz -dc prints both, while lzma.open returns only the first. Pre-PR these inputs were rejected outright, so it's worth either treating non-null unused_data after eof as an error via BZ2Decompressor/LZMADecompressor, or pinning the current behavior in a test and noting the limitation in the docs.
There was a problem hiding this comment.
Pinned current behavior in tests and documented the limitation; decompressor-level detection deferred.
| f"Unsupported file format {detected!r} for {path}. Supported formats: {', '.join(SUPPORTED_FILE_FORMATS)}." | ||
| ) | ||
| if compression and compression != "gzip": | ||
| if compression and compression not in _DECOMPRESSORS: |
There was a problem hiding this comment.
On a build without _lzma this branch fires first and reports Compression 'xz' is not supported for file analysis., the same message zstd gets, even though the docs now say xz is supported, so it points people at their file rather than at their Python build. It also makes the message for something like sample.parquet.bz2 depend on the build, since the format-combination check below only runs when the codec module is present. Checking a static {"gzip", "bzip2", "xz"} set here and raising AirflowOptionalProviderFeatureException for a known-but-missing codec would match what _render_parquet and _render_avro already do lower down, and the rst could gain a clause saying bzip2/xz need an interpreter built with those modules.
There was a problem hiding this comment.
Fixed: known-codec set, AirflowOptionalProviderFeatureException naming the missing module, docs note added.
| path.write_bytes(b"content") | ||
|
|
||
| with pytest.raises(LLMFileAnalysisUnsupportedFormatError, match="not supported for"): | ||
| with patch.dict(_DECOMPRESSORS, {"gzip": gzip.open}, clear=True): |
There was a problem hiding this comment.
This pins the rejection but not the half I was actually worried about, that plain and gzip inputs keep working while the codecs are missing, which is one more assert inside the same patched block. Related, import bz2 and import lzma at the top of this module are unguarded, so on exactly those builds the module fails to collect, and sample.avro.bz2 / sample.png.xz in the combination test would hit the codec-level message and fail its regex rather than skip. pytest.importorskip (already used for pyarrow and fastavro below) plus .gz params for the combination test covers it, and the expansion-bound tests at 262 and 279 are still gzip-only if you want to reuse the parametrize you added at 410.
There was a problem hiding this comment.
Added plain/gzip asserts and importorskip, parametrized per-file expansion; total-limit test stays gzip-only.
29ef04c to
efecc29
Compare
2 participants
Why
.log.xzor.csv.bz2were rejected outright.How
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 4.8) following the guidelines