fix: JSONConverter raises KeyError on ByteStream sources without file_path - #12208
Merged
julian-risch merged 1 commit intoAug 4, 2026
Merged
julian-risch merged 1 commit into
julian-risch merged 1 commit into
Conversation
…_path
All three exception handlers in _get_content_and_meta() (invalid UTF-8,
a failing jq_schema filter, malformed JSON) log a warning using
source.meta["file_path"] -- an unconditional dict-key access. A bare
ByteStream (e.g. ByteStream.from_string(...), the exact pattern shown
in this component's own docstring examples) has no such key, so the
error-handling code itself raised KeyError, masking the real error and
crashing run() instead of skipping that one source as intended.
Fixed by using source.meta.get("file_path", "unknown") at all three
sites.
|
@mittalpk is attempting to deploy a commit to the deepset Team on Vercel. A member of the Team first needs to authorize it. |
|
|
julian-risch
approved these changes
Aug 4, 2026
julian-risch
left a comment
julian-risch
left a comment
Member
There was a problem hiding this comment.
Looks good to me! Thank you for your contribution!
Contributor
Coverage report
Click to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related Issues
JSONConverter, in the same pass as fix: MSGToDocument raises KeyError on ByteStream sources without file_path #12207.Proposed Changes
All three exception handlers in
_get_content_and_meta()— invalid UTF-8 decoding, ajq_schemafilter that fails to apply, and malformed JSON — log a warning usingsource.meta["file_path"], an unconditional dict-key access. A bareByteStream(e.g.ByteStream.from_string(...), the exact pattern shown in this component's own docstring usage examples) has emptymetaby default, so on any of these three error paths the error-handling code itself raisesKeyError: 'file_path', masking the real error (UnicodeDecodeError, filter failure, orJSONDecodeError) and crashingrun()entirely instead of logging a warning and skipping that one source, which is the documented/intended behavior.Two prior merged PRs (#8775, #11980) touched this exact function, but both only tested via file-path sources, which auto-populate
file_pathinmeta— so this bug survived both.Fixed by using
source.meta.get("file_path", "unknown")at all three call sites, consistent with howrun()itself already safely doesbytestream.meta.get("file_path")a few lines below.How did you test it?
Reproduced the crash directly against
main(JSONConverter(content_key="text").run(sources=[ByteStream(data=b"\xff\xfe not valid utf-8")])), confirmed theKeyError.Added three new tests, one per exception path, each using a bare
ByteStreammirroring the repo's existing file-path-based tests for the same three paths (test_run_with_bad_filter,test_run_with_bad_encoding,test_run_with_malformed_json_and_content_key). Confirmed all three fail on unpatchedjson.py(git stash) with the sameKeyError, and pass after the fix. Ran the full converters unit suite (hatch run test:unit test/components/converters/) — 265 passed, no regressions.hatch run test:types— no issues.hatch run fmt— clean.Notes for the reviewer
Same shape of bug and same root cause as #12207 (
MSGToDocument), found in the same audit pass — flagging in case a reviewer wants to look at both together, though they're independent files/components and can be merged separately.Checklist
fix:) for the PR title.hatch run fmt/hatch run test:types/hatch run test:unitand fixed any issue.This PR was developed with AI-assisted tooling (Claude Code); I reviewed the diagnosis, the fix, and the tests, and ran the test suite and quality checks locally as noted above.