Repository navigation
Conversation
doc_batch_v2_from_ndjson attaches skipped blank lines to neighboring docs to stay zero-copy. Blank lines are detected with is_ascii_whitespace, which includes form feed, but JSON does not accept form feed as whitespace: a body with a form-feed-only line made its neighboring doc invalid JSON, whereas the previous line splitter dropped such lines. Fall back to copying the non-blank lines when a blank line contains a form feed.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Follow-up to #6869.
doc_batch_v2_from_ndjsonstays zero-copy by attaching skipped blank lines to neighboring docs. Blank lines are detected withis_ascii_whitespace, which includes form feed (\x0c), but JSON does not accept form feed as whitespace.Before #6869, the ingest v2 REST handler dropped blank lines, so
\x0c\n{"id":1}\ningested one valid doc. Now the doc is\x0c\n{"id":1}\n, which fails validation as invalid JSON. Trailing form-feed lines have the same effect on the last doc.Fix: when a blank line contains a form feed, fall back to copying only the non-blank lines (same output as
split_ndjson_lines). Bodies without form feeds keep the zero-copy path; the check is a singlememchrfor them.How was this PR tested?
test_doc_batch_v2_from_ndjson_form_feed_blank_lines(leading, inner, and trailing form-feed blank lines produce valid JSON docs with increasing doc UIDs) and a form-feed-only body case totest_doc_batch_v2_from_blank_ndjson.cargo test -p quickwit-ingest ndjson --lib: 6 passed.cargo clippy -p quickwit-ingest --testsandcargo fmt --check: clean.