Repository navigation
Report unreadable upload files as failed jobs with the parse error - #3114
Conversation
ingest_json_file, ingest_jsonl_file, ingest_csv_file and ingest_parquet_file logged a whole-file read error and returned the stats, so a corrupt file ended as "No data imported" and a truncated JSON past one batch ended as succeeded. They now go through _file_failed, which records first_error and counts an error, and /s3-file and /sync-s3-folder pass first_error on. The json.load retry stays because ijson rejects NaN and Infinity, but it now resumes after the items the stream already ingested instead of sending them again. The unreachable ijson and pyarrow ImportError branches are gone. Closes #3090 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
…wice Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
|
🏭 The factory's review of this PR did not finish; it tries again on a later tick. |
|
🏭 Advisory review (claude-haiku-5-5 + opus advisor) — merge Ran
Did not run
Left as follow-ups (exist on base, outside issue)
|
A text-mode read raises inside its 8KiB chunk, so a short file reported the decode error and stored nothing. Decode one line at a time. A JSON file that already read cases counts the failure as a case error. Co-authored-by: Sam <samarmstrong@users.noreply.github.com>
|
🧹 Handed the merge conflicts back to the agent that opened this (1 of 3): merge conflicts with main. |
…ng-ingest helpers) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
🏭 The factory's review of this PR did not finish; it tries again on a later tick. |
…ep first_error on /s3-file; ratchet 280)
_utf8_lines dropped the line breaks inside quoted CSV fields and re-scanned the buffer on every 8KiB chunk, so one 20MB line took 60s. Split each binary line on \r\n, \r and \n, decode per line, and keep the terminators, matching the base text-mode read. Add a CSV regression test for a quoted newline and rename the NaN retry test to say what it covers. Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
|
🏭 Advisory review (claude-haiku-5-5 + opus advisor) — merge Ran
Did not run
Pushed
Subagents
Entry check
Follow-ups, not blocking
|
|
🧹 Handed the merge conflicts back to the agent that opened this (2 of 3): merge conflicts with main. |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
🧹 Proof at Job: #3090, a fix job in feature
Blast radius: medium. Touches 5 file(s) in the api, tests layer(s), across feature(s)
The author says it is safe because: every whole-file read error, of any type, now lands in How each count was made, and how to rerun itSize. +427 −124 over the whole diff, as GitHub counts it. Reproduce with Callers. What the PR deletes and does not define again, searched for across the repo at
git fetch origin pull/3114/head
git grep -n -I -F -w -e '_ingest_json_full' 95bf98c120b5Style lint. No findings in the prose the PR adds. CI at Evidence. ingestion-and-callers-after.txt · json-resume-probe.txt · new-tests-on-base.txt · ratchets.txt Machine-readable{
"sha": "95bf98c120b54f3c54d39ae9607c1ef1d1ef2ade",
"pr": 3114,
"job": {
"issue": 3090,
"kind": "fix",
"feature": "data-ingestion",
"survey": "2026-10-24-5cf66803-dlib"
},
"blast": {
"severity": "medium",
"reasons": [
{
"severity": "medium",
"what": "touches API handlers: services/api/routers/ingestion.py"
}
],
"features": [
"data-ingestion"
],
"layers": [
"api",
"tests"
],
"routes": [],
"safeBecause": "every whole-file read error, of any type, now lands in `_file_failed`, and the JSON retry ingests only what the stream had not.",
"proofLevel": 4
},
"size": {
"additions": 427,
"deletions": 124
},
"verdicts": {
"size": "n/a",
"callers": "pass",
"lint": "pass",
"ci": "pass"
},
"callers": [
{
"name": "_ingest_json_full",
"kind": "symbol",
"from": "core/ingestion/ingestion_service.py",
"refs": []
}
],
"lint": 0,
"checks": [
{
"name": "Build Frontend",
"bucket": "pass"
},
{
"name": "MCP Runner Image",
"bucket": "pass"
},
{
"name": "Unit Tests - Frontend",
"bucket": "pass"
},
{
"name": "Lint Frontend",
"bucket": "pass"
},
{
"name": "Daemon Image",
"bucket": "pass"
},
{
"name": "Agent Image",
"bucket": "pass"
},
{
"name": "Helm Install Schema",
"bucket": "pass"
},
{
"name": "Unit Tests - Agent Layer",
"bucket": "pass"
},
{
"name": "Setup Path (ubuntu-latest)",
"bucket": "pass"
},
{
"name": "Security Scan - NPM",
"bucket": "pass"
},
{
"name": "Unit Tests - Backend (DB-backed)",
"bucket": "pass"
},
{
"name": "Unit Tests - Backend",
"bucket": "pass"
},
{
"name": "Lint Backend",
"bucket": "pass"
},
{
"name": "Security Scan - Python",
"bucket": "pass"
},
{
"name": "Integration Tests",
"bucket": "pass"
},
{
"name": "Generated API Types",
"bucket": "pass"
},
{
"name": "CodeQL",
"bucket": "pass"
},
{
"name": "Analyze (python)",
"bucket": "pass"
},
{
"name": "Analyze (actions)",
"bucket": "pass"
},
{
"name": "Backend Image",
"bucket": "pass"
},
{
"name": "Analyze (javascript-typescript)",
"bucket": "pass"
},
{
"name": "Setup Path (macos-latest)",
"bucket": "pass"
},
{
"name": "Lint Dockerfiles",
"bucket": "pass"
}
],
"evidence": [
{
"name": "ingestion-and-callers-after.txt",
"url": "https://github.com/Vigil-SOC/vigil/blob/deslop-evidence/pr-3114/ingestion-and-callers-after.txt"
},
{
"name": "json-resume-probe.txt",
"url": "https://github.com/Vigil-SOC/vigil/blob/deslop-evidence/pr-3114/json-resume-probe.txt"
},
{
"name": "new-tests-on-base.txt",
"url": "https://github.com/Vigil-SOC/vigil/blob/deslop-evidence/pr-3114/new-tests-on-base.txt"
},
{
"name": "ratchets.txt",
"url": "https://github.com/Vigil-SOC/vigil/blob/deslop-evidence/pr-3114/ratchets.txt"
}
]
} |
|
🏭 Factory review — merged. An unreadable upload now fails the job and names the cause, and a file ijson rejects for Ran
Did not run
|
Closes #3090
What changed
ingest_json_file,ingest_jsonl_file,ingest_csv_fileandingest_parquet_filecaught a whole-file read error, logged it and returned the stats. Nothing reachedfirst_erroror the error counts. A corrupt Parquet file, a non-UTF-8 CSV or JSONL file, or a malformed JSON file ended as a failed job with "No data imported". A truncated JSON array of 1500 findings ended as a succeeded job with "Imported 1000 findings".Each file reader now sends that error to one
_file_failed(kind, exc, data_type). It logs the error, records "Could not read the {kind} file: {exc}" asfirst_error, and counts one finding or case error./s3-fileand/sync-s3-foldernow passfirst_errortosummarize_stats, as the upload job already did.The issue asked to delete the
json.loadretry. Its check comment showed the retry is live: ijson rejectsNaNandInfinityandjson.loadaccepts them. So the retry stays, scoped toijson.JSONError. It used to re-send every finding the stream had already ingested (2500 rows for a 1500-finding file). Nowfindings_totalandcases_totalcount exactly the items already ingested, and_ingest_json_restresumes after them._ingest_findings_batchedflushes its pending batch in afinallyso that count holds when the stream raises. CSV and JSONL also ingest the rows they read before a decode error, so their counts match. JSONL now opens the file as UTF-8, as CSV and the JSON retry already did, instead of the locale default.The
ImportErrorbranches for ijson and pyarrow are gone. Both are pinned inrequirements.txt. A missing pyarrow now fails the job with "Could not read the Parquet file: No module named ...".test_id_keyed_top_level_array_is_recognised_without_streamingnow callsingest_json_fileon the fixture with a NaN in it, so it still covers the retry path. The f-string log ratchet drops from 288 to 283 because the removed branches held five f-string log calls.Callers checked
ingest_json_file,ingest_jsonl_file,ingest_csv_file,ingest_parquet_file: only_ingest_file_by_format, plustest_ingestion_common_keys.py._ingest_file_by_format:core/ingestion/ingestion_jobs.pyrun_job(upload job),services/api/routers/ingestion.py/s3-file, andingest_s3_folder(/sync-s3-folder). No other app, script, infra file or client calls them._ingest_json_fulland_ingest_json_streaming: onlyingest_json_fileand one test._ingest_json_fullis replaced by_ingest_json_rest. No string references._ingest_findings_batched:ingest_json_filepaths andingest_from_string.ingest_from_stringpasses lists, so thefinallyflush changes nothing there.summarize_stats: the upload job,/ingest-string,/sync-s3-folder,/s3-file, tests. Its signature is unchanged.Blast radius
The JSON the upload job,
/s3-fileand/sync-s3-folderreturn keeps its fields. Only values change. An unreadable file now hasfindings_errors(orcases_errors) of at least 1,success: false, and a message that ends with the cause. A file that partly imported before the error shows as failed with its counts. No DB schema, wire format or other language reads these stats._file_failed, and the JSON retry ingests only what the stream had not.tests/unit/ingestion/test_unreadable_upload_files.pydrivesrun_jobwith the realIngestionServiceand a fakebulk_create_findingsfor truncated JSON (small and past one batch),[1, 2, 3], corrupt Parquet, latin-1 CSV (finding and case), a CSV whose bad byte comes after 1500 rows, non-UTF-8 JSONL, and NaN files (array and{findings}), asserting each ID is stored once in order. On base:10 failed, 8 passed. On this branch: all pass. A probe also put NaN in thecasesarray after findings, in a finding with cases present, and in a case-only array. Each item was ingested once and the totals matched.run_job. Output is in the evidence files below.core/ingestion/ingestion_service.py:560: a JSON file that used to end "No data imported" (an array of non-objects, orfindings: nullwith a NaN) now fails with the exception text as its cause. Likely rare, low impact, and it is the intended outcome.core/ingestion/ingestion_service.py:618: the resume slices by count. If the stream ever counted an item it did not ingest, that item would be skipped. The counts and ingests sit in_countedand thefinallyflush, and the NaN tests check exact IDs. Unlikely, medium impact.core/ingestion/ingestion_service.py:663and:705: CSV and JSONL now ingest the rows read before a decode error. Those rows were already counted as total, so the job fails with the real imported count. Low impact./s3-fileand S3 listing tests, daemon and integration tests that importingestion_service(283 passed). Classification of a top-level array is the same in the stream and the retry (shared_is_case_array).Also seen
ingest_s3_foldersumsfile_statsintoself.stats, butfile_statsisself.statsand each file callsreset_stats(). Earlier files' counts are wiped, the last file's counts double, andfirst_errorreflects only the last file._ingest_json_streaming. It ends "No data imported" with no error.nullreads as empty, becausenext(items, None)usesNoneas the empty marker.ingest_from_string(/ingest-string) still logs and swallows a parse error, and per-line JSONL and per-row CSV errors set nofirst_error.Verification
Ran
pytest tests/unit/ingestionplus every test file that importsingestion_serviceor the ingestion router:283 passed. Evidence:ingestion-and-callers-after.txt.test_ingestion_common_keys.pywith the service and router reverted to base:10 failed, 8 passed. Evidence:new-tests-on-base.txt.json-resume-probe.txt.pytest tests/unit/_ratchets:260 passed, 6 skipped(helm not installed) after lowering the f-string baseline to 283. Evidence:ratchets.txt.black --checkandflake8on the changed files: clean.JSONErrorexceptions escapedingest_json_fileand that CSV and JSONL dropped buffered rows on a decode error. Both are fixed and tested. Its other points are listed under Also seen.Did not run
run_jobis the entry point the upload endpoint calls. The tests drive it with the real service and a fake DB.🤖 Generated with Claude Code