Skip to content

Report unreadable upload files as failed jobs with the parse error - #3114

Merged
samarmstrong merged 10 commits into
mainfrom
cleanup/3090-report-unreadable-upload-files-as-3ed25304
Oct 24, 2026
Merged

samarmstrong merged 10 commits into
mainfrom
cleanup/3090-report-unreadable-upload-files-as-3ed25304

Conversation

@G-r-ay

@G-r-ay G-r-ay commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator

Closes #3090

What changed

ingest_json_file, ingest_jsonl_file, ingest_csv_file and ingest_parquet_file caught a whole-file read error, logged it and returned the stats. Nothing reached first_error or the error counts. A corrupt Parquet file, a non-UTF-8 CSV or JSONL file, or a malformed JSON file ended as a failed job with "No data imported". A truncated JSON array of 1500 findings ended as a succeeded job with "Imported 1000 findings".

Each file reader now sends that error to one _file_failed(kind, exc, data_type). It logs the error, records "Could not read the {kind} file: {exc}" as first_error, and counts one finding or case error. /s3-file and /sync-s3-folder now pass first_error to summarize_stats, as the upload job already did.

The issue asked to delete the json.load retry. Its check comment showed the retry is live: ijson rejects NaN and Infinity and json.load accepts them. So the retry stays, scoped to ijson.JSONError. It used to re-send every finding the stream had already ingested (2500 rows for a 1500-finding file). Now findings_total and cases_total count exactly the items already ingested, and _ingest_json_rest resumes after them. _ingest_findings_batched flushes its pending batch in a finally so that count holds when the stream raises. CSV and JSONL also ingest the rows they read before a decode error, so their counts match. JSONL now opens the file as UTF-8, as CSV and the JSON retry already did, instead of the locale default.

The ImportError branches for ijson and pyarrow are gone. Both are pinned in requirements.txt. A missing pyarrow now fails the job with "Could not read the Parquet file: No module named ...".

test_id_keyed_top_level_array_is_recognised_without_streaming now calls ingest_json_file on the fixture with a NaN in it, so it still covers the retry path. The f-string log ratchet drops from 288 to 283 because the removed branches held five f-string log calls.

Callers checked

  • ingest_json_file, ingest_jsonl_file, ingest_csv_file, ingest_parquet_file: only _ingest_file_by_format, plus test_ingestion_common_keys.py.
  • _ingest_file_by_format: core/ingestion/ingestion_jobs.py run_job (upload job), services/api/routers/ingestion.py /s3-file, and ingest_s3_folder (/sync-s3-folder). No other app, script, infra file or client calls them.
  • _ingest_json_full and _ingest_json_streaming: only ingest_json_file and one test. _ingest_json_full is replaced by _ingest_json_rest. No string references.
  • _ingest_findings_batched: ingest_json_file paths and ingest_from_string. ingest_from_string passes lists, so the finally flush changes nothing there.
  • summarize_stats: the upload job, /ingest-string, /sync-s3-folder, /s3-file, tests. Its signature is unchanged.

Blast radius

The JSON the upload job, /s3-file and /sync-s3-folder return keeps its fields. Only values change. An unreadable file now has findings_errors (or cases_errors) of at least 1, success: false, and a message that ends with the cause. A file that partly imported before the error shows as failed with its counts. No DB schema, wire format or other language reads these stats.

  • Safe because: every whole-file read error, of any type, now lands in _file_failed, and the JSON retry ingests only what the stream had not.
  • Tried to break it: if the first half were false, a corrupt file of each format would still end "No data imported" or succeeded. If the second were false, a file with NaN after item 1200 would store more than 1500 rows or skip some. tests/unit/ingestion/test_unreadable_upload_files.py drives run_job with the real IngestionService and a fake bulk_create_findings for truncated JSON (small and past one batch), [1, 2, 3], corrupt Parquet, latin-1 CSV (finding and case), a CSV whose bad byte comes after 1500 rows, non-UTF-8 JSONL, and NaN files (array and {findings}), asserting each ID is stored once in order. On base: 10 failed, 8 passed. On this branch: all pass. A probe also put NaN in the cases array after findings, in a finding with cases present, and in a case-only array. Each item was ingested once and the totals matched.
  • Proof: 4. The tests call the real ingestion code through run_job. Output is in the evidence files below.
  • Severity: medium.
  • Risks:
    • core/ingestion/ingestion_service.py:560: a JSON file that used to end "No data imported" (an array of non-objects, or findings: null with a NaN) now fails with the exception text as its cause. Likely rare, low impact, and it is the intended outcome.
    • core/ingestion/ingestion_service.py:618: the resume slices by count. If the stream ever counted an item it did not ingest, that item would be skipped. The counts and ingests sit in _counted and the finally flush, and the NaN tests check exact IDs. Unlikely, medium impact.
    • core/ingestion/ingestion_service.py:663 and :705: CSV and JSONL now ingest the rows read before a decode error. Those rows were already counted as total, so the job fails with the real imported count. Low impact.
    • Settings > Data shows "Failed" for a file that partly imported. That is the issue's intent.
  • Cleared: the upload API tests, /s3-file and S3 listing tests, daemon and integration tests that import ingestion_service (283 passed). Classification of a top-level array is the same in the stream and the retry (shared _is_case_array).

Also seen

  • ingest_s3_folder sums file_stats into self.stats, but file_stats is self.stats and each file calls reset_stats(). Earlier files' counts are wiped, the last file's counts double, and first_error reflects only the last file.
  • A JSON file that starts with whitespace or a UTF-8 BOM matches neither peek branch in _ingest_json_streaming. It ends "No data imported" with no error.
  • A JSON array whose first item is null reads as empty, because next(items, None) uses None as the empty marker.
  • ingest_from_string (/ingest-string) still logs and swallows a parse error, and per-line JSONL and per-row CSV errors set no first_error.

Verification

Ran

  • pytest tests/unit/ingestion plus every test file that imports ingestion_service or the ingestion router: 283 passed. Evidence: ingestion-and-callers-after.txt.
  • The new tests and test_ingestion_common_keys.py with the service and router reverted to base: 10 failed, 8 passed. Evidence: new-tests-on-base.txt.
  • A probe of NaN in cases, in findings with cases, and in a case-only array: each item ingested once. Evidence: json-resume-probe.txt.
  • pytest tests/unit/_ratchets: 260 passed, 6 skipped (helm not installed) after lowering the f-string baseline to 283. Evidence: ratchets.txt.
  • black --check and flake8 on the changed files: clean.
  • A subagent reviewed the diff. It found that non-JSONError exceptions escaped ingest_json_file and that CSV and JSONL dropped buffered rows on a decode error. Both are fixed and tested. Its other points are listed under Also seen.

Did not run

  • The running app. Severity is medium, and run_job is the entry point the upload endpoint calls. The tests drive it with the real service and a fake DB.

🤖 Generated with Claude Code

ingest_json_file, ingest_jsonl_file, ingest_csv_file and ingest_parquet_file
logged a whole-file read error and returned the stats, so a corrupt file
ended as "No data imported" and a truncated JSON past one batch ended as
succeeded. They now go through _file_failed, which records first_error and
counts an error, and /s3-file and /sync-s3-folder pass first_error on.

The json.load retry stays because ijson rejects NaN and Infinity, but it now
resumes after the items the stream already ingested instead of sending them
again. The unreachable ijson and pyarrow ImportError branches are gone.

Closes #3090

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
G-r-ay pushed a commit that referenced this pull request Oct 24, 2026
@G-r-ay G-r-ay added the cleanup Opened by the cleanup factory label Oct 24, 2026
@G-r-ay G-r-ay added the cleanup:blast-medium Blast radius: how bad it is if this PR is wrong label Oct 24, 2026
@G-r-ay
G-r-ay marked this pull request as ready for review October 24, 2026 18:55
deeptempo-factory and others added 2 commits October 24, 2026 20:12
Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
…wice

Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
@samarmstrong

Copy link
Copy Markdown
Collaborator

🏭 The factory's review of this PR did not finish; it tries again on a later tick.

@G-r-ay

G-r-ay commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator Author

🏭 Advisory review (claude-haiku-5-5 + opus advisor) — merge 0e2945be296f. Advice only: nothing was merged or closed on it.

Ran

  • Merged origin/main (7 commits). One conflict in ingestion_service.py, resolved to keep PR logic plus main's use_float=True.
  • Subagents (all read/run only):
    • Part one, ingestion_service.py: logic. Found a double-send in the batch flush (fixed), a false "exact counts" claim, and a false CSV/JSONL claim (rows before a bad byte are lost in small files; PR body wrong, code not fixed).
    • Part two, router/tests/ratchet: tests real. Reverting service to base gives 10 failed, 8 passed, as claimed. Ratchet 283 correct. /sync-s3-folder is in the PR body.
    • Scope: keep json.load retry (NaN support). Inline single-caller helper (done). Cut resume not taken: cutting it fails NaN files >1000 findings that base imports.
    • Live check (API, branch vs base, own postgres): truncated JSON succeeded→failed; parquet, CSV, JSONL No data imported→failed with Could not read the … file: <cause>.
    • UI (Settings > Data, 1440 and 1920; 1440 is the documented minimum in docs/design/console/DESIGN.md §7): 16 screenshots, no overflow or sideways scroll.
  • Pushed to branch:
    • eba47278eb7b inline _ingest_json_rest into retry.
    • 0e2945be296f empty batch before flush, so a raised batch isn't resent. Probe: sends [1000, 1000, 0].
    • Black and flake8 clean on touched files.
  • Ingestion tests: 113 passed, 3 errors in test_json_fractional_values_db.py (need POSTGRES_PASSWORD; not run here, CI covers).
  • Evidence: /home/gray/Deeptempo/deeptempo-factory/telemetry/evidence/cebbd2e6-c865-4fa4-b3d1-3eaaaf8b733d/settings-data-{before,after}-{truncated-json,parquet,csv-latin1,jsonl-latin1}-{failed,done}-{1440,1920}.png (16 files).

Did not run

  • Full suite; postgres-backed tests.
  • Narrower width than 1440 (not documented minimum).
  • UI check agent briefly started a backend against shared postgres localhost:5432 before switching to its own DB; startup refused. Unverified whether it wrote anything there.

Left as follow-ups (exist on base, outside issue)

  • Leading whitespace/BOM/empty JSON: silent no-op.
  • [null, …] array drops file silently.
  • Junk item in array: findings_total overcounts.
  • JSONL per-line errors set no first_error.
  • /ingest-from-string still swallows parse errors.
  • Cases-first truncated dict loses cases.
  • ingest_s3_folder stats sum bug.
  • PR body claim "CSV/JSONL keep rows before decode error" is false for small files; author should correct.

A text-mode read raises inside its 8KiB chunk, so a short file reported
the decode error and stored nothing. Decode one line at a time. A JSON
file that already read cases counts the failure as a case error.

Co-authored-by: Sam <samarmstrong@users.noreply.github.com>
@G-r-ay

G-r-ay commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator Author

🧹 Handed the merge conflicts back to the agent that opened this (1 of 3): merge conflicts with main.

…ng-ingest helpers)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@samarmstrong

Copy link
Copy Markdown
Collaborator

🏭 The factory's review of this PR did not finish; it tries again on a later tick.

deeptempo-factory and others added 2 commits October 24, 2026 22:51
_utf8_lines dropped the line breaks inside quoted CSV fields and re-scanned
the buffer on every 8KiB chunk, so one 20MB line took 60s. Split each binary
line on \r\n, \r and \n, decode per line, and keep the terminators, matching
the base text-mode read. Add a CSV regression test for a quoted newline and
rename the NaN retry test to say what it covers.

Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
@G-r-ay

G-r-ay commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator Author

🏭 Advisory review (claude-haiku-5-5 + opus advisor) — merge 429295f043f0. Advice only: nothing was merged or closed on it.

Ran

  • Merged origin/main (4 new commits). Conflicts: router (main deleted /sync-s3-folder; kept first_error on /s3-file), ratchet baseline → 280.
  • Ratchet test_no_logger_fstrings: pass at 280.
  • Ingestion tests + S3 list tests: 134 pass. 3 errors in test_json_fractional_values_db.py need POSTGRES_PASSWORD; same on main.
  • After line-reader fix: test_ingestion_common_keys + test_unreadable_upload_files: 24 pass. New CSV test fails on prior code ('line oneline two'), passes now.
  • _utf8_lines probe vs base text mode: quoted newline, CRLF, bare CR, no-final-newline, bad byte mid-file all match. 20MB line: 60s → 0.13s.
  • black, flake8 clean on touched files.
  • Live API (own Postgres/Redis on 7410/7420, backends 7430/7431, stopped by PID), base vs head on /api/ingest/upload:
    • truncated 1500 JSON: succeeded/1000 → failed, cause shown
    • [1,2,3], corrupt parquet, latin-1 CSV, non-UTF-8 JSONL: failed "No data imported" → failed with cause
    • NaN 1500 JSON: succeeded, 1500 distinct, exactly once
  • Control good JSON: unchanged.

Did not run

  • Live API re-run on 429295f0. Live run was on b4860f8f. The only later change is the line reader, covered by unit tests through run_job.
  • Settings > Data screen at current head. No screenshots this pass. Earlier author run (0e2945be) reported no overflow at 1440/1920. Not re-verified.
  • Full suite, Bifrost paths, /ingest-string.

Pushed

  • b4860f8f: merge origin/main. Conflicts resolved as above.
  • 429295f0:
    • _utf8_lines rewritten: per-line decode, universal newlines, linear. Fixes CSV quoted-newline data loss and quadratic slowdown.
    • Regression test for quoted newline.
    • Renamed test_id_keyed_top_level_array_is_recognised_without_streaming → ..._with_nan_is_recognised_on_the_json_retry. Old name was wrong; test passes on main too.

Subagents

  • Part one (ingestion_service.py): CSV newline loss, quadratic line reader, null-first-element and BOM/whitespace silent no-op, error on wrong counter, parquet total drift. First two fixed. Others pre-existing or follow-up.
  • Part two (router/tests/ratchet): router correct; new tests fail on main (14/16); no stale refs; /ingest-string still has no first_error (out of issue scope).
  • Scope: core fix in scope, no duplicate on main. Body stale: /sync-s3-folder claim, ratchet 288→283 claim (really 285→280), ingest_s3_folder callers/bullets, line refs. Keep-rows and line-reader are extra scope but small. No .github/ changes.
  • Live: all rows above; exact-once NaN confirmed.

Entry check

Follow-ups, not blocking

  • Leading whitespace/BOM JSON: silent "No data imported".
  • [null, …] array: silent success.
  • Parquet findings_total = all rows on mid-file failure.
  • /ingest-string swallows parse errors.
  • Body corrections above.

@G-r-ay

G-r-ay commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator Author

🧹 Handed the merge conflicts back to the agent that opened this (2 of 3): merge conflicts with main.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@G-r-ay

G-r-ay commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator Author

🧹 Proof at 95bf98c120b5. Counted by the cleanup factory at this commit; a push or a settled CI run replaces it. Each count says how to reproduce it.

Job: #3090, a fix job in feature data-ingestion, from survey 2026-10-24-5cf66803-dlib.

Check Result
Blast radius medium: if wrong, one feature behaves wrong for its users
Size n/a (+427 −124)
Callers of what it deletes pass
Style lint pass
CI pass
Ready for review yes: taken out of draft

Blast radius: medium. Touches 5 file(s) in the api, tests layer(s), across feature(s) data-ingestion. Why:

  • medium: touches API handlers: services/api/routers/ingestion.py

The author says it is safe because: every whole-file read error, of any type, now lands in _file_failed, and the JSON retry ingests only what the stream had not. (proof level 4 of 5, a claim to check, not a count).

How each count was made, and how to rerun it

Size. +427 −124 over the whole diff, as GitHub counts it. Reproduce with gh pr view 3114 -R Vigil-SOC/vigil --json additions,deletions.

Callers. What the PR deletes and does not define again, searched for across the repo at 95bf98c120b5. A symbol is matched as a whole word, and a route by the fixed part of its path.

  • _ingest_json_full (symbol, from core/ingestion/ingestion_service.py): 0 references
git fetch origin pull/3114/head
git grep -n -I -F -w -e '_ingest_json_full' 95bf98c120b5

Style lint. No findings in the prose the PR adds.

CI at 95bf98c120b5. 23 of 23 pass. Reproduce with gh pr checks 3114 -R Vigil-SOC/vigil.

Evidence. ingestion-and-callers-after.txt · json-resume-probe.txt · new-tests-on-base.txt · ratchets.txt

Machine-readable
{
  "sha": "95bf98c120b54f3c54d39ae9607c1ef1d1ef2ade",
  "pr": 3114,
  "job": {
    "issue": 3090,
    "kind": "fix",
    "feature": "data-ingestion",
    "survey": "2026-10-24-5cf66803-dlib"
  },
  "blast": {
    "severity": "medium",
    "reasons": [
      {
        "severity": "medium",
        "what": "touches API handlers: services/api/routers/ingestion.py"
      }
    ],
    "features": [
      "data-ingestion"
    ],
    "layers": [
      "api",
      "tests"
    ],
    "routes": [],
    "safeBecause": "every whole-file read error, of any type, now lands in `_file_failed`, and the JSON retry ingests only what the stream had not.",
    "proofLevel": 4
  },
  "size": {
    "additions": 427,
    "deletions": 124
  },
  "verdicts": {
    "size": "n/a",
    "callers": "pass",
    "lint": "pass",
    "ci": "pass"
  },
  "callers": [
    {
      "name": "_ingest_json_full",
      "kind": "symbol",
      "from": "core/ingestion/ingestion_service.py",
      "refs": []
    }
  ],
  "lint": 0,
  "checks": [
    {
      "name": "Build Frontend",
      "bucket": "pass"
    },
    {
      "name": "MCP Runner Image",
      "bucket": "pass"
    },
    {
      "name": "Unit Tests - Frontend",
      "bucket": "pass"
    },
    {
      "name": "Lint Frontend",
      "bucket": "pass"
    },
    {
      "name": "Daemon Image",
      "bucket": "pass"
    },
    {
      "name": "Agent Image",
      "bucket": "pass"
    },
    {
      "name": "Helm Install Schema",
      "bucket": "pass"
    },
    {
      "name": "Unit Tests - Agent Layer",
      "bucket": "pass"
    },
    {
      "name": "Setup Path (ubuntu-latest)",
      "bucket": "pass"
    },
    {
      "name": "Security Scan - NPM",
      "bucket": "pass"
    },
    {
      "name": "Unit Tests - Backend (DB-backed)",
      "bucket": "pass"
    },
    {
      "name": "Unit Tests - Backend",
      "bucket": "pass"
    },
    {
      "name": "Lint Backend",
      "bucket": "pass"
    },
    {
      "name": "Security Scan - Python",
      "bucket": "pass"
    },
    {
      "name": "Integration Tests",
      "bucket": "pass"
    },
    {
      "name": "Generated API Types",
      "bucket": "pass"
    },
    {
      "name": "CodeQL",
      "bucket": "pass"
    },
    {
      "name": "Analyze (python)",
      "bucket": "pass"
    },
    {
      "name": "Analyze (actions)",
      "bucket": "pass"
    },
    {
      "name": "Backend Image",
      "bucket": "pass"
    },
    {
      "name": "Analyze (javascript-typescript)",
      "bucket": "pass"
    },
    {
      "name": "Setup Path (macos-latest)",
      "bucket": "pass"
    },
    {
      "name": "Lint Dockerfiles",
      "bucket": "pass"
    }
  ],
  "evidence": [
    {
      "name": "ingestion-and-callers-after.txt",
      "url": "https://github.com/Vigil-SOC/vigil/blob/deslop-evidence/pr-3114/ingestion-and-callers-after.txt"
    },
    {
      "name": "json-resume-probe.txt",
      "url": "https://github.com/Vigil-SOC/vigil/blob/deslop-evidence/pr-3114/json-resume-probe.txt"
    },
    {
      "name": "new-tests-on-base.txt",
      "url": "https://github.com/Vigil-SOC/vigil/blob/deslop-evidence/pr-3114/new-tests-on-base.txt"
    },
    {
      "name": "ratchets.txt",
      "url": "https://github.com/Vigil-SOC/vigil/blob/deslop-evidence/pr-3114/ratchets.txt"
    }
  ]
}

@samarmstrong
samarmstrong merged commit 487a4cd into main Oct 24, 2026
23 checks passed
@samarmstrong

Copy link
Copy Markdown
Collaborator

🏭 Factory review — merged.

An unreadable upload now fails the job and names the cause, and a file ijson rejects for NaN still imports each finding once. That is the bug in #3090, and this branch does it. origin/main merges cleanly. The two commits after this branch are the CrowdStrike lookback and the Darktrace poller removal, and neither touches these files. I did not need to change the code.

Ran

  • POST /api/ingest/upload, then GET /api/ingest/jobs/{id}, on the API at 127.0.0.1:6987:

    • Truncated 1500-finding JSON: failed, Could not read the JSON file: Unterminated string…, 1127 findings stored. Uploading that file again reported those 1127 as duplicates and failed with the same cause.
    • [1, 2, 3], a corrupt Parquet file, and a latin-1 CSV: failed, each message naming the cause (argument of type 'int' is not iterable, Parquet magic bytes not found, 'utf-8' codec can't decode byte 0xe9).
    • A short CSV and a JSONL file with a bad byte after three good rows: failed, and those three rows stored.
    • A 1500-finding JSON file with NaN at index 1200: succeeded, Imported 1500 findings, 1500 distinct nan-* rows in Postgres.
    • A well-formed JSON file and a CRLF CSV: succeeded. A case CSV with a quoted newline kept line one\nline two in the title.
    • A {findings, cases} file with NaN on the second case: both findings imported once, and the new case imported once.
  • Settings → Data & uploads at 1440×900 and 1920×1080. The trunc-1500.json row shows the parse error and the Failed pill. At both widths documentElement.scrollWidth equals clientWidth, and the job row and message fit their boxes.

  • pytest on test_unreadable_upload_files.py, test_ingestion_common_keys.py, and test_no_logger_fstrings.py: 25 passed. The f-string ratchet holds at 273.

Did not run

  • The CI suites. Those run on this commit.
  • /ingest-string and /s3-file live. /s3-file only passes first_error into the same summarize_stats the upload job already uses. /ingest-string still logs a parse error and returns stats with no first_error. That path is outside this fix.

@samarmstrong
samarmstrong deleted the cleanup/3090-report-unreadable-upload-files-as-3ed25304 branch October 24, 2026 23:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cleanup:blast-medium Blast radius: how bad it is if this PR is wrong cleanup Opened by the cleanup factory

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Report unreadable upload files as failed jobs with the parse error instead of "No data imported" or success

3 participants