Skip to content

Delete client-less POST /api/ingest/sync-s3-folder and the S3 code only it used - #3118

Merged
samarmstrong merged 3 commits into
mainfrom
cleanup/3091-delete-client-less-post-api-ingest-c8550611
Oct 24, 2026
Merged

samarmstrong merged 3 commits into
mainfrom
cleanup/3091-delete-client-less-post-api-ingest-c8550611

Conversation

@G-r-ay

@G-r-ay G-r-ay commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #3091

No client calls POST /api/ingest/sync-s3-folder. This PR deletes the route and the code that only it reached. The web app keeps using GET /api/ingest/s3-files and POST /api/ingest/s3-file, which are unchanged.

Removed:

  • the route and _FINDINGS_AND_CASES in services/api/routers/ingestion.py
  • IngestionService._S3_FORMAT_MAP, ingest_s3_folder, _s3_key_extension and the now unused import tempfile in core/ingestion/ingestion_service.py
  • S3Service.list_files and S3Service.test_connection in core/storage/s3_service.py
  • the two sync-s3-folder rows in tests/security/test_route_permissions.py and the sync test in tests/unit/storage/test_s3_service_list_errors.py

The list_files cases in that test file now run against list_files_detailed, so the "listing errors raise" coverage from #1577 stays. schema.d.ts is regenerated. The deleted code had 3 f-string logger calls, so the test_no_logger_fstrings baseline drops from 288 to 285.

Both bugs the issue names (stats from only the last file, doubled; default prefix read from the wrong level of the config) go away with the route.

Callers checked

  • grep -rnI "sync-s3\|sync_s3\|syncS3\|s3-folder\|s3_folder\|S3Folder\|ingest_s3_folder\|_S3_FORMAT_MAP\|_s3_key_extension\|_FINDINGS_AND_CASES" over the repo, without node_modules and .git. Hits were the deleted code, its two tests, and the generated schema.d.ts. No TS, script, doc, CLI or infra hit.
  • .list_files( in Python: ingestion_service.py (deleted) and the retargeted test. orchestrator.py:280 calls workdir.list_files, a different class.
  • services/api/routers/orchestrator.py:280 calls orch.workdir.list_files(investigation_id). orch.workdir is a WorkdirManager (services/daemon/orchestrator.py:684), and the method it calls is WorkdirManager.list_files at services/daemon/workdir.py:67. That is a different method that happens to share the name, not a caller of S3Service.list_files, so it is unaffected.
  • test_connection(: every hit outside s3_service.py is Splunk, Elastic, OpenSearch, vStrike or CrowdStrike. credential_check.py:100,128 call it on Elastic and OpenSearch services.
  • _ingest_file_by_format stays. core/ingestion/ingestion_jobs.py:170 still calls it.
  • S3_LIST_ERRORS, _s3_list_http_error, Query and summarize_stats stay. GET /s3-files and the other routes still use them.

Blast radius

  • Safe because: nothing in the repo sends a request to /api/ingest/sync-s3-folder or calls the deleted methods, and the S3 browse and import routes do not depend on them.
  • Tried to break it: if the fact were false, the grep above would show a caller, tsc would fail on a removed operation type, or GET /s3-files would change. The grep found no caller. npx tsc --noEmit in clients/web exited 0. A probe script built the real ingestion router with a stubbed S3 client and dumped the full app's OpenAPI spec, on main and on this branch. On main it printed POST sync-s3-folder 200 and spec has sync-s3-folder: True. On this branch it printed POST sync-s3-folder 404 and spec has sync-s3-folder: False. Both printed the same GET s3-files 200 body and s3-files: True | s3-file: True.
  • Proof: 5. The real API server (uvicorn services.api.main:app) ran from main and from this branch against a throwaway Postgres (schema from python -m core.upgrade) and a moto S3 server, and was driven over HTTP the way the web client does (CSRF cookie and header). Commands and output are under "Run in the app" below.
  • Severity: medium.
  • Risks:
    • An operator script outside this repo that posts to the route now gets 404 (services/api/routers/ingestion.py). Unlikely, since git history shows no client ever called it. Low impact, because /s3-file imports a single object.
    • Code outside the repo calling S3Service.list_files or test_connection would break (core/storage/s3_service.py:80). Unlikely, since S3Service is only built in ingestion.py:_get_s3_service and tests.
  • Cleared: the route was not in the /api/v1 contract. No DB column, stored data, feature flag or wire format changes. The IngestionStats model stays because other routes return it.

Run in the app

Setup: throwaway Postgres DB vigil_proof migrated with VIGIL_UPGRADE_ACK_VERSION=0.8.0 python -m core.upgrade; moto_server -p 47123 as S3 with bucket proof-bucket holding drop/findings.json (2 findings) and drop/readme.txt. Both servers ran with DEV_MODE=true VIGIL_COOKIE_SECURE=false AWS_ENDPOINT_URL=http://127.0.0.1:47123 AWS_ACCESS_KEY_ID=… AWS_SECRET_ACCESS_KEY=… and python -m uvicorn services.api.main:app --port <port>. Same migrations on both sides; findings truncated between runs.

Driver (drive.sh <base-url>):

#!/bin/bash
# Drives the ingestion API like the web client: CSRF cookie + header.
B=$1; J=$(mktemp -p $TMPDIR)
curl -s -c $J -b $J $B/api/health >/dev/null
T=$(awk '$6=="csrf_token"{print $7}' $J)
c(){ curl -s -b $J -c $J -H "X-CSRF-Token: $T" -H 'content-type: application/json' -w '  [HTTP %{http_code}]\n' "$@"; }
echo '$ POST /api/config/integrations (s3: bucket proof-bucket, parquet_prefix drop/)'
c -X POST $B/api/config/integrations -d '{"enabled_integrations":["s3"],"integrations":{"s3":{"bucket_name":"proof-bucket","region":"us-east-1","auth_method":"credentials","parquet_prefix":"drop/"}}}'
echo '$ GET /api/ingest/s3-files?prefix=drop/'
c "$B/api/ingest/s3-files?prefix=drop/"
echo '$ POST /api/ingest/sync-s3-folder?prefix=drop/'
c -X POST "$B/api/ingest/sync-s3-folder?prefix=drop/"
echo '$ POST /api/ingest/s3-file {"key":"drop/findings.json"}'
c -X POST "$B/api/ingest/s3-file" -d '{"key":"drop/findings.json"}'
echo '$ GET /openapi.json paths under /api/ingest'
curl -s $B/openapi.json | python3 -c 'import json,sys; [print("  ",m.upper(),p) for p,v in json.load(sys.stdin)["paths"].items() if "/ingest" in p for m in v]'
rm -f $J

Before, main (75ba31e), drive.sh http://127.0.0.1:46988:

$ POST /api/config/integrations (s3: bucket proof-bucket, parquet_prefix drop/)
{"success":true,"message":"Integrations configuration saved"}  [HTTP 200]
$ GET /api/ingest/s3-files?prefix=drop/
{"files":[{"key":"drop/findings.json","size":340,"last_modified":"2026-10-24T19:56:46+00:00"},{"key":"drop/readme.txt","size":1,"last_modified":"2026-10-24T19:56:46+00:00"}],"count":2,"prefix":"drop/"}  [HTTP 200]
$ POST /api/ingest/sync-s3-folder?prefix=drop/
{"findings_total":4,"findings_imported":4,"findings_skipped":0,"findings_errors":0,"cases_total":0,"cases_imported":0,"cases_skipped":0,"cases_errors":0,"success":true,"message":"Processed 1 file(s). Skipped 1 unsupported file(s). Imported 4 findings"}  [HTTP 200]
$ POST /api/ingest/s3-file {"key":"drop/findings.json"}
{"findings_total":2,"findings_imported":0,"findings_skipped":2,"findings_errors":0,"cases_total":0,"cases_imported":0,"cases_skipped":0,"cases_errors":0,"success":true,"message":"Skipped 2 duplicate findings"}  [HTTP 200]
$ GET /openapi.json paths under /api/ingest
   POST /api/ingest/upload
   GET /api/ingest/jobs
   GET /api/ingest/jobs/{job_id}
   POST /api/ingest/ingest-string
   POST /api/ingest/sync-s3-folder
   GET /api/ingest/s3-files
   POST /api/ingest/s3-file

select finding_id from findings afterwards: f-proof-0001, f-proof-0002. The deleted route reported 4 imported for 2 rows, the doubled-stats bug from #3091.

After, this branch (db7905d), drive.sh http://127.0.0.1:46987:

$ POST /api/config/integrations (s3: bucket proof-bucket, parquet_prefix drop/)
{"success":true,"message":"Integrations configuration saved"}  [HTTP 200]
$ GET /api/ingest/s3-files?prefix=drop/
{"files":[{"key":"drop/findings.json","size":340,"last_modified":"2026-10-24T19:56:46+00:00"},{"key":"drop/readme.txt","size":1,"last_modified":"2026-10-24T19:56:46+00:00"}],"count":2,"prefix":"drop/"}  [HTTP 200]
$ POST /api/ingest/sync-s3-folder?prefix=drop/
{"detail":"Not Found"}  [HTTP 404]
$ POST /api/ingest/s3-file {"key":"drop/findings.json"}
{"findings_total":2,"findings_imported":2,"findings_skipped":0,"findings_errors":0,"cases_total":0,"cases_imported":0,"cases_skipped":0,"cases_errors":0,"success":true,"message":"Imported 2 findings"}  [HTTP 200]
$ GET /openapi.json paths under /api/ingest
   POST /api/ingest/upload
   GET /api/ingest/jobs
   GET /api/ingest/jobs/{job_id}
   POST /api/ingest/ingest-string
   GET /api/ingest/s3-files
   POST /api/ingest/s3-file

select finding_id from findings afterwards: f-proof-0001, f-proof-0002. Backend log: "POST /api/ingest/sync-s3-folder?prefix=drop/ HTTP/1.1" 404 Not Found, "POST /api/ingest/s3-file HTTP/1.1" 200 OK.

The S3 browse (GET /s3-files) and single-file import (POST /s3-file) the web app uses behave the same; only sync-s3-folder is gone, from the routes and from the OpenAPI spec.

Also seen

S3Service.get_file returns None on any error and logs it, so POST /s3-file cannot tell a missing key from a denied read. That is outside this job.

Verification

Ran

  • pytest tests/unit/storage/test_s3_service_list_errors.py tests/security/test_route_permissions.py tests/unit/ingestion tests/unit/api/test_context_path.py tests/unit/_ratchets/test_no_logger_fstrings.py: 198 passed (pytest-related.txt).
  • pytest tests/unit/_ratchets: 260 passed, 6 skipped because helm is not installed (pytest-ratchets.txt). Before the baseline change it failed with "lower BASELINE in this file from 288 to 285".
  • flake8 on the changed Python files: clean (flake8.txt).
  • python scripts/generate_frontend_types.py, then npx tsc --noEmit in clients/web: exit 0 (tsc.txt).
  • The before and after probe described above (probe-before-after.txt).

Did not run

  • The full backend and frontend suites. CI runs them.
  • A browser check. No screen changes, since no client used the route.

🤖 Generated with Claude Code

…it used

Removes the route, IngestionService.ingest_s3_folder with _S3_FORMAT_MAP
and _s3_key_extension, S3Service.list_files and the uncalled
S3Service.test_connection. Retargets the S3 listing tests at
list_files_detailed, regenerates schema.d.ts and lowers the logger
f-string baseline to 285.

Closes #3091

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
G-r-ay pushed a commit that referenced this pull request Oct 24, 2026
@G-r-ay G-r-ay added cleanup Opened by the cleanup factory cleanup:callers Deletes something the repo still refers to labels Oct 24, 2026
@G-r-ay

G-r-ay commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator Author

🧹 This PR removes things the repo still refers to. Each is a broken caller or a different thing with the same name; check them before merging:

  • list_files (symbol, removed from core/storage/s3_service.py): still referenced at services/api/routers/orchestrator.py:280

@G-r-ay

G-r-ay commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator Author

🧹 Asked for proof in the running app, level 5, which a high or critical blast radius needs (1 of 3): a critical blast radius proven only to level 4.

deeptempo-factory and others added 2 commits October 24, 2026 21:06
The sync-s3-folder route this release deletes is the only thing the clause named.

Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
@G-r-ay

G-r-ay commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator Author

🏭 Advisory review (claude-haiku-5-5 + opus advisor) — merge f68e86a6d7c7. Advice only: nothing was merged or closed on it.

All four reviewers reported. Remote head is f68e86a6d7c7, matching local, tree clean.

Reviewers

Pushed

  • d89e06a5: merged origin/main in. Clean, no conflicts.
  • f68e86a6d7c7: removed the stale "S3 folder sync needs both" clause from CHANGELOG.md.

Ran

  • pytest on storage, ingestion, ratchet: 117 passed (py3.14 venv; repo pins 3.12).
  • pytest route permissions: 64 passed.
  • flake8 on changed files: clean.
  • Types regen vs committed schema.d.ts: identical.
  • Live before (75ba31ec) vs after (d89e06a5) on Postgres + moto, driven over HTTP.
  • No UI change, so no screenshots.

Did not run

  • Full backend and frontend suites (CI runs them).
  • tsc (no node_modules).
  • Real auth in live app (DEV_MODE). Before-side POST without CSRF header returned 200, not investigated.

@G-r-ay
G-r-ay marked this pull request as ready for review October 24, 2026 20:28
@G-r-ay

G-r-ay commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator Author

🧹 Proof at f68e86a6d7c7. Counted by the cleanup factory at this commit; a push or a settled CI run replaces it. Each count says how to reproduce it.

Job: #3091, a remove job in feature data-ingestion, from survey 2026-10-24-5cf66803-dlib.

Check Result
Blast radius critical: if wrong, data is lost or corrupted, access control weakens, or the service goes down
Size pass (+17 −302)
Callers of what it deletes fail
Style lint pass
CI pass
Ready for review no: callers fail

Blast radius: critical. Touches 8 file(s) in the data, security, contract, api, ui, tests, docs layer(s), across feature(s) data-ingestion, users-admin, auth-middleware, api-platform, storage-layer. Why:

  • critical: touches the storage layer: core/storage/s3_service.py, tests/unit/storage/test_s3_service_list_errors.py
  • critical: touches auth, permissions, or secrets: tests/security/test_route_permissions.py
  • high: touches a published API contract or generated client: clients/web/src/services/generated/schema.d.ts
  • high: spans 5 features: data-ingestion, users-admin, auth-middleware, api-platform, storage-layer
  • high: deletes what the repo still references: list_files
  • medium: touches API handlers: services/api/routers/ingestion.py
  • medium: touches what users see: clients/web/src/services/generated/schema.d.ts

The author says it is safe because: nothing in the repo sends a request to /api/ingest/sync-s3-folder or calls the deleted methods, and the S3 browse and import routes do not depend on them. (proof level 5 of 5, a claim to check, not a count).

How each count was made, and how to rerun it

Size. +17 −302 over the whole diff, as GitHub counts it. A remove job must delete more than it adds. Reproduce with gh pr view 3118 -R Vigil-SOC/vigil --json additions,deletions.

Callers. What the PR deletes and does not define again, searched for across the repo at f68e86a6d7c7. A symbol is matched as a whole word, and a route by the fixed part of its path.

  • ingest_s3_folder (symbol, from core/ingestion/ingestion_service.py): 0 references
  • _s3_key_extension (symbol, from core/ingestion/ingestion_service.py): 0 references
  • list_files (symbol, from core/storage/s3_service.py): still referenced at services/api/routers/orchestrator.py:280
  • sync_s3_folder (symbol, from services/api/routers/ingestion.py): 0 references
git fetch origin pull/3118/head
git grep -n -I -F -w -e 'ingest_s3_folder' f68e86a6d7c7
git grep -n -I -F -w -e '_s3_key_extension' f68e86a6d7c7
git grep -n -I -F -w -e 'list_files' f68e86a6d7c7
git grep -n -I -F -w -e 'sync_s3_folder' f68e86a6d7c7

Style lint. No findings in the prose the PR adds.

CI at f68e86a6d7c7. 23 of 23 pass. Reproduce with gh pr checks 3118 -R Vigil-SOC/vigil.

Evidence. flake8.txt · probe-before-after.txt · pytest-ratchets.txt · pytest-related.txt · tsc.txt

Machine-readable
{
  "sha": "f68e86a6d7c7ed3a3dd793925f79900707774861",
  "pr": 3118,
  "job": {
    "issue": 3091,
    "kind": "remove",
    "feature": "data-ingestion",
    "survey": "2026-10-24-5cf66803-dlib"
  },
  "blast": {
    "severity": "critical",
    "reasons": [
      {
        "severity": "critical",
        "what": "touches the storage layer: core/storage/s3_service.py, tests/unit/storage/test_s3_service_list_errors.py"
      },
      {
        "severity": "critical",
        "what": "touches auth, permissions, or secrets: tests/security/test_route_permissions.py"
      },
      {
        "severity": "high",
        "what": "touches a published API contract or generated client: clients/web/src/services/generated/schema.d.ts"
      },
      {
        "severity": "high",
        "what": "spans 5 features: data-ingestion, users-admin, auth-middleware, api-platform, storage-layer"
      },
      {
        "severity": "high",
        "what": "deletes what the repo still references: list_files"
      },
      {
        "severity": "medium",
        "what": "touches API handlers: services/api/routers/ingestion.py"
      },
      {
        "severity": "medium",
        "what": "touches what users see: clients/web/src/services/generated/schema.d.ts"
      }
    ],
    "features": [
      "data-ingestion",
      "users-admin",
      "auth-middleware",
      "api-platform",
      "storage-layer"
    ],
    "layers": [
      "data",
      "security",
      "contract",
      "api",
      "ui",
      "tests",
      "docs"
    ],
    "routes": [],
    "safeBecause": "nothing in the repo sends a request to `/api/ingest/sync-s3-folder` or calls the deleted methods, and the S3 browse and import routes do not depend on them.",
    "proofLevel": 5
  },
  "size": {
    "additions": 17,
    "deletions": 302
  },
  "verdicts": {
    "size": "pass",
    "callers": "fail",
    "lint": "pass",
    "ci": "pass"
  },
  "callers": [
    {
      "name": "ingest_s3_folder",
      "kind": "symbol",
      "from": "core/ingestion/ingestion_service.py",
      "refs": []
    },
    {
      "name": "_s3_key_extension",
      "kind": "symbol",
      "from": "core/ingestion/ingestion_service.py",
      "refs": []
    },
    {
      "name": "list_files",
      "kind": "symbol",
      "from": "core/storage/s3_service.py",
      "refs": [
        "services/api/routers/orchestrator.py:280"
      ]
    },
    {
      "name": "sync_s3_folder",
      "kind": "symbol",
      "from": "services/api/routers/ingestion.py",
      "refs": []
    }
  ],
  "lint": 0,
  "checks": [
    {
      "name": "Build Frontend",
      "bucket": "pass"
    },
    {
      "name": "Setup Path (ubuntu-latest)",
      "bucket": "pass"
    },
    {
      "name": "Unit Tests - Backend (DB-backed)",
      "bucket": "pass"
    },
    {
      "name": "Backend Image",
      "bucket": "pass"
    },
    {
      "name": "Security Scan - NPM",
      "bucket": "pass"
    },
    {
      "name": "Helm Install Schema",
      "bucket": "pass"
    },
    {
      "name": "Lint Frontend",
      "bucket": "pass"
    },
    {
      "name": "Integration Tests",
      "bucket": "pass"
    },
    {
      "name": "Agent Image",
      "bucket": "pass"
    },
    {
      "name": "Lint Dockerfiles",
      "bucket": "pass"
    },
    {
      "name": "Unit Tests - Backend",
      "bucket": "pass"
    },
    {
      "name": "Unit Tests - Agent Layer",
      "bucket": "pass"
    },
    {
      "name": "Lint Backend",
      "bucket": "pass"
    },
    {
      "name": "Unit Tests - Frontend",
      "bucket": "pass"
    },
    {
      "name": "Security Scan - Python",
      "bucket": "pass"
    },
    {
      "name": "Daemon Image",
      "bucket": "pass"
    },
    {
      "name": "Analyze (python)",
      "bucket": "pass"
    },
    {
      "name": "Analyze (javascript-typescript)",
      "bucket": "pass"
    },
    {
      "name": "Generated API Types",
      "bucket": "pass"
    },
    {
      "name": "CodeQL",
      "bucket": "pass"
    },
    {
      "name": "Setup Path (macos-latest)",
      "bucket": "pass"
    },
    {
      "name": "MCP Runner Image",
      "bucket": "pass"
    },
    {
      "name": "Analyze (actions)",
      "bucket": "pass"
    }
  ],
  "evidence": [
    {
      "name": "flake8.txt",
      "url": "https://github.com/Vigil-SOC/vigil/blob/deslop-evidence/pr-3118/flake8.txt"
    },
    {
      "name": "probe-before-after.txt",
      "url": "https://github.com/Vigil-SOC/vigil/blob/deslop-evidence/pr-3118/probe-before-after.txt"
    },
    {
      "name": "pytest-ratchets.txt",
      "url": "https://github.com/Vigil-SOC/vigil/blob/deslop-evidence/pr-3118/pytest-ratchets.txt"
    },
    {
      "name": "pytest-related.txt",
      "url": "https://github.com/Vigil-SOC/vigil/blob/deslop-evidence/pr-3118/pytest-related.txt"
    },
    {
      "name": "tsc.txt",
      "url": "https://github.com/Vigil-SOC/vigil/blob/deslop-evidence/pr-3118/tsc.txt"
    }
  ]
}

@samarmstrong
samarmstrong merged commit 60d8ad8 into main Oct 24, 2026
23 checks passed
@samarmstrong

Copy link
Copy Markdown
Collaborator

🏭 Factory review — merged.

POST /api/ingest/sync-s3-folder is unused, and the code that only it reached is gone. The console still lists and imports through GET /api/ingest/s3-files and POST /api/ingest/s3-file (clients/web/src/services/api.ts). The list_files hit at services/api/routers/orchestrator.py:280 is WorkdirManager.list_files, not S3Service.list_files.

Ran

  • Live API (uvicorn services.api.main:app) against Postgres and MinIO, with a bucket holding drop/findings.json (2 findings) and drop/readme.txt:
    • POST /api/config/s3 returned 200.
    • GET /api/ingest/s3-files?prefix=drop/ returned 200 and both objects.
    • POST /api/ingest/sync-s3-folder?prefix=drop/ returned 404.
    • POST /api/ingest/s3-file for drop/findings.json returned 200 and imported both findings.
    • OpenAPI still lists s3-files and s3-file, and no longer lists sync-s3-folder.

Did not run

  • The repo test suites. CI on this commit is the check for those.

No further changes. main is two commits ahead and merges cleanly; those commits do not touch this deletion.

@samarmstrong
samarmstrong deleted the cleanup/3091-delete-client-less-post-api-ingest-c8550611 branch October 24, 2026 21:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cleanup:blast-critical Blast radius: how bad it is if this PR is wrong cleanup:callers Deletes something the repo still refers to cleanup Opened by the cleanup factory

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Delete client-less POST /api/ingest/sync-s3-folder, ingest_s3_folder and the S3Service methods only it uses

2 participants