Repository navigation
Delete client-less POST /api/ingest/sync-s3-folder and the S3 code only it used - #3118
Conversation
…it used Removes the route, IngestionService.ingest_s3_folder with _S3_FORMAT_MAP and _s3_key_extension, S3Service.list_files and the uncalled S3Service.test_connection. Retargets the S3 listing tests at list_files_detailed, regenerates schema.d.ts and lowers the logger f-string baseline to 285. Closes #3091 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
🧹 This PR removes things the repo still refers to. Each is a broken caller or a different thing with the same name; check them before merging:
|
|
🧹 Asked for proof in the running app, level 5, which a high or critical blast radius needs (1 of 3): a critical blast radius proven only to level 4. |
…lient-less-post-api-ingest-c8550611
The sync-s3-folder route this release deletes is the only thing the clause named. Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
|
🏭 Advisory review (claude-haiku-5-5 + opus advisor) — merge All four reviewers reported. Remote head is Reviewers
Pushed
Ran
Did not run
|
|
🧹 Proof at Job: #3091, a remove job in feature
Blast radius: critical. Touches 8 file(s) in the data, security, contract, api, ui, tests, docs layer(s), across feature(s)
The author says it is safe because: nothing in the repo sends a request to How each count was made, and how to rerun itSize. +17 −302 over the whole diff, as GitHub counts it. A remove job must delete more than it adds. Reproduce with Callers. What the PR deletes and does not define again, searched for across the repo at
git fetch origin pull/3118/head
git grep -n -I -F -w -e 'ingest_s3_folder' f68e86a6d7c7
git grep -n -I -F -w -e '_s3_key_extension' f68e86a6d7c7
git grep -n -I -F -w -e 'list_files' f68e86a6d7c7
git grep -n -I -F -w -e 'sync_s3_folder' f68e86a6d7c7Style lint. No findings in the prose the PR adds. CI at Evidence. flake8.txt · probe-before-after.txt · pytest-ratchets.txt · pytest-related.txt · tsc.txt Machine-readable{
"sha": "f68e86a6d7c7ed3a3dd793925f79900707774861",
"pr": 3118,
"job": {
"issue": 3091,
"kind": "remove",
"feature": "data-ingestion",
"survey": "2026-10-24-5cf66803-dlib"
},
"blast": {
"severity": "critical",
"reasons": [
{
"severity": "critical",
"what": "touches the storage layer: core/storage/s3_service.py, tests/unit/storage/test_s3_service_list_errors.py"
},
{
"severity": "critical",
"what": "touches auth, permissions, or secrets: tests/security/test_route_permissions.py"
},
{
"severity": "high",
"what": "touches a published API contract or generated client: clients/web/src/services/generated/schema.d.ts"
},
{
"severity": "high",
"what": "spans 5 features: data-ingestion, users-admin, auth-middleware, api-platform, storage-layer"
},
{
"severity": "high",
"what": "deletes what the repo still references: list_files"
},
{
"severity": "medium",
"what": "touches API handlers: services/api/routers/ingestion.py"
},
{
"severity": "medium",
"what": "touches what users see: clients/web/src/services/generated/schema.d.ts"
}
],
"features": [
"data-ingestion",
"users-admin",
"auth-middleware",
"api-platform",
"storage-layer"
],
"layers": [
"data",
"security",
"contract",
"api",
"ui",
"tests",
"docs"
],
"routes": [],
"safeBecause": "nothing in the repo sends a request to `/api/ingest/sync-s3-folder` or calls the deleted methods, and the S3 browse and import routes do not depend on them.",
"proofLevel": 5
},
"size": {
"additions": 17,
"deletions": 302
},
"verdicts": {
"size": "pass",
"callers": "fail",
"lint": "pass",
"ci": "pass"
},
"callers": [
{
"name": "ingest_s3_folder",
"kind": "symbol",
"from": "core/ingestion/ingestion_service.py",
"refs": []
},
{
"name": "_s3_key_extension",
"kind": "symbol",
"from": "core/ingestion/ingestion_service.py",
"refs": []
},
{
"name": "list_files",
"kind": "symbol",
"from": "core/storage/s3_service.py",
"refs": [
"services/api/routers/orchestrator.py:280"
]
},
{
"name": "sync_s3_folder",
"kind": "symbol",
"from": "services/api/routers/ingestion.py",
"refs": []
}
],
"lint": 0,
"checks": [
{
"name": "Build Frontend",
"bucket": "pass"
},
{
"name": "Setup Path (ubuntu-latest)",
"bucket": "pass"
},
{
"name": "Unit Tests - Backend (DB-backed)",
"bucket": "pass"
},
{
"name": "Backend Image",
"bucket": "pass"
},
{
"name": "Security Scan - NPM",
"bucket": "pass"
},
{
"name": "Helm Install Schema",
"bucket": "pass"
},
{
"name": "Lint Frontend",
"bucket": "pass"
},
{
"name": "Integration Tests",
"bucket": "pass"
},
{
"name": "Agent Image",
"bucket": "pass"
},
{
"name": "Lint Dockerfiles",
"bucket": "pass"
},
{
"name": "Unit Tests - Backend",
"bucket": "pass"
},
{
"name": "Unit Tests - Agent Layer",
"bucket": "pass"
},
{
"name": "Lint Backend",
"bucket": "pass"
},
{
"name": "Unit Tests - Frontend",
"bucket": "pass"
},
{
"name": "Security Scan - Python",
"bucket": "pass"
},
{
"name": "Daemon Image",
"bucket": "pass"
},
{
"name": "Analyze (python)",
"bucket": "pass"
},
{
"name": "Analyze (javascript-typescript)",
"bucket": "pass"
},
{
"name": "Generated API Types",
"bucket": "pass"
},
{
"name": "CodeQL",
"bucket": "pass"
},
{
"name": "Setup Path (macos-latest)",
"bucket": "pass"
},
{
"name": "MCP Runner Image",
"bucket": "pass"
},
{
"name": "Analyze (actions)",
"bucket": "pass"
}
],
"evidence": [
{
"name": "flake8.txt",
"url": "https://github.com/Vigil-SOC/vigil/blob/deslop-evidence/pr-3118/flake8.txt"
},
{
"name": "probe-before-after.txt",
"url": "https://github.com/Vigil-SOC/vigil/blob/deslop-evidence/pr-3118/probe-before-after.txt"
},
{
"name": "pytest-ratchets.txt",
"url": "https://github.com/Vigil-SOC/vigil/blob/deslop-evidence/pr-3118/pytest-ratchets.txt"
},
{
"name": "pytest-related.txt",
"url": "https://github.com/Vigil-SOC/vigil/blob/deslop-evidence/pr-3118/pytest-related.txt"
},
{
"name": "tsc.txt",
"url": "https://github.com/Vigil-SOC/vigil/blob/deslop-evidence/pr-3118/tsc.txt"
}
]
} |
|
🏭 Factory review — merged.
Ran
Did not run
No further changes. |
Closes #3091
No client calls
POST /api/ingest/sync-s3-folder. This PR deletes the route and the code that only it reached. The web app keeps usingGET /api/ingest/s3-filesandPOST /api/ingest/s3-file, which are unchanged.Removed:
_FINDINGS_AND_CASESinservices/api/routers/ingestion.pyIngestionService._S3_FORMAT_MAP,ingest_s3_folder,_s3_key_extensionand the now unusedimport tempfileincore/ingestion/ingestion_service.pyS3Service.list_filesandS3Service.test_connectionincore/storage/s3_service.pysync-s3-folderrows intests/security/test_route_permissions.pyand the sync test intests/unit/storage/test_s3_service_list_errors.pyThe
list_filescases in that test file now run againstlist_files_detailed, so the "listing errors raise" coverage from #1577 stays.schema.d.tsis regenerated. The deleted code had 3 f-string logger calls, so thetest_no_logger_fstringsbaseline drops from 288 to 285.Both bugs the issue names (stats from only the last file, doubled; default prefix read from the wrong level of the config) go away with the route.
Callers checked
grep -rnI "sync-s3\|sync_s3\|syncS3\|s3-folder\|s3_folder\|S3Folder\|ingest_s3_folder\|_S3_FORMAT_MAP\|_s3_key_extension\|_FINDINGS_AND_CASES"over the repo, without node_modules and .git. Hits were the deleted code, its two tests, and the generatedschema.d.ts. No TS, script, doc, CLI or infra hit..list_files(in Python:ingestion_service.py(deleted) and the retargeted test.orchestrator.py:280callsworkdir.list_files, a different class.services/api/routers/orchestrator.py:280callsorch.workdir.list_files(investigation_id).orch.workdiris aWorkdirManager(services/daemon/orchestrator.py:684), and the method it calls isWorkdirManager.list_filesatservices/daemon/workdir.py:67. That is a different method that happens to share the name, not a caller ofS3Service.list_files, so it is unaffected.test_connection(: every hit outsides3_service.pyis Splunk, Elastic, OpenSearch, vStrike or CrowdStrike.credential_check.py:100,128call it on Elastic and OpenSearch services._ingest_file_by_formatstays.core/ingestion/ingestion_jobs.py:170still calls it.S3_LIST_ERRORS,_s3_list_http_error,Queryandsummarize_statsstay.GET /s3-filesand the other routes still use them.Blast radius
/api/ingest/sync-s3-folderor calls the deleted methods, and the S3 browse and import routes do not depend on them.tscwould fail on a removed operation type, orGET /s3-fileswould change. The grep found no caller.npx tsc --noEmitinclients/webexited 0. A probe script built the real ingestion router with a stubbed S3 client and dumped the full app's OpenAPI spec, onmainand on this branch. Onmainit printedPOST sync-s3-folder 200andspec has sync-s3-folder: True. On this branch it printedPOST sync-s3-folder 404andspec has sync-s3-folder: False. Both printed the sameGET s3-files 200body ands3-files: True | s3-file: True.uvicorn services.api.main:app) ran frommainand from this branch against a throwaway Postgres (schema frompython -m core.upgrade) and a moto S3 server, and was driven over HTTP the way the web client does (CSRF cookie and header). Commands and output are under "Run in the app" below.medium.services/api/routers/ingestion.py). Unlikely, since git history shows no client ever called it. Low impact, because/s3-fileimports a single object.S3Service.list_filesortest_connectionwould break (core/storage/s3_service.py:80). Unlikely, sinceS3Serviceis only built iningestion.py:_get_s3_serviceand tests./api/v1contract. No DB column, stored data, feature flag or wire format changes. TheIngestionStatsmodel stays because other routes return it.Run in the app
Setup: throwaway Postgres DB
vigil_proofmigrated withVIGIL_UPGRADE_ACK_VERSION=0.8.0 python -m core.upgrade;moto_server -p 47123as S3 with bucketproof-bucketholdingdrop/findings.json(2 findings) anddrop/readme.txt. Both servers ran withDEV_MODE=true VIGIL_COOKIE_SECURE=false AWS_ENDPOINT_URL=http://127.0.0.1:47123 AWS_ACCESS_KEY_ID=… AWS_SECRET_ACCESS_KEY=…andpython -m uvicorn services.api.main:app --port <port>. Same migrations on both sides;findingstruncated between runs.Driver (
drive.sh <base-url>):Before,
main(75ba31e),drive.sh http://127.0.0.1:46988:select finding_id from findingsafterwards:f-proof-0001,f-proof-0002. The deleted route reported 4 imported for 2 rows, the doubled-stats bug from #3091.After, this branch (db7905d),
drive.sh http://127.0.0.1:46987:select finding_id from findingsafterwards:f-proof-0001,f-proof-0002. Backend log:"POST /api/ingest/sync-s3-folder?prefix=drop/ HTTP/1.1" 404 Not Found,"POST /api/ingest/s3-file HTTP/1.1" 200 OK.The S3 browse (
GET /s3-files) and single-file import (POST /s3-file) the web app uses behave the same; onlysync-s3-folderis gone, from the routes and from the OpenAPI spec.Also seen
S3Service.get_filereturnsNoneon any error and logs it, soPOST /s3-filecannot tell a missing key from a denied read. That is outside this job.Verification
Ran
pytest tests/unit/storage/test_s3_service_list_errors.py tests/security/test_route_permissions.py tests/unit/ingestion tests/unit/api/test_context_path.py tests/unit/_ratchets/test_no_logger_fstrings.py: 198 passed (pytest-related.txt).pytest tests/unit/_ratchets: 260 passed, 6 skipped because helm is not installed (pytest-ratchets.txt). Before the baseline change it failed with "lower BASELINE in this file from 288 to 285".flake8on the changed Python files: clean (flake8.txt).python scripts/generate_frontend_types.py, thennpx tsc --noEmitinclients/web: exit 0 (tsc.txt).probe-before-after.txt).Did not run
🤖 Generated with Claude Code