Remote ingest: checkpoint preparation and detect source or configuration changes

[JSv4]

Remote ingestion should reuse valid local preparation work after a restart or failed upload, while ensuring that every uploaded source document matches the artifacts prepared from it.

The current ledger is resumable only at the document-upload boundary. It records PENDING, UPLOADED, COMPLETED, FAILED, and PARKED, and run selects only PENDING/FAILED rows. Parsing, enrichment, text reconstruction, and embedding remain in memory until POST /api/worker-uploads/documents/ returns successfully. A crash before that point, or a later retry after server failure, repeats all preparation.

Source identity is also recorded but not enforced. Ledger.upsert_doc() uses ON CONFLICT(rel_path) DO NOTHING, so replanning changed bytes at the same relative path leaves the old absolute path, size, hash, status, and receipt in place. _process_one() reads the current path without checking it against the recorded hash. It then parses bytes read into memory but reopens the path during upload, so a file changed between those operations can pair artifacts from the old bytes with the new source document.

Proposed scope

• Store preparation artifacts and a small manifest in the existing durable /ledger volume. Do not introduce a separate storage service. Use atomic writes and content digests so a stage is marked reusable only after its artifact is durable and verifiably complete.
• Define explicit checkpoint boundaries for parsing/text reconstruction, enrichment, and embeddings (or a simpler boundary if it provides the same restart behavior). Key each stage by its input artifact digest plus the effective settings that can change its output.
• Fingerprint the effective parser implementation/settings, ordered enricher identities and configuration, and embedding identity/dimension. If the parser or embedding service cannot report a model/version, support an explicit stable operator-supplied identity rather than requiring automatic discovery.
• Preserve the complete parsed/enriched export across stages. Incoming annotation IDs are correlation keys used to rebuild parent links, relationships, and the annotation-embedding map on the server; cached stages must preserve those correlation keys consistently across the export and its dependent artifacts. Validate required IDs and uniqueness, including collisions after JSON stringification, before treating dependent artifacts as reusable.
• Reconcile a planned source during replanning and immediately before preparation/upload. The bytes uploaded must have the same digest as the bytes used by the reusable preparation chain, either by uploading a stable snapshot or by detecting a concurrent change and restarting preparation.
• For a same-path content change that has not been accepted by the server, invalidate the affected artifacts, clear stale receipt/error state, and reset retry exhaustion for the new source version. Preserve valid earlier stages when only a downstream configuration changes.
• For a changed source whose prior version is UPLOADED or COMPLETED, retain the prior receipt and surface a conflict that requires an explicit replace/new-document policy. Do not silently treat the receipt as current or automatically create another document.
• Treat missing, corrupt, or incompatible artifacts as cache misses and recompute the affected stage and its dependents. Provide retention/cleanup rules that do not remove artifacts referenced by active ledger rows, and do not persist credentials in artifacts or manifests.
• Migrate existing ledgers in place or document a compatible one-time migration. Existing rows without artifact metadata should remain runnable as uncached work.

Lifecycle boundary

These checkpoints recover local preparation only. They do not make upload replay idempotent. The upload client retries a POST without an idempotency key, while the server creates a fresh corpus document for each accepted upload. If the server accepts an upload but the response is lost, the client cannot determine locally whether replay is safe. Keep an ambiguous upload receipt/state separate from preparation state and address server-backed request idempotency independently.

Worker-upload receipts are also scoped to the exact CorpusAccessToken used to create them. A replacement token cannot read an old receipt even when it targets the same corpus. Handling token rotation for outstanding receipts is a separate API/design decision; a local checkpoint should not claim to solve it.

Acceptance criteria

• After a process restart or upload failure following successful parsing, enrichment, or embedding, run reuses every valid completed stage. Tests assert stage invocation counts and final payload equality across each failure boundary.
• Replanning changed bytes at the same relative path detects the change. A changed source cannot inherit COMPLETED, UPLOADED, or PARKED as though it were the same version.
• A file changed between initial read, checkpoint reuse, and upload is never uploaded with artifacts derived from different bytes.
• Parser, enricher, or embedding identity/configuration changes deterministically invalidate only that stage and its dependents.
• Missing files, truncated/modified artifacts, digest mismatches, interrupted writes, and legacy ledger rows recover predictably without uploading partial data.
• Annotation IDs, parent references, relationship endpoints, and embedding keys remain consistent when artifacts are resumed from different stage boundaries.
• Tests cover an accepted-but-response-lost upload as an explicit ambiguous state and verify that local artifact reuse does not present POST replay as idempotent.

Related: #2316 defines parser/format identity and normalization, #2320 defines embedding completeness before a stage can be reusable, and #2317 keeps ledger/cache traversal bounded. Reuse the existing enrichment overlay and annotation-ID contract rather than inventing a parallel metadata format.

Source references at the audited revision:

• Ledger schema, selection, and state transitions
• Planning conflict behavior
• Preparation and upload sequence
• Effective remote parser settings
• Annotation import ID mapping
• Exact-token receipt scoping
• Server creates a new corpus document for each add
• Existing durable ledger volume

Existing regression coverage to retain/adapt: enricher ID assignment and overlay preservation, parent/duplicate-ID validation, and bulk-ledger transition/retry tests. These do not currently test remote CLI stage checkpoints; add focused local tests for that boundary.