Skip to content

Expose ingestion readiness and targeted embedding repair #2336

Description

@JSv4

Worker-upload COMPLETED means the import transaction committed. Annotation fallback, document embedding, and thumbnails may still be asynchronous. The existing verification command correctly reports that boundary, but no consolidated server contract establishes embedding/search readiness.

Implementation

  • Add an authorized, bounded status API for document/corpus readiness, separate from upload receipt status. Report required stages, outstanding/failed work, diagnostic reasons, and the processing/configuration generation being assessed.
  • Report eligible versus valid document/annotation embedding coverage for the effective embedder/model identity and dimension. Respect shared structural annotation sets and intentional empty-text exclusions; do not infer readiness solely from backend_lock, preferred_embedder, or an embedding row's existence.
  • Reuse CorpusService.annotations_in_corpus, count_annotations_missing_embeddings, and existing embedding tasks. Distinguish required search stages from optional artifacts such as thumbnails.
  • Add an idempotent, bounded repair operation for missing/invalid embeddings, with observable progress and failures. Preserve valid vectors and parsed artifacts; do not require full document reprocessing.
  • Reconcile corpus-copy processing state with the completed worker import. Corpus.add_document currently suppresses parsing and unlocks the copy without copying its completed processing status.
  • Extend CLI verification with an explicit readiness mode while retaining the existing receipt-only contract.

Acceptance criteria

  • Tests distinguish transaction completion from asynchronous readiness and cover partial/failed batches, empty annotations, shared structural sets, wrong model/dimension, unavailable status, and configuration changes.
  • Repeated/concurrent repair requests do not duplicate work or reparse documents; readiness changes only when all required checks pass.
  • Direct and corpus-linked documents expose consistent terminal processing state.

Follow-up to #2320, which explicitly leaves asynchronous readiness outside receipt verification.

Code pointers

worker processing; corpus copy creation; embedding coverage helper; CLI verification.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions