Skip to content

Add server-backed idempotency and recovery for worker uploads #2337

Description

@JSv4

The worker-upload endpoint creates a new staging row for every accepted POST. The CLI now preserves AMBIGUOUS outcomes rather than replaying uncertain requests, but cannot reconcile an accepted upload whose response was lost. Receipt access is also tied to the exact token that submitted it.

Implementation

  • Accept a stable client idempotency key, persisted before submission in the CLI ledger. Scope it to an authorized worker/corpus identity and bind it to a digest of source bytes plus semantically relevant metadata/configuration.
  • Atomically return the existing receipt for the same key and payload, including concurrent requests. Reject reuse with a different payload using a stable conflict response. Preserve documented behavior for legacy requests without keys.
  • Provide authorized receipt lookup by client key. Define lookup/rotation policy so replacement credentials can resume authorized work without exposing another worker's receipts; revoked credentials remain invalid.
  • Add bounded retry/recovery operations for failed or abandoned processing attempts, preserving receipt identity, error history, and valid preparation artifacts.
  • Make server processing safe under task redelivery and stale recovery. Use per-upload ownership/locking or a fenced lease so recovery cannot race a live processor into creating duplicate corpus documents.
  • Teach the CLI to reconcile ambiguous outcomes before retrying. Never interpret an unavailable lookup as proof that the original upload was rejected.

Acceptance criteria

  • Concurrent identical POSTs and an accepted-but-response-lost retry produce one logical upload and one corpus-linked document.
  • Tests cover conflicting payloads, crashes before/after commit, task redelivery, stale recovery, retry exhaustion, token rotation, and authorization boundaries.
  • Receipt reconciliation survives CLI restarts and does not repeat successful preparation.

Builds on #2318's durable checkpoints and explicit ambiguity state; local checkpointing alone cannot provide server idempotency.

Code pointers

upload views; receipt models; batch processing and recovery; CLI upload client.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions