Skip to content

Add run-scoped ingestion policy and budget admission #2340

Description

@JSv4

Remote-ingest admission currently limits queue backlog, and local checkpoints fingerprint preparation settings. Neither provides a server-enforced run policy or expenditure ceiling covering asynchronous processing and retries.

Implementation

  • Introduce a durable ingestion-run identity with an immutable, secret-free policy snapshot: permitted processing stages/providers, parser/embedder/model identities and dimensions, allowed fallback behavior, and a configurable monetary ceiling with a versioned pricing/estimation basis.
  • Bind uploads and resulting asynchronous work to the run. Validate the effective configuration at execution as well as admission; a global settings change must not silently select a different or unapproved provider.
  • Reserve a conservative cost bound atomically before chargeable work is admitted. Include queued/in-flight work, retries, and downstream actions; reconcile reservations against accounted usage without double charging on task redelivery.
  • When the next operation cannot fit the remaining budget, stop admitting it and expose a resumable budget-exhausted state. Unknown/unbounded costs must not be treated as zero; define how external worker preparation is reserved/reported or explicitly excluded from the guaranteed coverage.
  • Expose authorized run totals, reservations, remaining allowance, policy violations, and pause/resume status through the API and CLI. Keep credentials out of snapshots and reports.
  • Reuse pipeline resolution, checkpoint identities, and queue admission. Document exactly which costs are bounded; do not present estimates as a guarantee of an entire infrastructure bill.

Acceptance criteria

  • Concurrent admissions cannot exceed the configured ceiling through competing reservations.
  • Tests cover exact-boundary rejection, retries/redelivery, crashes with outstanding reservations, cancellation, unknown pricing, configuration changes, and prohibited fallback.
  • Budget exhaustion preserves receipts/checkpoints, and status separates accounted usage from reservations and estimates.
  • No downstream chargeable stage escapes the declared run policy.

Related: #2318 provides preparation identities; #2319 provides backlog admission, which remains a separate limit.

Code pointers

admission governor; checkpoints; worker processing; pipeline settings API; document-triggered work.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions