Skip to content

Add token budgets for validation retries - #70

Open
h0rv wants to merge 4 commits into
mainfrom
feat/token-budget
Open

h0rv wants to merge 4 commits into
mainfrom
feat/token-budget

Conversation

@h0rv

@h0rv h0rv commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #69.

Changes

Users can set WithTokenBudget(n) to limit tokens spent on retries after JSON parsing or validation fails.

  • The first request runs normally. Successful results are returned even if total usage goes over budget.
  • The client stops retries when the budget is reached or usage is missing or incomplete.
  • Callers can inspect available usage through TokenBudgetError and get the last error with errors.Unwrap.
  • Retry limits and context deadlines still apply.

The option applies to regular and union requests across all four providers. Zero and negative budgets are rejected before a request.

Streaming calls with a budget are rejected before sending a request. Callers can check errors.Is(err, instructor.ErrTokenBudgetStreaming). Streaming without a budget still works. Canceled or expired contexts prevent the stream request, and provider errors are returned unchanged.

I also fixed Gemini usage totals so they include failed attempts and the successful attempt. Dependencies are unchanged.

Checks

  • The full test suite passed.
  • Package tests passed with the race detector.
  • I checked the public API through the OpenAI SDK against a local HTTP server. Retries stopped at the budget and when usage was missing. A successful retry still returned its result after total usage went over budget.
  • I reproduced streaming requests ignoring budgets before the fix. After the fix, positive and invalid budgets made no provider requests. Streaming without a budget returned the expected object.
  • Tests cover streaming rejection, invalid budgets, cancellation and expired deadlines. They also cover returned provider errors and successful streaming without a budget.

@h0rv

h0rv commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator Author

Additional verification passed on Go 1.27.3: go test -race ./pkg/instructor/... (core and all four providers).

@h0rv h0rv changed the title feat: add token budgets for validation retries Add token budgets for validation retries Nov 4, 2026
@h0rv

h0rv commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator Author

@jxnl @jxnl-oai Could you review this PR too? GitHub will not let me request a review from this account because it is not a collaborator on the repo.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a token budget for validation retries

1 participant