Skip to content

Relax eval inference preflight timeouts#498

Merged
d42me merged 3 commits intomainfrom
fix/eval-preflight-thinking-timeout
Apr 8, 2026
Merged

Relax eval inference preflight timeouts#498
d42me merged 3 commits intomainfrom
fix/eval-preflight-thinking-timeout

Conversation

@d42me
Copy link
Copy Markdown
Contributor

@d42me d42me commented Apr 8, 2026

Summary

  • add explicit inference timeout handling so eval preflight failures surface cleanly instead of crashing with a raw httpx timeout
  • increase the Prime eval preflight timeout to 5 minutes and make timeouted model/billing probes warning-only so slow thinking models can still start
  • add regression coverage for the longer preflight timeout and timeout fallback paths

Testing

  • pytest packages/prime/tests/test_eval_billing.py -q
  • pytest packages/prime/tests/test_hosted_eval.py -q

Note

Medium Risk
Changes eval preflight behavior by passing custom HTTP timeouts into InferenceClient and treating timeout failures as warnings, which could allow eval runs to proceed with an invalid/unwarmed model. Risk is limited to CLI UX/flow and is covered by added tests.

Overview
Relaxes prime eval inference preflight by introducing a dedicated EVAL_PREFLIGHT_TIMEOUT and wiring it through to InferenceClient, rather than using the default client timeout.

Model validation and billing-probe preflights now catch httpx.TimeoutException and continue with a warning instead of crashing/failing the run.

Updates tests to stub InferenceClient construction, assert the preflight timeout is passed through, and adds regression coverage for the timeout-continue paths and custom client timeout support.

Reviewed by Cursor Bugbot for commit e4d6139. Bugbot is set up for automated code reviews on this repo. Configure here.

@d42me d42me requested a review from AmeenP April 8, 2026 21:21
Copy link
Copy Markdown

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit e4d6139. Configure here.

Comment thread packages/prime/src/prime_cli/verifiers_bridge.py
@d42me d42me merged commit 9301c5a into main Apr 8, 2026
12 checks passed
@d42me d42me deleted the fix/eval-preflight-thinking-timeout branch April 8, 2026 21:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants