Add property-based tests for token counting accuracy and Zen provider functionality - #3
Conversation
… functionality - Introduced property-based tests for token counting accuracy in `token_counting_accuracy.rs`, validating deterministic behavior, positive counts for non-empty content, and consistency across models. - Created `token_counting_accuracy.proptest-regressions` to store seeds for previously failing cases. - Added comprehensive property tests for the Zen provider in `zen_properties.rs`, ensuring consistent behavior, valid model IDs, and correct token counting. - Implemented unit tests for Zen provider in `zen_provider.rs`, covering creation, model availability, token counting, and health check reliability. - Ensured all models have valid IDs and pricing values, and verified that token counting scales with content length.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| fn count_tokens(&self, content: &str, model: &str) -> Result<usize, ProviderError> { | ||
| // Validate model | ||
| if !self.models().iter().any(|m| m.id == model) { | ||
| return Err(ProviderError::InvalidModel(model.to_string())); | ||
| } |
There was a problem hiding this comment.
count_tokens validates the requested model against the hard-coded models() list (only the two defaults). Chat requests fetch and cache the actual models from the Zen API, so any API-supplied model ID outside that static pair will be accepted by chat but rejected as InvalidModel when counting tokens, breaking token accounting for dynamically provisioned models. Consider validating against the cached API models instead of the fixed list.
Useful? React with 👍 / 👎.
1 participant
token_counting_accuracy.rs, validating deterministic behavior, positive counts for non-empty content, and consistency across models.token_counting_accuracy.proptest-regressionsto store seeds for previously failing cases.zen_properties.rs, ensuring consistent behavior, valid model IDs, and correct token counting.zen_provider.rs, covering creation, model availability, token counting, and health check reliability.