[egafni]

Fixes from a review of the repo, then a migration of the provider tests from json_cache to HTTP cassettes.

Correctness

• Score was computed over an unnormalized distribution. score is an expected value, so it only means anything over weights summing to 1. An LLM distribution summing to 0.2 produced score=0.1 on a 0..1 scale. It is now computed over an explicitly normalized copy while the reported probabilities stay untouched when normalize_probabilities=False.
• Token usage under-reported retried calls. Only the successful attempt's usage was counted, so tokens burned by an attempt that later failed were never billed. A single RunUsage is now threaded through every attempt. Because request counts accumulate across attempts, malformed-structure retries are counted from a per-attempt snapshot rather than the run total.
• Two prompt defects, both found by recording real traffic. The answers field was described as "keyed by question ID", so gpt-4o-mini invented an ID and nested every answer under it. Discrete score questions carried no criteria at all, so the model guessed its own scale and returned 5 on a 0-4 score.

Tests

json_cache memoized system_one's return value, so on a cache hit none of the client ran — the test compared a stored blob against a literal in the file beside it. VCR (vcrpy via pytest-recording) replays at the HTTP layer instead, so prompt building, output parsing, normalization, confidence, and usage aggregation all execute for real. It caught both prompt bugs above on the first recording run.

• Expected responses stay inline in the test as readable JSON, now the single source of truth instead of being duplicated in json_cache.json.
• latency is dropped before comparison rather than pinned to a value no later run can reproduce (REGENERATE_JSON_CACHE=1 previously failed by construction).
• Cassettes are JSON, matched on request body because every call posts to the same endpoint. On vcrpy's default matchers they would all replay the first interaction.
• Request credentials filtered; response headers reduced to a content-type allowlist, since providers return organization and workspace IDs, request IDs, cookies, and per-account rate-limit quotas.
• --block-network is on by default, so a missing cassette fails instead of silently making a live call.

Coverage went from 30 to 46 tests: timeout mapping, TypeSafeUnknownError status passthrough, retry exhaustion, the async retry path, invalid question collections, and a test asserting tokens from failed attempts are billed.

Also

• Ruff lint families enabled (the existing # noqa: B008 / BLE001 referenced rules that were never on), testpaths set so pytest stops collecting from .venv, distribution renamed to match the imported module, real description.
• .venv console scripts were shebanged to a stale OpenTypeSafe2 path from the directory rename, so uv run pytest failed outright. Rebuilt.
• Comment on why OpenTypeSafeClient.__init__ skips super().__init__(), and the invariant that creates.

Test plan

• uv run pytest -> 46 passed.
• With OPENAI_API_KEY, ANTHROPIC_API_KEY, and TYPESAFE_API_KEY unset and network blocked -> 46 passed.
• uv run pytest tests/test_live_apis.py --record-mode=rewrite re-records against the live APIs; verified end to end.
• Cassettes scanned for the real key values, sk-/Bearer patterns, and identifying headers -> clean.

Notes for review

• 011551f hardens JsonCache and 24e7595 then deletes it. That is the order the work was requested in; squash if you would rather not carry it.
• Two fidelity gaps this surfaced, not addressed here: the TypeSafe API returns model: "speed_v10_mango_loco" rather than the requested speed_latest alias, and its ScoreAnswer carries a legend mapping each score level to its criterion, which OpenTypeSafeClient never emits.
• Still open from the review: _resolve_pydantic_ai_model only knows OpenAI and Anthropic prefixes, and _TOKEN_ERROR_MARKERS substring matching is fragile.