Repository navigation
Description
A refusal from either provider is treated as malformed JSON. Anthropic returns stop_reason="refusal" with empty content, and Responses returns a completed message whose only part is a refusal, so output_text is empty. anthropic._result only checks max_tokens and openai._responses_result only checks status, so the empty text reaches _decode_or_correct, which spends a corrective retry on it and replays the empty string as the assistant turn. Anthropic stop_reason="model_context_window_exceeded" goes the same way.
What a caller sees with n_retry_malformed_structure=2:
- Anthropic: 2 requests, then
TypeSafeBadRequestError: 400 messages.1: all messages must have non-empty content except for the optional final assistant message, because the Messages API rejects the empty assistant turn. Nothing in the error says refusal. - Responses: 3 requests, all refused, then
TypeSafeAPIResponseValidationError: 200 Invalid response data at 'answers'.
The README already says a max_tokens cutoff does not consume malformed-output retries. Refusal and context overflow are the same kind of provider-declared non-answer and should raise the same way.
Repro, no network:
git fetch https://github.com/Dnyaneshvn/system-one-adapter-python fix/provider-refusals
git checkout FETCH_HEAD -- tests/test_provider_requests.py tests/test_openai_transports.py
uv run pytest tests/test_provider_requests.py tests/test_openai_transports.py -k "refusal or stop_reason" -q
8 failed on main. The client-level tests go through the real SDK parsers with the HTTP layer stubbed.
Fix on https://github.com/Dnyaneshvn/system-one-adapter-python/tree/fix/provider-refusals:
anthropic._resultraisesTypeSafeError("Anthropic response did not complete: {stop_reason}.")for any stop reason other thanend_turn,stop_sequence, orNone. Themax_tokensmessage is unchanged.openai._responses_resultraisesTypeSafeError("OpenAI response was a refusal: {text}")when a message item carries arefusalpart.- Both run after
record_response, sollm_attemptskeeps the full response and finish reason. - 11 regression tests in the existing test files. Chat Completions is left alone since Chat Completions accepts finish_reason="length" as a successful typed evaluation #38 covers
finish_reasonthere.
evals/refusal_handling/ on the branch runs 12 cases (abusive tickets in English, Hinglish and Marathi, a self-harm transcript, a doxxing post, a contract over the context window, two benign controls) through both providers with the HTTP layer stubbed:
| Outcome | Cases | Requests per case, main | Error names the reason, main | Requests per case, branch | Error names the reason, branch |
|---|---|---|---|---|---|
refusal, Anthropic |
4 | 2 | 0/4 | 1 | 4/4 |
refusal, Responses |
3 | 3 | 0/3 | 1 | 3/3 |
model_context_window_exceeded, Anthropic |
1 | 3 | 0/1 | 1 | 1/1 |
max_tokens, Anthropic (already handled) |
1 | 1 | 1/1 | 1 | 1/1 |
incomplete, Responses (already handled) |
1 | 1 | 1/1 | 1 | 1/1 |
| benign, both providers | 2 | 1, succeeds | 1, succeeds |
On main the 8 unhandled non-answers cost 12 extra requests and 12 corrective retries between them, and every corrective request replayed an empty assistant turn. On the branch they cost 0 of each. The two already-handled outcomes and the two benign controls are unchanged.
Tests: uv run pytest -q (240 passed). uv run pyrefly check: 0 errors.
Activity
Dnyaneshvn commentedon Sep 22, 2026
The fix and tests are on the fork branch linked above. Is this something you would take as a PR, or was it closed with #38 and #39 as part of a broader change?