Skip to content

Refusals are retried as malformed output and fail with an unrelated error #43

Closed

Description

@Dnyaneshvn

A refusal from either provider is treated as malformed JSON. Anthropic returns stop_reason="refusal" with empty content, and Responses returns a completed message whose only part is a refusal, so output_text is empty. anthropic._result only checks max_tokens and openai._responses_result only checks status, so the empty text reaches _decode_or_correct, which spends a corrective retry on it and replays the empty string as the assistant turn. Anthropic stop_reason="model_context_window_exceeded" goes the same way.

What a caller sees with n_retry_malformed_structure=2:

  • Anthropic: 2 requests, then TypeSafeBadRequestError: 400 messages.1: all messages must have non-empty content except for the optional final assistant message, because the Messages API rejects the empty assistant turn. Nothing in the error says refusal.
  • Responses: 3 requests, all refused, then TypeSafeAPIResponseValidationError: 200 Invalid response data at 'answers'.

The README already says a max_tokens cutoff does not consume malformed-output retries. Refusal and context overflow are the same kind of provider-declared non-answer and should raise the same way.

Repro, no network:

git fetch https://github.com/Dnyaneshvn/system-one-adapter-python fix/provider-refusals
git checkout FETCH_HEAD -- tests/test_provider_requests.py tests/test_openai_transports.py
uv run pytest tests/test_provider_requests.py tests/test_openai_transports.py -k "refusal or stop_reason" -q

8 failed on main. The client-level tests go through the real SDK parsers with the HTTP layer stubbed.

Fix on https://github.com/Dnyaneshvn/system-one-adapter-python/tree/fix/provider-refusals:

  • anthropic._result raises TypeSafeError("Anthropic response did not complete: {stop_reason}.") for any stop reason other than end_turn, stop_sequence, or None. The max_tokens message is unchanged.
  • openai._responses_result raises TypeSafeError("OpenAI response was a refusal: {text}") when a message item carries a refusal part.
  • Both run after record_response, so llm_attempts keeps the full response and finish reason.
  • 11 regression tests in the existing test files. Chat Completions is left alone since Chat Completions accepts finish_reason="length" as a successful typed evaluation #38 covers finish_reason there.

evals/refusal_handling/ on the branch runs 12 cases (abusive tickets in English, Hinglish and Marathi, a self-harm transcript, a doxxing post, a contract over the context window, two benign controls) through both providers with the HTTP layer stubbed:

Outcome Cases Requests per case, main Error names the reason, main Requests per case, branch Error names the reason, branch
refusal, Anthropic 4 2 0/4 1 4/4
refusal, Responses 3 3 0/3 1 3/3
model_context_window_exceeded, Anthropic 1 3 0/1 1 1/1
max_tokens, Anthropic (already handled) 1 1 1/1 1 1/1
incomplete, Responses (already handled) 1 1 1/1 1 1/1
benign, both providers 2 1, succeeds 1, succeeds

On main the 8 unhandled non-answers cost 12 extra requests and 12 corrective retries between them, and every corrective request replayed an empty assistant turn. On the branch they cost 0 of each. The two already-handled outcomes and the two benign controls are unchanged.

Tests: uv run pytest -q (240 passed). uv run pyrefly check: 0 errors.

Activity

  1. Dnyaneshvn commented on Sep 22, 2026

    @Dnyaneshvn
    Author

    The fix and tests are on the fork branch linked above. Is this something you would take as a PR, or was it closed with #38 and #39 as part of a broader change?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions