Skip to content

Chat Completions: an empty choices array raises a bare IndexError instead of TypeSafeError #47

Open

Description

@minhtien2405

Summary

When an OpenAI-compatible endpoint returns a schema-valid Chat Completions response whose choices array is empty, the adapter raises a bare IndexError: list index out of range from providers/openai.py::_result. Callers that catch typesafe_sdk.TypeSafeError, which is the adapter's failure contract, do not catch it, and error.debug (llm_attempts, llm_response) is not attached. Sync and async clients are both affected. This is the same kind of problem as #39 (omitted usage) and #38 (finish_reason="length"), in the same function.

Reproduction

Everything runs in memory, with no network and no API key. The fixture comes from tests/test_provider_nonanswers.py:

payload.update(choices=[], usage={"prompt_tokens": 12, "completion_tokens": 0, "total_tokens": 12})
provider = OpenAIProvider("test-model", api="chat_completions")
_evaluate(provider, structured=True)

At e1d4cc9 (v0.2.1):

src/system_one_adapter/providers/openai.py:45: IndexError
E       IndexError: list index out of range
4 failed   # sync/async × structured/prompted

The openai SDK types ChatCompletion.choices as a plain list with no minimum length, so the response passes the SDK's parsing and the failure only shows up when the adapter indexes it. Other clients have run into empty choices from OpenAI-compatible gateways, for example pydantic/pydantic-ai#7909.

Expected behavior

The client raises TypeSafeError that says the completion had no choices, with the recorded attempt preserved in error.debug. It does not spend malformed-output or transient retries. This matches how #38 and #39 were resolved.

Actual behavior

A bare IndexError is raised before record_response runs. The error is outside the TypeSafeError hierarchy and carries no debug trace.

Root cause

_result() evaluates response.choices[0].finish_reason on its first line, before any check on the number of choices. The Responses API path (_responses_result) is not affected because it iterates over response.output.

Proposed fix

A 3-line change:

 def _result(response: Any) -> ProviderResult:
+    if not response.choices:
+        record_response(response, finish_reason=None)
+        raise TypeSafeError("OpenAI chat completion returned no choices.")
     record_response(response, finish_reason=response.choices[0].finish_reason)

Regression test

test_chat_completion_empty_choices in tests/test_provider_nonanswers.py is parametrized over OpenAIProvider/AsyncOpenAIProvider × structured/prompted. It asserts that TypeSafeError is raised, that exactly one request was sent, that retry_reasons == [], and that the attempt records llm_response.choices == [] with finish_reason=None and an error. It fails 4/4 on e1d4cc9 and passes 4/4 with the fix. The full suite gives 428 passed, and pyrefly reports 0 errors.

The patch and test are ready. Since PRs are restricted to collaborators, the fix is on my fork: https://github.com/minhtien2405/system-one-adapter-python/tree/fix/chat-completions-empty-choices (commit 61fa1ac).

Environment

system-one-adapter 0.2.1 (e1d4cc9), openai 3.17.0, Python 3.14.7, Linux.

Activity

  1. aoright commented on Sep 25, 2026

    @aoright

    Can corroborate this finding. On main (e1d4cc9), accessing response.choices[0] before verifying response.choices raises an unhandled IndexError across both sync (OpenAIProvider) and async (AsyncOpenAIProvider) clients in structured and prompted modes.

    In our patch for Issue #46 (covering choice.message.refusal), we incorporated the identical guard:

    if not response.choices:
        record_response(response, finish_reason=None)
        raise TypeSafeError("OpenAI chat completion returned no choices.")

    Our regression test test_chat_completion_empty_choices in tests/test_provider_nonanswers.py verifies that TypeSafeError is raised with retry_reasons == [], len(llm_attempts) == 1, and finish_reason=None preserved in debug info. All 432 unit tests pass and pyrefly check reports 0 errors.

    Branch and commit reference:
    aoright@27d71f1

  2. minhtien2405 commented on Sep 25, 2026

    @minhtien2405
    Author

    For reference, the fix and regression test proposed in this issue are on minhtien2405/system-one-adapter-python@fix/chat-completions-empty-choices (commit 61fa1ac). Happy to adjust anything if the maintainers prefer a different approach.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions