fix: make the llm:choose prompt describe the enum schema it is sent with - #77
Merged
Conversation
The choose prompt told the model to reply with only the bare option text
while the request carried an EnumFormat schema requiring {"choice": ...}.
Providers that validate the reply against the schema after generation
(Groq) rejected the whole response with HTTP 400 json_validate_failed, so
llm:choose failed on every call there.
The prompt now asks for the same {"choice": "<option>"} object the schema
enforces. Providers that ignore the constraint still work through the
existing bare-text fallback.
Adds a deterministic test pinning the prompt's final instruction to the
schema shape, and a __TEST_ECHO_PROMPT_TAIL marker in the test provider.
Verified live against Groq gpt-oss-20b: 25/25 e2e assertions pass.
This was referenced Sep 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
llm:choosefailed on every call against Groq with HTTP 400json_validate_failed. Found by the new T7 live test in #75.The system prompt told the model to reply with only the bare option text ("no quotes, no punctuation"), while the request carried an
EnumFormatschema requiring{"choice": "..."}. Groq validates the reply against the schema after generation rather than constraining decoding, so a model that followed the prompt producedsouth, and Groq rejected the response.Fix
The prompt now asks for the same
{"choice": "<option>"}object the schema enforces. Providers that ignore the constraint still work through the existing bare-text fallback inextractEnumChoice.Tests
LLMChoosePromptMatchesEnumSchema(tests.txt) pins the prompt's final instruction to the schema shape via a new__TEST_ECHO_PROMPT_TAILmarker inDeterministicTestProvider.openai/gpt-oss-20b, 25/25 pass (was 24/25 before this fix).