Feat: structured output#4207
Conversation
Port follow-ups on top of OpenHands#2808: - reject response_schema fields named 'summary' or 'security_risk': the SDK injects meta-fields with those names into every action schema after the response-schema merge, so a user field would be silently shadowed on the way out (risk) or double as the event summary on the way back - renumber the example to 56_ (48_ was taken by conversation_fork) - update the example NOTE: the reserved names are now enforced, not a convention
Co-authored-by: Vasco Schiavo <115561717+VascoSch92@users.noreply.github.com>
|
📁 PR Artifacts Notice This PR contains a |
Address review: collapse the reverse-scan loop into next() over a generator expression. Co-authored-by: Vasco Schiavo <115561717+VascoSch92@users.noreply.github.com>
VascoSch92
left a comment
VascoSch92
left a comment
There was a problem hiding this comment.
Moreover, could you make the comments and docstring minimal and not verbose. Thanks
| type[Action], | ||
| _create_action_type_with_schema(action_type, self.response_schema), | ||
| ) | ||
| return action_type.model_validate(arguments) |
There was a problem hiding this comment.
This is the blocker for me. Returning an instance of the generated class puts the dynamic name into kind, and kind is exactly what the event log round-trips on.
Real conversation with persistence_dir set, one FinishTool call carrying a response_schema:
events/event-00000-….json
"action": {"message":"done","success":true,"summary_text":"s",
"kind":"FinishActionWithTaskResult"}
base_state.json
"tools": [{"name":"FinishTool","params":{}}]
Reopening it (event_store.py:163,176 read through Event.model_validate_json):
ValidationError: 1 validation error for Event
action Value error, Unknown kind 'FinishActionWithTaskResult' for
openhands.sdk.tool.schema.Action; Expected one of: ['MCPToolAction',
'FinishAction', 'InvokeSkillAction', 'SwitchLLMAction', 'ThinkAction',
'VisionInspectAction']
It can't self-heal either — the spec that would rebuild the class was stripped on dump (see my note on spec.py). I also tried resuming with the identical schema re-declared in code, on the theory that it was just an ordering problem. Still fails: the class is only built lazily by _get_tool_schema/action_from_arguments, which runs after the event log is read.
Worth calling out that this is what separates the schema classes from _create_action_type_with_summary / create_action_type_with_risk on main — those are only ever used to render JSON schema (called solely from _get_tool_schema, tool.py:517/520), never instantiated, so they never reach kind. This PR is the first place a dynamic Action subclass gets stored.
I don't have a clean fix to hand. Validating against the augmented type but constructing the base type would keep kind stable, except Action is extra="forbid" so the extra fields have nowhere to go and parse_response loses its source. Might be worth putting the structured payload in a declared field on the action instead of merging fields onto the class — that keeps the wire format stable and static.
| annotations[field_name] = field_info.annotation | ||
| attrs["__annotations__"] = annotations | ||
|
|
||
| new_type = type( |
There was a problem hiding this comment.
The generated name is derived from the user's class name, which makes collisions reachable — and _get_checked_concrete_subclasses (utils/models.py:152-158) treats a duplicate name as a hard error for the entire Action union.
Two modules each defining a Result schema is enough:
attach Result(alpha) to FinishTool -> ok
attach Result(beta) to FinishTool -> ok
Event.model_validate({..., "action": {"message":"hi","kind":"FinishAction"}})
ValidationError: Duplicate class definition for openhands.sdk.tool.schema.Action:
abc.FinishActionWithResult : abc.FinishActionWithResult
Note what breaks there: a plain, pre-existing FinishAction event that deserialized fine seconds earlier. Once poisoned it stays poisoned for the life of the process, and it takes every Action payload with it, not just the schema-bearing tool's.
main's naming (f"{action_type.__name__}WithSummary") derives only from the action type, so it can't collide. This is new.
Minor, but related: type() leaves __module__ as abc, which is where that abc.FinishActionWithResult comes from. Passing __module__ explicitly would at least make the error readable.
| Pydantic classes) so the spec can be persisted as part of conversation | ||
| state. These runtime values are reapplied by the registry on resolve. | ||
| """ | ||
| return {k: v for k, v in params.items() if not isinstance(v, type)} |
There was a problem hiding this comment.
This drops the class in python mode as well as JSON, so anything that round-trips the Agent loses structured output silently:
agent.tools persisted : [{'name':'FinishTool','params':{}}]
after round-trip params : {}
resolved response_schema: None
tool schema properties : ['message','summary']
Practical effect: remote / agent-server conversations never get structured output at all (the Agent is serialized to the server), and resumed local conversations quietly lose it.
The docstring says these values "are reapplied by the registry on resolve", but the registry can only reapply what's still in the in-memory spec — after a dump/load there's nothing left to reapply. That's also what turns the kind problem into an unrecoverable one rather than something that heals on reload.
At an absolute minimum I'd log a warning when a class-valued param gets dropped, so it isn't invisible.
| response_schema = params.pop("response_schema", None) | ||
| tools = resolver(params, conv_state) | ||
| if response_schema is not None: | ||
| tools = [t.set_response_schema(response_schema) for t in tools] |
There was a problem hiding this comment.
This applies the schema to every tool the resolver returns, which gets surprising with toolsets:
resolve_tool(Tool(name="browser_tool_set", params={"response_schema": Anno}))
tools resolved: 14
carrying response_schema: 14
So one response_schema on browser_tool_set forces those fields onto all 14 browser tools. Same shape for task_tool_set and workflow_tool_set. I doubt that's intended — either reject it when the resolver returns more than one tool, or document it very loudly.
|
|
||
| # Merge the structured response schema (if any) into the action schema | ||
| if self.response_schema is not None: | ||
| action_type = _create_action_type_with_schema( |
There was a problem hiding this comment.
Merging here means only subclasses that don't override _get_tool_schema pick it up. Two in-tree ones do override it, and they fail in opposite directions.
ClientTool (client_tool.py:294-299) lifts only security_risk/summary out of the super() schema, so the response fields are dropped — but it doesn't override action_from_arguments, so validation still demands them:
baseline advertised (no schema): ['q','summary']
with response_schema : ['q','summary'] <- unchanged
action_from_arguments({"q":"hi"})
ValidationError: success Field required
The model is never told about the field it's required to send, so every call to that tool fails.
MCPToolDefinition (mcp/tool.py:358 and :273) overrides both, so a response_schema is a silent no-op there and parse_response later gives AttributeError: 'MCPToolAction' object has no attribute 'success'.
Unrelated to those two but same area: to_mcp_tool (tool.py:481) builds inputSchema straight off self.action_type and never routes through here, so it advertises the un-augmented schema. Only tests reach that path today — both in-tree callers pass an explicit input_schema — so it's latent rather than broken. But the PR description says this touches to_mcp_tool, and as far as I can tell it doesn't.
| # return when invoking this tool. When set, the model's fields are merged | ||
| # into the action schema sent to the LLM, and ``action_from_arguments`` | ||
| # validates the call against that augmented schema. Runtime-only. | ||
| response_schema: SkipJsonSchema[type[BaseModel] | None] = Field( |
There was a problem hiding this comment.
Another consequence of hanging the fields off the action type: Agent._get_action_event repairs malformed LLM output against tool.action_type (agent.py:1210), which doesn't know about them. So the GLM-4.6 / minimax "list arrived as a JSON string" fixups skip response-schema fields entirely:
{"message":"done","files_changed":'["a.py","b.py"]'}
after fix_malformed_tool_arguments: files_changed still the raw string
action_from_arguments -> ValidationError: Input should be a valid list
Passing the augmented type there is enough to fix it:
repaired against augmented type: {'files_changed': ['a.py','b.py']}
| f"existing fields on {action_type.__name__}." | ||
| ) | ||
|
|
||
| reserved = _RESERVED_META_FIELDS & set(response_schema.model_fields) |
There was a problem hiding this comment.
The guard itself is a good addition, but it only fires from _get_tool_schema(), so the ValueError lands in the middle of the agent's LLM step rather than where the mistake was made:
resolve_tool + set_response_schema -> succeeds, no error
tool.response_schema = <class 'Reserved'>
...later, on the first LLM call:
ValueError: response_schema fields ['summary'] are reserved: ...
Any reason it can't move into set_response_schema? Then a bad schema blows up at config time, where the user can actually see it.
| # Meta-field names injected into every action schema after the | ||
| # response-schema merge (see ``_create_action_type_with_summary`` and | ||
| # ``create_action_type_with_risk``); rejected in user response schemas. | ||
| _RESERVED_META_FIELDS = frozenset({"summary", "security_risk"}) |
There was a problem hiding this comment.
kind belongs in this set too. It's the discriminator injected by DiscriminatedUnionMixin — same category as the other two — and a schema declaring it skips the nice error entirely:
UserWarning: Field name "kind" in "FinishActionWithKindSchema" shadows an attribute in parent "FinishAction"
TypeError: Field 'kind' of class 'FinishActionWithKindSchema' overrides symbol of same
name in a parent class. This override with a computed_field is incompatible.
| raise ValueError(f"Tool '{self.name}' has no response_schema configured.") | ||
| # Pick only the schema's own fields so meta-fields (kind, summary, ...) | ||
| # don't leak in and we don't have to chase them. | ||
| data = {k: getattr(action, k) for k in self.response_schema.model_fields} |
There was a problem hiding this comment.
Bare getattr gives a fairly opaque error when the action doesn't carry the fields:
tool.parse_response(FinishAction(message="hi"))
AttributeError: 'FinishAction' object has no attribute 'foo'
Not hypothetical — that's precisely the post-resume state, where the tool has its schema back but the stored actions don't have the fields. A sentinel check with a real message would be kinder.
| ObservationT = TypeVar("ObservationT", bound=Observation) | ||
| _action_types_with_risk: dict[type, type] = {} | ||
| _action_types_with_summary: dict[type, type] = {} | ||
| _action_types_with_schema: dict[tuple[type, type], type] = {} |
There was a problem hiding this comment.
This never evicts, and the keys hold strong refs to both the action type and the user's schema class, so generated classes live for the life of the process (51 entries after a 50-schema loop). Every type() call also triggers __init_subclass__ -> _bump_subclass_generation(), invalidating the global subclass caches.
Probably fine in practice, since schemas are normally declared once at module scope. But it is a real leak for anything constructing schemas dynamically, and worth at least a comment saying the cache is intentionally permanent.
2 participants
HUMAN:
Implemented a general version of #4116 after @VascoSch92 suggested it. Credit also to his original design idea #2808 remained unfinished.
AGENT:
Why
#2566 asks for first-class structured output. Today, getting reliably
formatted responses out of an agent means manual prompting plus brittle
post-processing. @VascoSch92 implemented the mechanism in #2808 — reviewed
favorably, never landed for lack of time — and offered the final pass to me.
This PR completes that work: his three commits rebased onto current
main(authorship preserved), plus a fix for the known meta-field collision issue.
Summary
Tool(name="FinishTool", params={"response_schema": ProjectFacts}).The model's fields are merged into the action schema sent to the LLM
(
_create_action_type_with_schema, cached per(action_type, schema)pair),validated on receipt in
action_from_arguments, and recoverable typed viaparse_response()/parse_last_response().response_schemafrom spec params before calling thetool's
create()(factories with fixed signatures, e.g.FinishTool.create,never see it) and applies it via
set_response_schema()— amodel_copy,consistent with
set_executor().response_schemais runtime-only (SkipJsonSchema,exclude=True): it nevercrosses the serialization boundary, so persisted events/specs are unchanged
and older readers are unaffected.
Tool.paramsdrops class values on dump.summaryorsecurity_riskare rejected with an explicitValueError. The SDK injectsmeta-fields with those names into every action schema after the merge: a
user field would be silently absorbed as the event summary (
summary) orredefined and swallowed by the risk-analyzer flow (
security_risk).if response_schema is None.Issue Number
Closes #2566. Supersedes / completes #2808.
How to Test
19 tests cover: schema extension, payload validation (accept/reject), nested
Pydantic models, class-creation caching, spec serialization dropping class
values,
parse_last_responseacross multiple tools, executor unchanged,tool-without-schema unchanged, action/schema field collision, and the two
reserved meta-field names. Full
tests/sdk/tool+tests/crosssuites pass(verified on two machines);
tests/sdk/agentpasses except one failure(
test_acp_agent.py::…::test_gemini_046_uses_set_session_model) reproduced onunmodified
main, i.e. pre-existing.pre-commithook set (ruff, pyright,pycodestyle, import rules, tool registration) clean on all changed files.
End-to-end with a real LLM (
gemini/gemini-2.5-flashvia litellm):Video/Screenshots
Both directions of the mechanism, from the run above:
Schema attached to
TerminalTool— the model must justify every command:Schema attached to
FinishTool— the final answer comes back as a typedobject via
parse_last_response()(Pydantic-validated, no text parsing):Type
Notes
authorship preserved; they applied onto current
mainwithout conflicts.Commit 4 adds the reserved-name guard, renumbers the example to
56_(48 was taken), and updates its NOTE.
action_from_arguments/to_mcp_tool, i.e.the path of every tool call. The no-schema fast path is covered by
test_finish_tool_without_schema_is_unchanged; happy to have the eval suiterun before merge if maintainers want the extra confidence — guidance on how
to trigger it welcome.