Repository navigation
Adapter function metrics record success for contract-level IOContract mismatches #1611
Description
Summary
mellea.adapter_function.invocations/.parse_failures mis-record one class of failure for adapter functions called through call_intrinsic: a contract-level IOContract mismatch on output that is valid JSON.
Background
PR #1609 (issue #1560) wired adapter_function_invocation_complete firing for the Embedded/Granite Switch binding. It fires the outcome once each backend's granite_formatters_processing closure resolves — success if the response parses as JSON, schema_error if it doesn't, error otherwise.
That closure runs before call_intrinsic (mellea/stdlib/components/intrinsic/_util.py:171) calls adapter.io_contract.parse(result_str) — the actual contract check (required fields, types, shape). So when a response is syntactically valid JSON but fails that contract check, AdapterSchemaMismatchError is raised to the caller — but the invocation was already recorded as outcome="success".
Impact
- The caller is unaffected:
AdapterSchemaMismatchErrorpropagates uncaught through every wrapper (rag.py,core.py,guardian.py) straight to application code. No silent failures. - The metric is wrong:
.parse_failuresundercounts this specific failure mode, and.invocations{outcome="success"}overcounts it. A model that drifted into returning valid-but-wrong-shape JSON would look 100% healthy on a dashboard built from these metrics while every call actually fails downstream. - Scope is narrow: only applies to output that's valid JSON but violates the adapter's declared schema — not malformed JSON (already
schema_error) and not a generation/network failure (alreadyerror). - Applies to
local_file/PEFT too, not justembedded—AdapterMixin.adapter_scope's outcome closes at generation return, before parsing or contract validation ever run, so this binding never records anything for this failure mode either.
Proposed fix
Move (or add) the invocation-complete fire point into call_intrinsic, after io_contract.parse() resolves, so the recorded outcome reflects contract validation too. Complications:
call_intrinsicis shared by both bindings — this needs handling that's correct forlocal_fileas well asembedded, not an embedded-only patch.call_intrinsicis synchronous; hook dispatch (invoke_hook) is async. Needs the same sync→async bridgingadapter.py's_fire_invocation_completealready does via_run_async_in_thread— the embedded fire helper introduced in fix(backends): fire adapter_function_invocation_complete for embedded adapter calls #1609 (_fire_embedded_invocation_complete) deliberately avoids that bridging since it's always called from an already-running coroutine, so this can't just reuse it as-is.- Need to decide whether the backend-level
successfire (in the closures) should be suppressed/deferred entirely in favour of this one, to keep the "fires exactly once per invocation" invariant intact.
cc @ajbozarth
Activity
refactor: trim remaining verbose comments/docstrings, point follow-up…
fix(backends): fire adapter_function_invocation_complete for embedded…
fix(backends): fix Qwen-review findings on composed-Adapter path (gen…
feat(backends): accept composed Adapter directly (Epic generative-com…
planetf1 commentedon Oct 5, 2026
Status: root cause confirmed (2026-10-05)
Verified against
mainat4c53add4e. The ordering is exactly as the issue describes. The backend invocation closures fireadapter_function_invocation_completewithoutcome="success"the moment the response parses as JSON — two sites,mellea/backends/huggingface.py:1654andmellea/backends/openai.py:1272(Ollama has no embedded fire site). The actual contract check,adapter.io_contract.parse(result_str), runs after that, at the end ofcall_intrinsic(mellea/stdlib/components/intrinsic/_util.py:255). A response that is valid JSON but wrong in shape therefore raisesAdapterSchemaMismatchErrorto the caller after the invocation was already recorded successful.Callers are unaffected — the error propagates uncaught through every wrapper. What is wrong is only the measurement:
.invocations{outcome="success"}overcounts and.parse_failuresundercounts, so an adapter that has drifted into "valid JSON, wrong shape" reads as 100% healthy on a dashboard while every call fails downstream. Scope confirmed as the issue states: malformed JSON is alreadyschema_error, generation/network failures alreadyerror; the gap is this one class only.Proposed direction: move or re-fire the outcome decision past the contract parse. The closure can still fire a generation-complete event, but the invocation outcome needs to be decided where the parse result is visible — which means each binding's close point (embedded: the two backend sites above; local_file/PEFT: the
adapter_scopeclose that the issue notes also precedes the parse) rather than a single call site. I want to walk all three binding paths before drafting, because a naive re-fire would double-count.invocations— that constraint should shape the design.Metric behaviour change to flag: after the fix, drifted adapters will surface as
schema_errorinstead ofsuccess. Success rates for affected adapters will drop relative to pre-fix dashboards — a correction, not a regression, but anyone charting the history across the fix will see the step.