Skip to content

Adapter function metrics record success for contract-level IOContract mismatches #1611

Open

Description

@planetf1

Summary

mellea.adapter_function.invocations/.parse_failures mis-record one class of failure for adapter functions called through call_intrinsic: a contract-level IOContract mismatch on output that is valid JSON.

Background

PR #1609 (issue #1560) wired adapter_function_invocation_complete firing for the Embedded/Granite Switch binding. It fires the outcome once each backend's granite_formatters_processing closure resolves — success if the response parses as JSON, schema_error if it doesn't, error otherwise.

That closure runs before call_intrinsic (mellea/stdlib/components/intrinsic/_util.py:171) calls adapter.io_contract.parse(result_str) — the actual contract check (required fields, types, shape). So when a response is syntactically valid JSON but fails that contract check, AdapterSchemaMismatchError is raised to the caller — but the invocation was already recorded as outcome="success".

Impact

  • The caller is unaffected: AdapterSchemaMismatchError propagates uncaught through every wrapper (rag.py, core.py, guardian.py) straight to application code. No silent failures.
  • The metric is wrong: .parse_failures undercounts this specific failure mode, and .invocations{outcome="success"} overcounts it. A model that drifted into returning valid-but-wrong-shape JSON would look 100% healthy on a dashboard built from these metrics while every call actually fails downstream.
  • Scope is narrow: only applies to output that's valid JSON but violates the adapter's declared schema — not malformed JSON (already schema_error) and not a generation/network failure (already error).
  • Applies to local_file/PEFT too, not just embedded — AdapterMixin.adapter_scope's outcome closes at generation return, before parsing or contract validation ever run, so this binding never records anything for this failure mode either.

Proposed fix

Move (or add) the invocation-complete fire point into call_intrinsic, after io_contract.parse() resolves, so the recorded outcome reflects contract validation too. Complications:

  • call_intrinsic is shared by both bindings — this needs handling that's correct for local_file as well as embedded, not an embedded-only patch.
  • call_intrinsic is synchronous; hook dispatch (invoke_hook) is async. Needs the same sync→async bridging adapter.py's _fire_invocation_complete already does via _run_async_in_thread — the embedded fire helper introduced in fix(backends): fire adapter_function_invocation_complete for embedded adapter calls #1609 (_fire_embedded_invocation_complete) deliberately avoids that bridging since it's always called from an already-running coroutine, so this can't just reuse it as-is.
  • Need to decide whether the backend-level success fire (in the closures) should be suppressed/deferred entirely in favour of this one, to keep the "fires exactly once per invocation" invariant intact.

cc @ajbozarth

Activity

  1. planetf1 commented on Oct 5, 2026

    @planetf1
    ContributorAuthor

    Status: root cause confirmed (2026-10-05)

    Verified against main at 4c53add4e. The ordering is exactly as the issue describes. The backend invocation closures fire adapter_function_invocation_complete with outcome="success" the moment the response parses as JSON — two sites, mellea/backends/huggingface.py:1654 and mellea/backends/openai.py:1272 (Ollama has no embedded fire site). The actual contract check, adapter.io_contract.parse(result_str), runs after that, at the end of call_intrinsic (mellea/stdlib/components/intrinsic/_util.py:255). A response that is valid JSON but wrong in shape therefore raises AdapterSchemaMismatchError to the caller after the invocation was already recorded successful.

    Callers are unaffected — the error propagates uncaught through every wrapper. What is wrong is only the measurement: .invocations{outcome="success"} overcounts and .parse_failures undercounts, so an adapter that has drifted into "valid JSON, wrong shape" reads as 100% healthy on a dashboard while every call fails downstream. Scope confirmed as the issue states: malformed JSON is already schema_error, generation/network failures already error; the gap is this one class only.

    Proposed direction: move or re-fire the outcome decision past the contract parse. The closure can still fire a generation-complete event, but the invocation outcome needs to be decided where the parse result is visible — which means each binding's close point (embedded: the two backend sites above; local_file/PEFT: the adapter_scope close that the issue notes also precedes the parse) rather than a single call site. I want to walk all three binding paths before drafting, because a naive re-fire would double-count .invocations — that constraint should shape the design.

    Metric behaviour change to flag: after the fix, drifted adapters will surface as schema_error instead of success. Success rates for affected adapters will drop relative to pre-fix dashboards — a correction, not a regression, but anyone charting the history across the fix will see the step.

  2. added
    p2Medium/low: minor bugs, niche features, polish, docs, tests, cleanup. Scoped, lower urgency.
    bugSomething isn't working
    on Oct 5, 2026
  3. added
    area/adapter-functionsGranite adapter functions: framework and adaptiers including RAG, Guardian, Core
    on Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/adapter-functionsGranite adapter functions: framework and adaptiers including RAG, Guardian, Corearea/telemetryOTel spans, metrics, tracing, semconvbugSomething isn't workingp2Medium/low: minor bugs, niche features, polish, docs, tests, cleanup. Scoped, lower urgency.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions