Skip to content

feat(providers): watsonx.ai embeddings - #1356

Merged
vgmartinez merged 3 commits into
developfrom
claude/ibm-watsonx-native-integration-rtssu4
Nov 15, 2026
Merged

vgmartinez merged 3 commits into
developfrom
claude/ibm-watsonx-native-integration-rtssu4

Conversation

@vgmartinez

@vgmartinez vgmartinez commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Follow-up to #1349 (native IBM watsonx.ai provider). A watsonx registry now serves /v1/embeddings, and embeddings calls are now priced: a fix in the shared OpenAI embeddings adapter also makes OpenAI, Azure, Mistral and openai_compatible embeddings record their tokens.

Commits

1. fix(adapter): report the tokens of an OpenAI embeddings response

decodeResponseMeta in pkg/app/proxy/provider.go prices a call from canonical.Usage. OpenAIEmbeddingsAdapter.DecodeResponse only kept the usage inside ProviderExtensions["embedding"]. As a result, every OpenAI-format embeddings call on develop is recorded without tokens or cost, and dollar budgets never count it. That covers OpenAI, Azure, Mistral, openai_compatible, and watsonx through this PR.

The Vertex predict and Bedrock Titan adapters already set Usage; this one now does too. TestProviderInvoke_VertexPredictCrossFormatToOpenAI now asserts resp.Usage. It failed before the one-line fix and passes after.

Cohere embeddings have the same symptom for a different reason: cohereEmbedResponse does not read meta.billed_units. That is left for a separate change.

2. feat(providers): watsonx.ai embeddings

The client translates the OpenAI embeddings body to POST /ml/v1/text/embeddings (with the registry's IAM token and project ID) and the answer back to OpenAI's shape:

OpenAI request What watsonx gets / what the caller gets
input: string or list of strings inputs. More than 1,000 (IBM's per-call limit) are split into several calls; vectors are joined in order and token counts summed.
input: token arrays 400: watsonx.ai embeds text.
encoding_format: base64 (the OpenAI SDKs' default) Vectors returned as little-endian float32, standard base64, as OpenAI does.
dimensions 400: watsonx.ai models return their own fixed size, and a silently different length would break a vector store.
model model_id, with the same precedence as chat (configured, then body, then default).
response {object: list, data[].embedding, model, usage: {prompt_tokens, total_tokens}}. usage.prompt_tokens comes from input_token_count, so the call is priced.

watsonx errors (for example IBM's 512-token input limit) come back as OpenAI error envelopes with IBM's trace id. A 401 retries once with a fresh IAM token, as chat does.

Model list:

  • The region listing now includes embedding models (function: embedding) as well as chat ones. Frankfurt and Dallas both serve ibm/granite-embedding-278m-multilingual, intfloat/multilingual-e5-large, two ibm/slate-* models and sentence-transformers/all-minilm-l6-v2.
  • providers.LiveModel / appcatalog.LiveModel gain optional Capabilities ("chat", "embed"). deploymentModels and LiveCatalogLister.List use them when the catalog has no row of its own for a model, so a watsonx embedding model is not shown as a chat model. A catalog row keeps its own flags.
  • SupportsCapability(watsonx, embeddings) is now true, so routing puts watsonx registries in the embeddings pool. The console's provider card picks up the capability from the catalog.

3. docs(openapi): regenerate for the A2A skill scan (not this PR's change)

The first CI run failed OpenAPI Drift. Running make docs with the pinned swag on develop d7f267e alone produces the same diff: skill_scan fields from #1353, which merged without regenerating docs/. Nothing in it concerns watsonx. This commit is that make docs output. go test ./docs/... passes, and OpenAPI Drift is green on 00d04d3. It can also go to develop on its own.

Testing

  • pkg/infra/providers/watsonx (90% coverage) has new tests covering:
    • the translation both ways
    • a single string input
    • 2,500 inputs, split 1,000 / 1,000 / 500 with order and token sum checked
    • base64 decoded back to float32
    • every refusal, none of which calls IAM or watsonx
    • error relay
    • the retry on 401
    • a short upstream answer
    • the gateway adapter reading the usage the proxy prices from
  • Catalog: the watsonx live-listing test checks that an embedding model the catalog lacks keeps IBM's flags and a catalog row keeps its own.
  • Locally on this branch (based on develop d7f267e):
    • go test -race ./...: 215 packages ok, 0 failures.
    • Functional suite with the CI command (go test -tags functional -race -p 1 ./tests/functional/... ./pkg/infra/database/migrations) against Postgres 16 and Redis: 717 passed, 0 failed, 4 skipped. The skips are environmental: no OPENAI_API_KEY, plan rate limiter disabled, needs a real TrustGuard, and RUN-1621.
    • golangci-lint v2.14.0 on the touched packages: 0 issues.
    • addlicense -check and scripts/check-comments.sh pass.

Not yet verified against live watsonx: no IBM Cloud key was available here. Request and response shapes follow IBM's ibm-watsonx-ai SDK (model_id, inputs, project_id, parameters.return_options; results[].embedding, input_token_count; 1,000 inputs per call). The regional model specs were read live. Please run one embeddings call with a real key before merging.

Not included

  • No truncate_input_tokens: a text over 512 tokens is IBM's 400, like an over-long OpenAI input. A registry option to truncate could follow if needed.
  • No embeddings model prices: models.dev publishes none for watsonx. Registry overrides (for example ibm/*) cover them.

Docs: NeuralTrust/docs#561.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KF89yDnBWqZ6yB3bLqAtb9

The proxy prices a call from canonical.Usage, but the OpenAI embeddings
adapter only kept the usage inside the embedding extension. Every
OpenAI-format embeddings call (OpenAI, Azure, Mistral, openai_compatible)
was therefore recorded without tokens or cost, and dollar budgets never
counted it. The Vertex predict and Bedrock Titan adapters already set
Usage; this one now does too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KF89yDnBWqZ6yB3bLqAtb9
A watsonx registry now serves /v1/embeddings. The client translates the
OpenAI embeddings body to /ml/v1/text/embeddings with the registry's IAM
token and project, and the answer back to OpenAI's shape.

- input: a string or a list of strings. Token arrays are a 400, since
  watsonx.ai embeds text. More than 1,000 inputs (IBM's limit per call)
  are split into calls whose vectors are joined in order and whose
  token counts are summed.
- encoding_format base64, which the OpenAI SDKs request by default,
  returns little-endian float32 vectors as OpenAI does.
- dimensions is a 400: watsonx.ai models return their own fixed size,
  and a silently different length would break a vector store.
- usage.prompt_tokens comes from input_token_count, so calls are priced.
- The region listing now includes embedding models. A live model can
  carry capability flags ("chat", "embed"), which the catalog uses when
  it has no row of its own for the model, so a watsonx embedding model
  is not shown as a chat model.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KF89yDnBWqZ6yB3bLqAtb9
@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Trivy code scan — HIGH/CRITICAL

No HIGH or CRITICAL vulnerabilities found. (run)

develop's A2A skill scan (#1353) added skill_scan fields to the agent
card and registry responses without regenerating docs/, so OpenAPI
Drift fails on develop and on every PR based on it. This is the output
of make docs with the pinned swag; nothing else changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KF89yDnBWqZ6yB3bLqAtb9
@vgmartinez
vgmartinez marked this pull request as ready for review November 15, 2026 16:36
@vgmartinez
vgmartinez merged commit d2220ac into develop Nov 15, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants