Skip to content

feat(providers): native IBM watsonx.ai provider - #1349

Merged
vgmartinez merged 4 commits into
developfrom
claude/ibm-watsonx-native-integration-rtssu4
Nov 15, 2026
Merged

vgmartinez merged 4 commits into
developfrom
claude/ibm-watsonx-native-integration-rtssu4

Conversation

@vgmartinez

@vgmartinez vgmartinez commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds watsonx as a native LLM provider, so IBM watsonx.ai traffic can go through TrustGate directly instead of through IBM's model gateway on IBM Cloud. watsonx's API is not OpenAI-compatible and its IAM tokens expire after about an hour. This provider handles both.

Console side: NeuralTrust/app#4855 (same branch name).

What a registry takes

  • Auth: api_key holding an IBM Cloud API key. Users never paste a bearer token.
  • provider_options:
    • region: required, one of us-south, eu-de, eu-gb, jp-tok, au-syd, ca-tor. It is an allow-list because the region picks the host the IAM token is sent to.
    • project_id: required, a UUID. It always comes from the registry; a project_id or space_id in the request body is dropped, so callers cannot point the stored key at another project.
    • api_version: optional YYYY-MM-DD, default 2025-10-01.
    • base_url: optional override for private endpoints.

How it works

  • IAM tokens (watsonx/iam.go)
    • The API key is exchanged at iam.cloud.ibm.com. The token is cached per key, with keys held only as SHA-256 digests.
    • Tokens refresh at 80% of their lifetime, and concurrent callers share one exchange.
    • A still-valid token keeps being used through a transient IAM outage. A rejected key fails at once.
    • When watsonx answers 401, the call is retried once with a fresh token.
  • Wire format (watsonx/translate.go)
    • The gateway encodes OpenAI Chat Completions for watsonx (resolveProviderWireFormat). Policies, guardrails, cross-format callers (Anthropic, Gemini, Responses) and usage accounting therefore work unchanged.
    • The client converts to /ml/v1/text/chat and /ml/v1/text/chat_stream:
      • model becomes model_id, and project_id is added
      • tool_choice none/auto/required becomes tool_choice_option; a named function stays tool_choice
      • max_completion_tokens becomes max_tokens
      • system, assistant and tool content is flattened to text; user content parts, images included, pass through
      • only function tools are sent, and fields watsonx does not know are dropped
    • Coming back, responses and stream chunks are rewritten as OpenAI shapes:
      • tool-call arguments become JSON strings, and stream tool-call deltas get an index
      • finish reasons are mapped to OpenAI's
      • streams end with [DONE]; a watsonx event: error becomes an OpenAI in-band error chunk
      • error bodies become OpenAI error envelopes that keep IBM's trace id
  • Model list (watsonx/models.go)
    • Comes from the region's foundation_model_specs (text_chat). Withdrawn models are skipped and deprecated ones are labelled.
    • The listing is authoritative per region: regions serve different models (eu-de has 5 today, us-south 7). Prices are borrowed from the catalog where a model matches.
    • providerListingIsNotAuthoritative now includes watsonx, so routing never treats a model missing from models.dev as absent.
  • Pricing: models.dev's watsonx provider is synced into the catalog. Registry pricing.overrides (for example ibm/*) cover contract rates and models models.dev does not price.
  • Secret stores: a watsonx credential's destinations are the region's host (or base_url) plus iam.cloud.ibm.com, which receives the API key. This matches how GCP and Azure list their token hosts. The host rules live in pkg/domain/provider (WatsonxHost, WatsonxIAMHost), and the options decoder and IAM client read them from there.
  • Connection test: mints an IAM token, then lists the region's models with it.

Testing

  • The new package is at 89% coverage. It has tests for:
    • token refresh, concurrency, outage fallback and rejection
    • request, response and stream translation, including tool calls and error events
    • an end-to-end client run against stub IAM and watsonx servers, covering the retry on 401
    • a composition test running the gateway's real adapter: an Anthropic-dialect caller's tool call goes through watsonx and comes back as tool_use
  • Existing provider table tests are extended (locator, capabilities, option validation, models.dev sync with pricing, live listing, routing verdicts, host rules, secret-store destinations).
  • Local run on the merged branch:
    • go test -race ./...: 214 packages ok, 0 failures.
    • Functional suite with the CI command (go test -tags functional -race -p 1 ./tests/functional/... ./pkg/infra/database/migrations) against Postgres 16 and Redis: 717 passed, 0 failed, 4 skipped. The skips are environmental: no OPENAI_API_KEY, plan rate limiter disabled, needs a real TrustGuard, and RUN-1621.
    • golangci-lint run ./... (v2.14.0): 0 issues.
    • make docs with the pinned swag: no diff.
    • addlicense -check and scripts/check-comments.sh pass.

Not yet verified against live watsonx: no IBM Cloud API key was available. The payload shape follows IBM's ibm-watsonx-ai SDK, and the regional endpoints were confirmed reachable. Before merging, please run one buffered chat, one streamed chat and one tool call with a real key and project. That is why this PR is a draft.

CI: what changed after the first run

The first run failed Lint, OpenAPI Drift and Functional Tests. develop's own last CI run (88e4288) failed the same three jobs. This branch now:

  1. Merges current develop (a30823e). That brings in the fixed switch status in pkg/app/a2a/translated.go (staticcheck QF1002) and the regenerated docs/, which clears Lint and OpenAPI Drift.
  2. Fixes three A2A functional tests (4be294d, test-only). Since 53a9e19, registering a generic agent asks its endpoint for a task that does not exist. The fake agent recorded that call, so TestA2A_RefusesMethodsItDoesNotForward, TestA2A_PolicyBlocksTheMessage and TestA2A_AuthenticatesAndRoutes each saw one call more than they expected. The new forgetRegistrationCheck asserts the registration made exactly that one check (tasks/get on 0.3, GetTask on 1.0), then drops it. Gateway behaviour is unchanged.
  3. Gives watsonx host rules (9405502). After the merge, develop's new TestEveryProviderHasAHostRule failed for watsonx. This one was caused by this PR: without a rule, a secret store with allowed destinations would refuse every watsonx reference. See "Secret stores" above. TestHonorsBaseURLMatchesTheOptions now covers WatsonxOptions.

Known gaps / follow-ups

  • The connection test validates the key and region, not the project ID. A wrong project shows up as IBM's error on the first call.
  • Not supported yet:
    • deployment spaces (space_id)
    • the private IAM endpoint
    • Mumbai (ap-south-1), which runs on AWS with a different IAM
    • inbound watsonx-native API calls (clients use the OpenAI, Anthropic or Gemini dialects)

🤖 Generated with Claude Code

https://claude.ai/code/session_01KF89yDnBWqZ6yB3bLqAtb9

Adds a watsonx provider so watsonx.ai traffic can sit behind TrustGate
without IBM's model gateway.

- Registry auth is an IBM Cloud API key. The client exchanges it for an
  IAM bearer token, caches it per key (SHA-256), refreshes at 80% of its
  lifetime, shares one exchange across concurrent callers, keeps serving a
  still-valid token through a transient IAM outage, and retries a call
  once with a fresh token when watsonx answers 401.
- provider_options: region (us-south, eu-de, eu-gb, jp-tok, au-syd,
  ca-tor), project_id (UUID, always taken from the registry, never the
  request body), api_version (default 2025-10-01), optional base_url for
  private endpoints.
- The gateway encodes OpenAI Chat Completions for watsonx; the client
  translates to /ml/v1/text/chat and /ml/v1/text/chat_stream: model_id,
  project_id, tool_choice modes to tool_choice_option, max_completion_tokens
  to max_tokens, text-only system/assistant/tool content, function tools
  only, unknown fields dropped. Responses, stream chunks, tool-call
  arguments, finish reasons and errors come back OpenAI-shaped; streams get
  a [DONE] and a watsonx error event becomes an in-band error chunk.
- Model list: the region's foundation_model_specs (text_chat, withdrawn
  models skipped, deprecated ones labelled). The listing is authoritative
  per region, borrowing models.dev prices where a model matches; routing
  treats models.dev's partial watsonx catalog as unable to prove absence.
- Pricing: models.dev "watsonx" is synced into the catalog; registry
  pricing overrides cover contract rates and unpriced models.
- Connection test mints an IAM token and lists the region's models.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KF89yDnBWqZ6yB3bLqAtb9
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Trivy code scan — HIGH/CRITICAL

No HIGH or CRITICAL vulnerabilities found. (run)

…-native-integration-rtssu4

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KF89yDnBWqZ6yB3bLqAtb9
Registering a generic agent now asks its endpoint for a task that does
not exist (53a9e19). The fake agent records that call, so the three
tests that expect no forwarded call saw one more than expected:
TestA2A_RefusesMethodsItDoesNotForward, TestA2A_PolicyBlocksTheMessage
and TestA2A_AuthenticatesAndRoutes.

forgetRegistrationCheck asserts the registration made exactly that one
check (tasks/get on 0.3, GetTask on 1.0), then drops it, so each test
counts only what the gateway forwards afterwards. The gateway's
behaviour is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KF89yDnBWqZ6yB3bLqAtb9
develop's secret stores check each credential reference against a
store's allowed destinations, using a host rule per provider; the
watsonx provider predates that and had none, so
TestEveryProviderHasAHostRule failed and a store with allowed
destinations would have refused every watsonx reference.

- pkg/domain/provider owns the watsonx host rules: the regions that
  serve watsonx.ai, WatsonxHost(region) and WatsonxIAMHost. watsonx
  honors base_url. The options decoder and the IAM client read them
  from there instead of keeping their own copy.
- secretref sends a watsonx credential to the region's host (or
  base_url) and to iam.cloud.ibm.com, which receives the API key and
  returns the bearer token, as GCP and Azure list their token hosts.
  A region that serves no watsonx.ai leaves no host and fails closed.
- TestHonorsBaseURLMatchesTheOptions now covers WatsonxOptions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KF89yDnBWqZ6yB3bLqAtb9
@vgmartinez
vgmartinez marked this pull request as ready for review November 13, 2026 18:06
@vgmartinez
vgmartinez merged commit 931643e into develop Nov 15, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants