Skip to content

Gateway proxy: serve /v1/embeddings for OpenAI and more providers #503

Closed

Description

@vgmartinez

Goal

Serve embeddings through the gateway proxy so clients can call POST /{consumer}/v1/embeddings against more upstream providers (same governance/observability path as chat).

Out of scope: semantic cache / LB internal embedding creators (pkg/infra/embedding/*). Victor (2026-08-22): do not expand that stack for this issue.

What already works (proxy)

  • Route + capability: /{consumer}/v1/embeddings → CapabilityEmbeddings + FormatOpenAIEmbeddings
    (pkg/api/resolver/proxy_path_resolver.go)
  • Invoker: adapts via AdaptEmbeddingRequest/Response, then calls providers.EmbeddingsClient.Embeddings
    (pkg/app/proxy/provider.go)
  • Adapters: OpenAI embeddings + Cohere embed (+ rerank on /v1/rerank)
  • Cohere only implements EmbeddingsClient today — functional test TestCohereProvider_EmbeddingsCrossFormat proves OpenAI-shaped client → Cohere upstream

Gap

Calling /v1/embeddings against OpenAI / Azure / openaicompat / other registries fails with provider does not support embeddings because those clients only implement chat (Completions / CompletionsStream).

Target format today: Cohere → FormatCohereEmbed; everyone else → FormatOpenAIEmbeddings.

Work (PR-only, after #502 chat providers if desired — but this can ship independently)

  1. OpenAI client: implement Embeddings → POST …/v1/embeddings (honour base_url)
  2. Azure: embeddings deployment URL
  3. openaicompat: …/embeddings on configured base_url (Together/Fireworks/vLLM/Ollama/etc.)
  4. Optional next: Bedrock Titan embed, Google, Voyage, …
  5. Functional tests (mirror Cohere coverage) + docs/curl
  6. Catalog/UI: mark providers that support embeddings capability

Acceptance criteria

  • OpenAI registry serves POST /{slug}/v1/embeddings end-to-end
  • At least Azure and openaicompat also support proxy embeddings (or document why blocked)
  • Existing Cohere embeddings + rerank paths stay green
  • Docs: curl examples for OpenAI-shaped and Cohere upstreams
  • PR only (never merge from automation)

Related

Activity

  1. changed the title [-]Support embeddings endpoint across more providers[/-] [+]Gateway proxy: serve /v1/embeddings for OpenAI and more providers[/+] on Aug 22, 2026
  2. vgmartinez commented on Aug 22, 2026

    @vgmartinez
    ContributorAuthor

    Architecture alignment (2026-08-22, shared with #506–#510)

    When implementing OpenAI/Azure/openaicompat embeddings, prefer pre-filtering the consumer’s CandidateSet to registries whose provider implements EmbeddingsClient, instead of only failing at invoke with a retryable “does not support embeddings” (today’s pattern). Pinned incapable registry should be terminal. Keeps embeddings consistent with the multimodal capability model: capability on provider → registry inherits → consumer pool filters.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions