Repository navigation
Add support for OpenAI models as a backend #1605
Description
Integrate OpenAI's models into the Gemini CLI, allowing users to select OpenAI as an alternative model provider. This would increase the tool's flexibility and allow ranking model outputs with the same tool. This also attracts users who are already invested in the OpenAI ecosystem and have API keys. Like Kubernetes, but for models in the CLI.
Proposed Implementation Plan
I propose creating a provider-agnostic abstraction layer. This will prevent tight coupling to a single provider and make it easier to add other providers (e.g., Anthropic, xAI, etc) in the future.
The work can be broken down as follows:
-
Configuration Update:
- Extend the configuration in
packages/core/src/config/config.tsandpackages/cli/src/config/config.tsto include:provider: A new field to select the model provider (e.g.,"google" | "openai").openaiApiKey: A field to store the OpenAI API key.
- Update the auth flow in
packages/cli/src/config/auth.tsto support reading the OpenAI key from an environment variable (OPENAI_API_KEY) and storing it securely.
- Extend the configuration in
-
Provider Abstraction Layer:
- In
packages/core/src/core/, create a newLanguageModelProvider.tsfile defining a genericLanguageModelProviderinterface. - This interface will standardize the methods for generating content, handling streaming, and managing tool calls, creating a common contract for all providers.
- In
-
Refactor Existing Gemini Integration:
- Create a
GeminiProvider.tsclass that implements theLanguageModelProviderinterface. - Migrate the existing Gemini-specific logic from
geminiRequest.tsandclient.tsinto this new class. - Refactor
geminiChat.tsto depend on theLanguageModelProviderinterface rather than the concrete Gemini implementation.
- Create a
-
Implement OpenAI Provider:
- Add the official
openainpm package as a dependency topackages/core. - Create a new
OpenAIProvider.tsclass that also implements theLanguageModelProviderinterface. - This class will be responsible for:
- Translating the application's generic request format into the format required by the OpenAI Chat Completions API.
- Mapping tool definitions and function-calling requests/responses between the generic format and OpenAI's format.
- Translating the OpenAI API response back into the application's generic response format.
- Add the official
-
Provider Factory:
- Create a factory function that reads the user's configuration and instantiates the appropriate provider (
GeminiProviderorOpenAIProvider). - The core chat logic in
geminiChat.tswill use this factory to get the active model provider.
- Create a factory function that reads the user's configuration and instantiates the appropriate provider (
Rollout
- PR 1: Core Abstraction & Refactoring: Introduce the
LanguageModelProviderinterface and refactor the existing Gemini code to use it. This PR will contain no user-facing changes but sets the foundation. - PR 2: OpenAI Provider Implementation: Add the
OpenAIProvider, configuration options, and factory logic. This PR will make the feature available to users. - PR 3: Documentation: Update the documentation in the
/docsdirectory to explain how to configure and use the new OpenAI integration.
Happy to discuss this proposal further and open to feedback or alternative approaches the maintainer team might suggest.
Thank you for considering this feature!
Activity
cperry-goog commentedon Jun 25, 2025
I appreciate this issue, and grateful to you for filing it. We're not quite ready to work on adding support for other models at this point in time, as you will see we're tightly coupled to Gemini's capabilities. We'd like to focus on making a great tool for Gemini.
I aspire to make this work in the longer run, I'd like to keep this open as the canonical issue for other model support we can dupe others into.
gerrywastaken commentedon Jun 25, 2025
Somebody may already be forking this to create a billion dollar company. The Cursor of CLI tools with support for all models. Anybody still using VSCode?
arpanpreneur commentedon Jun 26, 2025
An OpenAI compatible endpoint support would go a long way into supporting most Open Source models who are exposed as OpenAI compatible endpoints. This would help to use this tool in a completely private environment.
tarruda commentedon Jun 27, 2025
Yes, it makes sense to create the tool in a way that makes the best use of Gemini capabilities. However, in the end this is just an agent that provides a system prompt plus a set of tools to the LLM, so it can be used with other models even if Gemini continues to be the main focus. There's a few ways with which this project could allow for other models without explicitly supporting it or adding maintenance overhead.
For example, would you consider accepting a cheap abstraction to hide all Gemini-specific code behind a plugin interface, where Gemini support could be implemented as a builtin and default plugin (and maintained with the rest of the code), but it would allow loading other LLM plugins with code from other repositories (in a way that requires explicit configuration, without first class UI support)?
In a way Gemini-cli already provides some level of abstraction through the
ContentGeneratorinterface (implemented for all gemini endpoints byCodeAssistServer), but there's a few ways it could be improved:ContentGeneratorspecific concernContentGeneratorspecific implementations for vertex, google auth and gemin api keyJust restructuring the code in this direction would make it much simpler for anyone to add support for other LLMs/providers in their own forks. Plus Gemini-cli could take this a step further by adding explicit support for loading
ContentGeneratorimplementations from external plugins (thus avoiding fork fragmentation).ksecurity45 commentedon Jun 29, 2025
Much needed future. Else we cant use it in our workflow.
acoliver commentedon Jul 2, 2025
I'm really excited about Gemma-CLI. Google has created a compelling tool and by making it open source they've opened up a lot of potential! Unfortunately right now it isn't viable as a competitor to Claude Code (for $200 you get all you can realistically eat without throttling), it immediately throttles you to flash which isn't good enough for most things you'll want to use it for (if you're smart enough to use a CLI then you're not working on code that flash can handle). It also unfortunately doesn't work with other models (especially local, fireworks or openrouter).
To alleviate those issues I've enhanced it with full multi-provider support here: https://github.com/acoliver/llxprt-code
** completions (older models and open models like gemma)
** responses (more efficient for newer models, required for o3-pro but much lower token usage/discounted for 4x, o3-o4)
** use Claude through gemini-cli - pay a lot more than through claude code with ultra subscription but the point is you can
There are specific tool parsers for Gemma and I've tested locally against lmstudio. I plan to add the rest.
As soon as I'm done testing deepseek and qwen, I'll submit a pull request but Google has already closed multiple versions of this feature request and said it isn't in their immediate plans (my paraphrase). They've also set up a kind of "reject first, ask questions later" system for pull requests, so I don't anticipate acceptance. Assuming they refuse, I'll do a RooCode vs Cline style fork and we'll have ours.
I'm open to collaboration for anyone interested :-)
six-ddc commentedon Jul 6, 2025
One potential approach could be to introduce the ai-sdk as an intermediary layer for model capability provisioning, which would enable compatibility with nearly all AI providers.
acoliver commentedon Jul 6, 2025
I submitted as a PR #3405 - it does this and more.
tarruda commentedon Jul 6, 2025
Quite a bit more, it seems?
No offense, but I find it hard to imagine someone will review a PR with 40k added lines.
Have you seen #1975? That seems to work pretty well and the added line count is an order of magnitude lower than in your PR.
acoliver commentedon Jul 6, 2025
Sure @tarruda - here is Gemini 2.5 Pro's analysis -
Analysis of PR #1975
The Strategic Argument for #3405 (Now Stronger)
This correction makes the argument for Pull Request #3405 even more compelling. Here’s the refined reasoning:
They Are Already Halfway There, But Unofficially: The existence of openai_compatible.rs proves that Google recognizes the need to interact with the broader ecosystem. However, it is clearly a
partially implemented or unofficial feature. PR support openai like api #1975 is a necessary patch on this unofficial component.
PR Multi provider #3405 Makes This Vision Official and Robust: Pull Request Multi provider #3405 takes their hidden, buggy implementation and elevates it into a fully-fledged, officially supported, and core pillar of the
gemini-cli. It provides:Gemini, OpenAI, Anthropic, and any number of OpenAI-compatible endpoints.
multi-provider architecture.
feature slightly less broken.
acoliver commentedon Jul 19, 2025
if you want to use the fork you can try
Install
npm install -g @vybestack/llxprt-code
npx @vybestack/llxprt-code
docker run -it ghcr.io/acoliver/llxprt-code/sandbox:0.1.12
TommrraraSnow commentedon Jul 23, 2025
Qwen has made a OpenAI compatible version called qwen-code: qwen-code
mattKorwel commentedon Aug 1, 2025
Hi all. Thanks for taking time to share thoughts and discussions around this topic. At this time we're focused on just Gemini, per #1974 (comment). Closing this out for now. I'll follow up here if anything changes.
acoliver commentedon Aug 1, 2025
As much as I would have disagreed with this originally, I think Google has proven correct here. They are doing a lot of work making Gemini-CLI work well with a lot of IDEs and tools.
LLxprt takes that work (we keep up with their main) and adds multi-model/multi-provider and local model support as well as agentic development features like Claude code style, to-do lists. https://github.com/acoliver/llxprt-code - This is a better process - Google can control their roadmap internally, and the community can have the multi-provider, more agentic version via LLxprt Code. It's like Gemini-cli is the Linux kernel, and LLxprt and Qwen-code are different operating systems based on it.
Address PR #240 bot comments: gemini scope honesty + ordering fix
Add custom LLM API endpoint support to agent-cli-wrapper (#240)