Repository navigation
CodeGate doesn't work with either LiteLLM or OpenRouter #878
Description
Activity
I think how it works aldy
As my litellm proxy uses os.environ/GEMINI_API_KEY for the config.yaml file, I need to pass in GEMINI's api key instead of LiteLLM master key: https://docs.litellm.ai/docs/providers/gemini
Extract of sample
config.yamlfile for my self-hosted litellm proxymodel_list: - model_name: gemini-pro litellm_params: model: gemini/gemini-1.5-pro api_key: os.environ/GEMINI_API_KEYHere is the amendment I made for CodeGate to work:
Should be working now, and I will close this issue.
But, it seems like leading to another error, which it doesn't trigger when there are secrets inside the file:
Here is the same credentials in my
config.inifile
And when I check codegate via docker logs, and I found a lot of errors there:
And not sure if it's related to the errors above, I found there are some 304 errors for my litellm proxy's UI? But, I think I didn't access my litellm proxy's UI that time.... (Not that sure)
I am thinking it's better for me to try to configure it via openai API first, and then transfer to litellm API ...
@tan-yong-sheng thanks for filing the issue!
The rate limits were a codegate bug that was fixed in main but not released yet. We'll get a release out shortly.
I've not used the litellm proxy myself, but I assume it works somewhat like openrouter? Sounds like you expected the litellm master key to work? If yes, it might be worth to reopen this issue and investigate.
I see, noted on the info and thanks.
Yes, litellm proxy is kinda similar to openrouter which is to unify the llm endpoints from different providers.
Ya, I previously expect LITELLM_MASTER_KEY to work, but in the end, I found I need to pass the GEMINI_API_KEY here I want to Gemini llm. Also, I could pass in MISTRAL_API_KEY if I want to Mistral llm. It means LITELLM_MASTER_KEY not being used to fill in the image below, but it still works (note: perhaps this is how litellm proxy works and I would add on more info about this when I am available)
As shown in my previous images:
ok, reopening for further triage. We need to make sure codegate works well without the workarounds.
Thanks again for finding the bug and filing the issue.
Hi @jhrozek, could I request any docker container (e.g., ghcr.io/stacklok/codegate:test) for testing? I am not that sure if I am doing right to build the docker image for codegate's main repository:
I tried to
git clone https://github.com/stacklok/codegate.gitand then follow your guide to build docker container on my linux cloud, as shown below:
Yet, when I use cline (even with openai endpoint: https://api.openai.com/v1), it still couldn't trigger and detect credentials. The prompt I used to reproduce the error is
read config.ini file
However, for docker logs, no error logs for codegate's docker container as before:
Hi @jhrozek, could I request any docker container (e.g., ghcr.io/stacklok/codegate:test) for testing? I am not that sure if I am doing right to build the docker image for codegate's main repository:
I don't think you need to build a local container (unless you want to!) we release several times a month and the current release is just a couple of days old.
I tried to
git clone https://github.com/stacklok/codegate.gitand then follow your guide to build docker container on my linux cloud, as shown below:Yet, when I use cline (even with openai endpoint: https://api.openai.com/v1), it still couldn't trigger and detect credentials. The prompt I used to reproduce the error is
read config.ini fileI think this is a configuration issue. In order for cline to talk to codegate, you should point it to codegate as the base URL, not to the LLM
However, for docker logs, no error logs for codegate's docker container as before:
I think this would be solved by pointing the extension to codegate.
btw does it still stand that the main issue is not being able to use litellm through the main key?
Reacted by Tan Yong ShengI see, thanks a lot. I will test again when the bug (= codegate is not triggered when I ask LLM to read config.ini file to safeguard my credentials) is fixed.
Thanks a lot for your efforts in delivering this product. Tbh, I am impressed with the idea of this project to safeguard credentials and filter risky dependencies when doing AI coding.
btw does it still stand that the main issue is not being able to use litellm through the main key?
Ya, initially I thought we can't use LiteLLM proxy with Cline and Codegate, and it keeps saying authentication error. That's the reason I create this issue.
But, for now, I think it's not that much matter, as I found the workaround to solve this issue. We could use the GEMINI_API_KEY or CODESTRAL_API_KEY for respective LLM models, instead of LITELLM_MASTER_KEY. And it works!
Demonstration
Just for your info, here is what I did to use Cline (together with LiteLLM proxy) with and without codegate:
(1) WITHOUT codegate and with LiteLLM proxy
(2) WITH codegate and with LiteLLM proxy
Reply to issue
I hope this explanation is clear. I believe we can maintain the existing configuration, since we have a workaround to enable Cline and Codegate to work together with LiteLLM. Therefore, I suggest closing this issue.
Additional info
(i) the
config.yamlfor LiteLLM proxyYou could ignore this, but it's for clarify for my setup to self-host LiteLLM proxy on my Linux cloud server. Here is what I setup for
config.yamlfile for my litellm proxy to access Gemini and Codestral models:model_list: - model_name: gemini/* litellm_params: # to include all gemini models available model: gemini/* api_key: <GEMINI_API_KEY_IN_TEXT_FORM> - model_name: codestral/* litellm_params: # to include all codestral models available model: codestral/* api_key: <CODESTRAL_API_KEY_IN_TEXT_FORM>Reference: https://docs.litellm.ai/docs/proxy/configs
(ii) How to run codegate via docker container
Note: I change CODEGATE_OPENAI_URL from https://api.openai.com/v1 to https://<MY_LITELLM_URL>/v1
docker run --name codegate -d -p 8989:8989 -p 9090:9090 -e CODEGATE_OPENAI_URL=https://<MY_LITELLM_URL>/v1 --mount type=volume,src=codegate_volume,dst=/app/codegate_volume --restart unless-stopped ghcr.io/stacklok/codegate
3 remaining items
Looping back to the secrets sensing, we would need to pattern match those tokens, you will find the matching list here and we currently match the crypto itself, not the variable name.
Ah I see, no wonder.
However are just about to rework this system to match both #209 and also look and use of entropy for matching.
Interesting ideas and discussions
Would love to have you try this out when we have something in the way of a prototype?
Yaya, should be no problem. My pleasure.
Thanks as well.
@tan-yong-sheng I opened a PR that fixes a very similar issue for OpenRouter. I think fixing your issue for litellm would be along the same lines. Can you suggest an easy way to test litellm proxy to simulate your environment? I found https://docs.litellm.ai/docs/proxy/deploy so I can try that - I'm familiar with using litellm as a library, but not as a proxy
Here is the the way to start an localhost instance for litellm. But I am not that sure if this is working as I never tried localhost version one (note: not available at the moment to test this before writing)..
Reference: https://thinhdanggroup.github.io/litellm-proxy/#setting-up-litellm-proxy-locally
- Install pypi package
pip install litellm[proxy]- Then, on your directory, create config.yaml file
(Note: I get Gemini API key for free in Google ai studio)
model_list: - model_name: gemini/* litellm_params: # to include all gemini models available model: gemini/* api_key: <GEMINI_API_KEY_IN_TEXT_FORM> drop_params: true - model_name: codestral/* litellm_params: # to include all codestral models available model: codestral/* api_key: <CODESTRAL_API_KEY_IN_TEXT_FORM> drop_params: true general_settings: master_key: sk-1234(Note 1: You could refer https://docs.litellm.ai/docs/proxy/configs for the format of config file, if you need to add more configuration)
(Note 2: your LITELLM_MASTER_KEY is sk-1234 set by config above)- start litellm instances
litellm --config=config.yaml- At last, you could use either openai or litellm module to call the llm API
https://docs.litellm.ai/docs/proxy/user_keys
For example,
import openai client = openai.OpenAI( api_key="anything", # can try to pass in LITELLM_MASTER_KEY first base_url="http://localhost:4000" ) # request sent to model set on litellm proxy, `litellm --model` response = client.chat.completions.create( model="gemini/gemini-2.0-flash-exp", messages = [ { "role": "user", "content": "this is a test request, write a short poem" } ], ) print(response)Let me know if you need further help, thanks.
- added 2 commits that reference this issue
on Feb 5, 2025 Hi, just wanna check if the method above works to use litellm proxy on localhost (in linux environment)? Let me know if you need any help.
Hi, just wanna check if the method above works to use litellm proxy on localhost (in linux environment)? Let me know if you need any help.
Thank you for the pointers. We merged the the openrouter patches last week and the litellm support should be along the same lines. I got sidetracked by other work, but I'll try to get this back on track soon!
If other codegate developers want to take this issue from me, feel free to.
Reacted by Tan Yong Shenghey, @tan-yong-sheng 👋
So I've tried an example setup locally and I think I was able to reproduce the issue.
TLDR: The main issue is that our internal dependency on litellm (not the proxy, but the library which we are in the works of removing 🥳 ) re-routes on its own the model provider URL based on the model name. We are expecting to release a fix for this in the next 1 or 2 releases.
The setup that I have is the following:
- Cline: Configured to talk to CodeGate's OpenAI compatible API (note the model name is what LiteLLM Proxy expects it)
- CodeGate: Configured to talk to LiteLLM's Proxy (running locally on
localhost:4000) by overwriting the OpenAI URL flag
<codegate serve command> --openai-url=http://localhost:4000- LiteLLM Proxy: Configured to talk to a local Ollama server (but can be configured to other providers as well)
model_list: - model_name: phi4 litellm_params: model: ollama/phi4 # LiteLLM expects the models to be prefixed by their provider api_base: http://localhost:11434 # This is where Ollama is serving general_settings: master_key: sk-1234 # That secret is used in the Cline config for API key
The issue:
- This should work, but our internal use of LiteLLM as a library tries to be smart in this case and once it sees the prefixed model name (
ollama/phi4) it overwrites the provider URL from what we configured above via--openai-url=http://localhost:4000tohttp://localhost:11434instead essentially bypassing the LiteLLM Proxy. - It does the same for OpenAI or other providers too, which is why using your LiteLLM key doesn't work, but using the OpenAI key for example works.
Solution:
- The good news is @blkt and @jhrozek are already in the process of replacing our internal use of LiteLLM with our own implementation which will simplify the codebase on one side, but also resolve this and other issues caused by LiteLLM.
I'll ping you once this is released 🙏 The work is almost complete so it should drop in some of the next releases.
Hi, sorry for missing this and being for reply.
I'll ping you once this is released 🙏 The work is almost complete so it should drop in some of the next releases.
Sure, thanks a lot.
hi @tan-yong-sheng , we have completely removed LiteLLM now, so I will close this issue, re-open if you are still having problems.
Hi @lukehinds, thanks a lot for your help. Yes, it could work if I put my LITELLM_MASTER_KEY as credentials for codegate now.
(Note: I build the docker image myself on the github's main repo)
I am trying to use codegate with cline, and connect LLM api via LiteLLM proxy (hosted on cloud)
Environment:
Here are the steps I did:
docker run --name codegate -d -p 8989:8989 -p 9090:9090 -e CODEGATE_OPENAI_URL=https://<MY_LITELLM_URL>/v1 --restart unless-stopped ghcr.io/stacklok/codegateI thought as I set the environment variable to CODEGATE_OPENAI_URL=https://litellm.tanyongsheng.site/v1, it will change the default environment variable for openai url
However, I still get this error:
And when I tried to switch to gemini 2.0 that I set up in litellm proxy. There is still error coming up:
Hope for help, thanks