Repository navigation
Muxing Support for Model Routing #566
Description
Activity
Filter by file: can do support wildcards (eg doc*.txt), as well as specific file names?
Would we also want to know the directory structure too?Would we want to specify a version of a model, or something broader (eg claude_sonnet vs claude_sonnet_3.5).
What would happen if 3.5 wasn't available?Would users select their repo, or just type it in (eg can we see a list or is it just manual?)
Would this be on a global level or per workspace?
Filter by file: can do support wildcards (eg doc*.txt), as well as specific file names?
I think it would be nice, I can envision being OK to send
tests/database.sqlbut notprod/database.sql(or anything under prod)Would we also want to know the directory structure too?
We would want to but I don't think it was easy. Tagging @aponcedeleonch and @JAORMX who were looking into this. I suspect it's not fully possible until codegate is hooked into the IDE, but I don't know 100%
(Similar to the other reply, this is my 2 öre, not a canonical reply)
Would we want to specify a version of a model, or something broader (eg claude_sonnet vs claude_sonnet_3.5).
This is a question about model aliases right_
What would happen if 3.5 wasn't available?
I think we could start with just being specific - in other words, just require the full model name to by typed. Many providers have aliases anyway, e.g. anthropic supports
claude-3-5-sonnet-latest. OpenAI has the same. The "major versions" are usually different enough I would let the user type the version.But I can also see that codegate having some logic to pick the best model might be good, e.g. "for simple requests, use anthropic haiku, for more complex changes use anthropic sonnet, for very complex tasks use anthropic opus".
We would want to but I don't think it was easy. Tagging @aponcedeleonch and @JAORMX who were looking into this. I suspect it's not fully possible until codegate is hooked into the IDE, but I don't know 100%
That's right, codegate can't know unless it's actually hooked into the IDE, which it isn't
First pass at model routing UI, love to hear your feedback, or if I have misunderstood anything
Designs
https://www.figma.com/design/TtIS3Uk8qBdoNijeezfJYl/Dashboard?node-id=156-8633&t=TJnz3nwd0u64aokC-11)6 remaining items
The work on this issue has finished. The next enhancement is tracked in #1059
Introduce support for "Muxing" in CodeGate, enabling users to route specific types of logic to different large language models (LLMs) based on file type, individual files and workspace. This would provide cost savings , a common complaint from users is the expense of tokens and how to optimise those costs along with a way of isolation use to specific models (based on privacy, security , protection of IP)
It would also enable context switching (context window refers to the amount of text data a language model can consider at one time when generating responses) , whereby we could switch models during a long protracted session of prompt / response.
For example:
.md,.rst) could be processed by a free, local model, while source code files (e.g.,.py,.js) are handled by an advanced model like Claude Sonnet 3.5.Why is this feature important?
Possible Solution
Expand CodeGate’s functionality to include:
Repository-Level Configuration:
local_model, Repository B ->advanced_model.File-Type-Based Routing:
.md->local_model,.py->claude_sonnet_3.5.Per-File Routing:
config.py->local_model_v2.Challenges and Considerations