Repository navigation
allow llama serve -hf to use llama-server in router mode #26116
Description
Allow llama serve -hf to use llama-server in router mode
Behavior:
llama serve -hfwill run a new llama-server in router mode if one doesn't yet exist- if there was already a server, then reuse it
- models that don't yet exist in cache should be downloaded automatically
Some considerations (TBD):
- server should run on which port? --> port 9931 == "ggml"
- maybe add a "model name resolve" API; for example, input
user/modelwithout tag resolve touser/model:Q4_K_M
(this is an assigned issue; contributor should not take it)
Metadata
Metadata
Assignees
Labels
No labels
Activity
EJainDev commentedon Aug 3, 2026
How will this work (especially with on-demand downloads) for model configuration? Right now it's a fixed cache so a
models.inihandles the configuration fine.woof-dog commentedon Aug 16, 2026
I would like to point out that I currently enjoy the current
llama-serversetup in which I feel like I have control over which models I have and where I got the models from (not necessarily Hugging Face). I also have control over howllama-serveris run.I am personally not interested in having it run as a daemon only, or in having another ollama. Being able to Ctrl+C the process and view the logs as they stream in are actual UX benefits I rely on, especially when other software is accessing the llama.cpp server.
mcneiljt commentedon Aug 17, 2026
Echoing @woof-dog a bit here.
Is there more context on why this is changing? Can this change be made without dramatically affecting how
llama-serveroperates today?stevecs commentedon Aug 23, 2026
Just a comment here as I just started using llama.cpp directly, but saw the note on the default port change from 8080 -> 9931 but doing a quick search did NOT see any hits/lookups at IANA that this new port number was going through the registration process. Any port < 49152 should be registered if used on a project and is for a unique application/service/function. This helps identification and management of flows through application aware firewalls (so custom service checks, etc can be assigned or excluded for particular ports). Since this seems to be just http traffic but on a different port would argue against changing this from 80/8080 normal web ports or if needed to a port > 49152 as otherwise if unregistered will just be laying a future land-mine when that port IS registered via the correct channels.
ngxson commentedon Aug 23, 2026
To clarify:
llama-serverwill work as-is, the only thing change isllama servecommand, as the title of this issue suggestedThe change from 8080 to 9931 is because 8080 is a generic port that might be used by other apps. This is mostly for better UX, not just a technical change. That being said, (1) many other apps also use ports not registered on IANA and (2) even if there are conflict, usually with too obscure app that less than 1% users gonna use, and (3) user can always set explicit
--port, that option is not going anywhereSo I don't see any other blocking issues for this feature.
woof-dog commentedon Aug 23, 2026
Thank you for clarifying. But many of us likely came from #26508 after noticing the server log, and in that PR you wrote that this issue was
which sounds exactly like
llama-serverwas being replaced withllama serve.So, I am glad you clarified this is not actually the case.
ngxson commentedon Aug 23, 2026
the only thing gonna change on
llama-serverwill be just the--portbecause defaulted to 9931. users can always setLLAMA_ARG_PORT=8080env var to bashrc (or similar) force it use the port 8080 without explicit--portgithub-actions commentedon Oct 7, 2026
This issue was closed because it has been inactive for 14 days since being marked as stale.
danielzgtg commentedon Oct 18, 2026
I am annoyed that the port changed for not good enough reasons. A port change would force me to inconveniently reconfigure my firewall, bookmarks, IDEs, and other agents.
Most people were previously fine with port 8080. I think that the burden should be placed on whoever wants port 9931 to specify that option.
danielzgtg commentedon Oct 19, 2026
I've just hit a hard problem. Firefox's localhost AI is hardcoded to
localhost:8080. I can't find a way to reconfigure it.So now when people update llama.cpp, their Firefox AI will stop working. If they're following instructions older than 2 months, it'll lack
--port 8080and they'll be confused. Furthermore, I'm not going to submit a patch to Firefox, because another OpenAI-compatible LLM server might be still using port 8080.ArclightMat commentedon Oct 19, 2026
@danielzgtg Considering enabling localhost as a chat provider in Firefox isn't exposed in the normal UI, I assume you're aware of
about:config. You just need to updatebrowser.ml.chat.providerto the new port, there is no need to patch upstream behavior.It is not hardcoded, and the upstream default could be changed if Mozilla finds it sensible enough.
This default did work for llama.cpp because it has a WebUI on the same port as the API, otherwise if it were vLLM, TabbyAPI, etc. you would need to put your frontend on port 8080 or change the config flag either way. Pointing a plain OpenAI compatible API wouldn't work at all. And again, this isn't exposed to end users, so they must be aware on how to configure things.