Tune LLM API infra, add benchmark/eval scripts, document in docs/AGENTS.md - #4
Open
elimelt wants to merge 1 commit into
Open
Tune LLM API infra, add benchmark/eval scripts, document in docs/AGENTS.md#4elimelt wants to merge 1 commit into
elimelt wants to merge 1 commit into
Conversation
…TS.md Caddyfile (llm.elimelt.com): - Raise rate limit 20 -> 100 req/min per IP - Exempt internal origins (*.elimelt.com, localhost:5173) from rate limiting - Restructure CORS/preflight and blocked-endpoint handling into handle blocks docker-compose.yml (ollama): - OLLAMA_ORIGINS=* (Ollama's own Origin check returned 403; policy is enforced by Caddy at the edge instead) - OLLAMA_NUM_PARALLEL=4 (concurrency benchmarks plateau at ~4 on CPU) - OLLAMA_MAX_QUEUE=20 for backpressure - Memory limit 50G (26B models OOM-killed the container at 6G) infra/scripts: - bench_ollama.py: TTFT/decode/prompt benchmark; distinguishes first token (incl. thinking) from first content, runs thinking models in both modes - bench_ollama_concurrency.py, bench_ollama_prefill.py - eval_tts_rewrite.py, eval_tts_prompts.py: quality evals for the notes -> TTS rewrite task docs/AGENTS.md: operational notes (model lineup rationale, thinking-model gotchas, benchmark results, CORS/rate-limit behavior) Model lineup changes (server-side, recorded here for history): pulled gemma4:e4b, gemma4:26b, gpt-oss:20b, qwen2.5-coder:7b; removed gemma3:27b and llama3.2:3b as Pareto-dominated.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Infrastructure tuning, benchmarking tooling, and operational docs for the LLM API at
llm.elimelt.com(Ollama behind Caddy).Changes
Caddyfile
https://*.elimelt.com,http://localhost:5173) exempted from rate limitinghandleblocksdocker-compose.yml (ollama)
OLLAMA_ORIGINS=*— Ollama's built-in Origin validation was returning 403; origin policy is enforced by Caddy at the edge insteadOLLAMA_NUM_PARALLEL=4— concurrency benchmarks showed CPU-bound throughput plateaus at ~4OLLAMA_MAX_QUEUE=20— backpressure instead of unbounded queueingNew scripts (
infra/scripts/)bench_ollama.py— TTFT/decode/prompt benchmark; distinguishes first token (incl. thinking) from first content, and runs thinking-capable models (gemma4, gpt-oss) in both modesbench_ollama_concurrency.py,bench_ollama_prefill.pyeval_tts_rewrite.py,eval_tts_prompts.py— quality evals for the notes→TTS rewrite taskDocs
docs/AGENTS.md— model lineup rationale, thinking-model gotchas, benchmark results, CORS/rate-limit behaviorServer-side model changes (not in repo, recorded for history)
Pulled
gemma4:e4b,gemma4:26b,gpt-oss:20b,qwen2.5-coder:7b; removedgemma3:27bandllama3.2:3bas Pareto-dominated. Final lineup benchmarks (warm, medians of 3):gemma2:2bgemma4:e4b(think: false)gemma4:26b(think: false)gpt-oss:20b(think: "low")qwen2.5-coder:7bVerification
https://llm.elimelt.com/v1/chat/completionslocalhost:5173and*.elimelt.com