feat(selfhost): GPU + AI-provider observability (exporter, Prometheus, dashboard) - #4536
Merged
Merged
Conversation
…, dashboard) - gpu-exporter service (utkuozdemir/nvidia_gpu_exporter:1.3.2), own --profile gpu, matching the ollama/postgres-exporter style - gpu Prometheus scrape job, absent/down gracefully on non-GPU installs - New gittensory_ai_provider_request_duration_seconds histogram and gittensory_ai_provider_request_errors_total counter, labeled by provider and request_kind, instrumented around the existing withReviewSpan call site in runProviderWithOtel — the error counter mirrors gittensory_ai_provider_failures_total's exemption for expected embedding-routing fallbacks - grafana/dashboards/gpu-metrics.json: GPU Summary (utilization/VRAM/ temp/power stats), AI Request Performance (rate/p50-p95-p99 latency/ error rate), and GPU Trends rows - Resource-footprint table entry in the self-hosting operations docs This is the baseline the rest of the GPU-optimization work (embedding cache, advisory routing, vision-verify) needs to measure real impact against, rather than one-off manual benchmarks.
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
gittensory-ui | 0f71bc6 | Commit Preview URL Branch Preview URL |
Jul 09 2026, 11:26 PM |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4536 +/- ##
=======================================
Coverage 94.04% 94.04%
=======================================
Files 422 422
Lines 37579 37584 +5
Branches 13729 13729
=======================================
+ Hits 35340 35345 +5
Misses 1583 1583
Partials 656 656
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
gpu-exporterservice (utkuozdemir/nvidia_gpu_exporter:1.3.2), own--profile gpu, matching the existingollama/postgres-exporterstyle (<<: *default-logging,exposenotports, device reservation only evaluated once the profile is activated)gpuPrometheus scrape job — absent/down gracefully on a non-GPU install, no alert fires on it, mirroring the existingpostgres/gittensory-backupjob commentsgittensory_ai_provider_request_duration_secondshistogram +gittensory_ai_provider_request_errors_totalcounter insrc/selfhost/metrics.ts, labeled byprovider/request_kind, instrumented inrunProviderWithOtel(src/selfhost/ai.ts) around the existingwithReviewSpancall — the error counter mirrorsgittensory_ai_provider_failures_total's exemption for expected embedding-routing fallbacks (verified via 4 new tests)grafana/dashboards/gpu-metrics.json: GPU Summary (utilization/VRAM/temp/power stat panels), AI Request Performance (rate, p50/p95/p99 latency viahistogram_quantile, error rate), GPU Trends (all four metrics over time, including VRAM used-vs-total).env.examplevar — nothing here needs one; the gpu-exporter has no tunable settings worth exposing, matching sibling exporters)Notes
nvidia_smi_utilization_gpu_ratio,nvidia_smi_memory_used_bytes/_total_bytes,nvidia_smi_temperature_gpu,nvidia_smi_power_draw_watts) and its default port (9835) verified against the exporter's own docs, not guessedrequestKind()stays"embedding" | "review"for now — extending it to vision/advisory is real future work once those call paths are distinguishable at this exact function, not done speculatively hereTest plan
npx vitest run test/unit/selfhost-ai.test.ts test/unit/selfhost-metrics.test.ts(182 tests)npx tsc --noEmitclean on touched files (pre-existing unrelated drift inminer-opportunity-ranker.test.tsuntouched)docker compose config --quietresolves cleanlyPart of #4325 (closed). Fixes #4367