Repository navigation
feat(observability): extend codex dashboard + add spike alerts - #203
Merged
Merged
Conversation
Member
Author
|
@codex review |
|
Codex Review: Didn't find any major issues. 🎉 ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
added 2 commits
May 13, 2026 02:07
Builds on gitops PR #201: Dashboard (openclaw-codex-usage-dashboard ConfigMap): - Headline row: 24h totals (requests/prompt/completion/cached) + cache-hit ratio stat with 0.3/0.6 thresholds - 7-day trend row: prompt tokens/hour and requests/hour, useful for baseline drift and weekday pattern detection - Per-command Loki bargauges: top-15 commands by prompt tokens and by request count over 24h (replaces "only-logs" attribution) VMRule openclaw-codex-alerts (new file): - OpenclawCodexPromptTokenSpike >25k tokens/min for 5m (warning) - OpenclawCodexRequestSpike >20 req/min for 5m (warning) - OpenclawCodexMetricsAbsent absent series 30m (info) The 25k tokens/min threshold sits below the known ChatGPT Plus soft-limit (~30k tok/min) so we get a leading indicator before the well-known "hit your ChatGPT usage limit" wedge. ArgoCD `monitoring` Application already sources both grafana-dashboards/ and vm-rules/ directories — no app wiring change needed.
…th provider label
mctl-openclaw#20 (in flight) repartitions the /metrics counters by
provider label (codex + anthropic) so the dashboard catches model
fallback traffic. This PR updates the gitops side:
Dashboard:
- ConfigMap rename: openclaw-codex-usage-dashboard → openclaw-llm-usage-dashboard
- Dashboard uid: openclaw-codex-usage → openclaw-llm-usage
- Title: "openclaw — Codex Token Usage" → "openclaw — LLM Token Usage"
- All `openclaw_codex_*` queries → `openclaw_llm_*`
- Rate panels: sum by (pod) → sum by (pod, provider); legend
"{{pod}}" → "{{pod}} {{provider}}"
- Headline 24h stats: sum(...) → sum by (provider) (...); legend
"requests" → "{{provider}} requests" (same for prompt/completion/cached)
- 7d trend: same per-provider grouping
- Cache hit ratio stat keeps cross-provider aggregate (single value
is more useful than per-provider for the ratio)
- Loki log filter: |= "codex_usage" → |= "llm_usage" matching the new
log event name in mctl-openclaw#20
VMRule (openclaw-llm-alerts):
- File rename: openclaw-codex-alerts → openclaw-llm-alerts
- Group rename: openclaw.codex-throughput → openclaw.llm-throughput
- OpenclawCodexPromptTokenSpike: scoped to provider="codex", same
25k tok/min threshold (matches ChatGPT Plus tier)
- NEW: OpenclawAnthropicPromptTokenSpike at 60k tok/min for the
anthropic per-API-key quota (higher than Plus, but still detects
fallback runaway)
- OpenclawLlmRequestSpike: provider-agnostic, sum by (pod, provider)
so codex and anthropic each get their own alert firing
- OpenclawLlmMetricsAbsent: queries openclaw_llm_requests_total
Cannot merge until mctlhq/mctl-openclaw#20 ships image
2026.5.2-mctl.3 — current image (mctl.2) still emits the old
openclaw_codex_* series and these queries would go blank on merge.
mashkovd
force-pushed
the
feat/codex-dashboard-extensions
branch
from
May 13, 2026 00:10
5e6e80d to
18fd4b4
Compare
Member
Author
|
@codex review |
|
Codex Review: Didn't find any major issues. Keep it up! ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #201. Extends the dashboard with 24h headline stats, a 7-day trend row, and Loki-based per-command bargauges; adds a sibling
VMRulewith spike alerts.Rebased on 2026-05-13 to track the provider-partitioned metric family being shipped in mctlhq/mctl-openclaw#20 (codex + anthropic via
openclaw_llm_*{provider="..."}). Image bump and ConfigMap rename rolled in.Dashboard rebase (this PR delta)
openclaw-codex-usage-dashboard→openclaw-llm-usage-dashboarduidandtitlerebranded to LLM (covers codex + anthropic fallback)openclaw_llm_*sum by (pod, provider) (...)with legend{{pod}} {{provider}}sum by (provider) (increase(...))so codex vs anthropic show side-by-side|= "codex_usage"→|= "llm_usage"matching new event nameVMRule (
openclaw-llm-alerts)OpenclawCodexPromptTokenSpike— scoped toprovider="codex", 25k tok/min (ChatGPT Plus headroom)OpenclawAnthropicPromptTokenSpike— new, 60k tok/min (anthropic per-API-key quota is higher)OpenclawLlmRequestSpike— provider-agnostic, fires per(pod, provider)OpenclawLlmMetricsAbsent— health check onopenclaw_llm_requests_totalMerge gate
BLOCKED on mctlhq/mctl-openclaw#20 + image
2026.5.2-mctl.3published. The current running image (mctl.2) still emits the oldopenclaw_codex_*series; merging this PR before the new image deploys would blank the dashboard. Sequence:v2026.5.2-mctl.3→ docker-release workflow publishes image2026.5.2-mctl.3(separate small PR, will follow)Test plan
provider="codex"+provider="anthropic")/eth ETHin Telegram → dashboard bargauge shows/ethin either codex or anthropic depending on fallback state🤖 Generated with Claude Code