fix(observability): report actual orb ai token usage on the dashboards - #3281
Conversation
The Codex usage dashboard's durable SQL panels filtered on model LIKE '%codex%' and only surfaced estimated_neurons, even though migration 0109 added real provider/effort/input_tokens/output_tokens/ total_tokens/cost_usd columns to ai_usage_events months ago: - Rewrite every durable panel in codex-usage.json to filter on provider = 'codex' instead of pattern-matching the model string. - Replace the "Estimated neuron usage by day" panel with a real "Durable token usage by day" panel (input/output tokens), add a new "Durable cost by day" panel, and add effort/token/cost columns to the recent-events table (keeping estimated_neurons alongside for reference, not as the only metric). - Add grafana/dashboards/orb-ai-usage.json: a provider-neutral dashboard over ai_usage_events showing every configured provider (Codex, Claude Code, OpenAI-compatible, Ollama, Anthropic BYOK) broken down by provider/model/effort/repo/PR, plus an "events missing real usage" counter to track which features still haven't threaded real usage through (see #3270). - Clarify claude-usage.json's description: it's Claude Code's own OTEL/session telemetry, not the durable ai_usage_events DB attribution — cross-references the new orb-ai-usage dashboard for that. All three dashboard JSON files are auto-provisioned from grafana/dashboards/ (no config wiring needed for the new file) and validated by scripts/validate-observability-configs.mjs.
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
loopover-orb
Bot
added
the
gittensor:bug
|
Warning 🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨 ⏸️ Gittensory review result - manual review recommendedReview updated: 2026-07-05 04:41:05 UTC
⏸️ Suggested Action - Manual Review
Review summary Nits — 6 non-blocking
Concerns raised — review before merging
Review context
Contributor next steps
Signal definitions
🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed 💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →. Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.
|
…-usage filter - Wrap every sum(input_tokens|output_tokens|total_tokens|cost_usd) in coalesce(..., 0) across codex-usage.json's and orb-ai-usage.json's SQL panels, matching the existing coalesce(sum(estimated_neurons), 0) convention already used in sumAiEstimatedNeuronsSince — an empty result set now renders 0, not a blank/missing data point. - Tighten orb-ai-usage.json's "Events missing real usage" panel to filter status = 'ok' instead of a bare feature exclusion. Verified against the actual status literals ever written to ai_usage_events (quota_exceeded/disabled/unavailable never reach a DB write at all; ai_key_change audit rows use "set"/"replace"/"delete") that the old filter was overcounting: quota_exceeded and BYOK key-lifecycle rows never carry provider/usage data by design, not because a feature is missing usage tracking. Reworded the panel + dashboard descriptions to match what the tightened query actually measures. - Verified all four rewritten SQL queries against a real sqlite3 instance seeded with rows matching each status/provider combination.
github-project-automation
Bot
moved this to Todo
in Self-Hosted Review Stack (LoopOver Orb)
… counter explicitly The "Events missing real usage" panel's query relied on an implicit invariant in a different file (recordAiKeyChange's status param is typed "set"|"replace"|"delete", never "ok") to keep ai_key_change audit rows out of the count, even though the panel's own description already promised that exclusion explicitly. Add the feature <> 'ai_key_change' filter directly to the query so the panel is self-contained and doesn't silently over-count if that invariant ever changes elsewhere. Verified behavior-neutral against a seeded sqlite3 table: identical result (1) for realistic data with and without the new filter, and the new filter correctly excludes a row that violates the current type invariant (an ai_key_change row with status='ok'), which the old query would have wrongly counted.
|
Addressed the flagged blocker and the test-expectations note in 842e728: Blocker (orb-ai-usage.json:62): added the Test expectations: these are Grafana dashboard JSON files with no corresponding unit test in the repo (confirmed no existing test references
|
…ge too
The "Events missing real usage" panel only checked provider IS NULL,
missing a second reachable gap: coerceAiUsage (ai-review.ts) extracts
provider independently of input_tokens/output_tokens/total_tokens/
cost_usd, so a raw provider response like {"usage": {"provider":
"codex"}} with no parseable token counts yields a stored row with
provider='codex' but all four numeric columns at their default.
The AI reviewer's literal suggested fix (checking those four columns
IS NULL) is dead code against this schema — migration 0109 declares
them NOT NULL DEFAULT 0, and recordAiUsageEvent's finiteNumber()
coercion means they can never be SQL NULL, only 0. Verified this with
an independent two-agent analysis (both traced the same schema/
coercion/extraction code) plus two adversarial agents that built a
scratch sqlite3 table from the actual migration SQL and ran the fixed
predicate against 16 seeded rows total across both verification
passes, confirming: the original provider-IS-NULL case still counts,
the new provider-set-all-zero case now counts, a provider-set-with-
real-nonzero-usage case does NOT count (no false positive), and ai_key_
change/quota_exceeded rows stay excluded. Also empirically confirmed a
literal NULL insert into these columns is rejected by the schema itself.
Query is now: status = 'ok' AND feature <> 'ai_key_change' AND
(provider IS NULL OR (input_tokens = 0 AND output_tokens = 0 AND
total_tokens = 0 AND cost_usd = 0)). Description updated to explain why
a zero-value check, not an IS NULL check, is the correct "no data"
signal for this schema.
|
Fixed in c743eed, and this one needed a real code change, not just a defensive no-op — the underlying concern was genuine. What was actually wrong: the panel only checked Why I didn't copy the reviewer's literal suggested SQL: Final query: SELECT count(*) AS unattributed FROM ai_usage_events
WHERE status = 'ok' AND feature <> 'ai_key_change'
AND (provider IS NULL OR (input_tokens = 0 AND output_tokens = 0 AND total_tokens = 0 AND cost_usd = 0))Validation: |
…/recent-events panels The status-breakdown and recent-events panels filtered strictly on provider = 'codex', but a quota-blocked ai_review_pr call never gets provider attribution — record() short-circuits before any provider ever runs, so `actualUsage` (and therefore `provider`) is never passed. That call's `model` field, however, is computed unconditionally via reviewerModelLabel(env, input) inside record() regardless of status, so it always reflects the configured/intended reviewer (e.g. "codex", "codex:o4-mini", "codex+claude-code") even when the call never executed. Filtering by provider alone therefore made every quota-blocked codex review invisible to these panels — a real regression, not the "disabled"/"unavailable" rows the review comment also named (those two statuses never reach a DB write at all, confirmed by tracing every return path in runGittensoryAiReview, so no filter change could have dropped them). Fixed the "Successful review records" (for consistency with the panel below), "Review record status", and "Recent Codex-attributed review events" panels to match 'codex' as a reviewer-name token in EITHER provider OR the "+"-joined model label — reusing the exact matching pattern already established (and covered by test/unit/selfhost-grafana-reporting.test.ts) for the pre-existing model-string-based filter this PR replaced. Left the token/cost sum panels (10, 13) on the strict provider filter, since an unattributed row contributes 0 to a sum regardless of which filter is used, and documented why in both the dashboard and panel descriptions to preempt an "inconsistent filters" flag on a future pass. Verified via two independent agents tracing the exact code paths (early-return statements, record()'s model/provider computation) plus two adversarial agents running the corrected predicate against 9-10 seeded sqlite3 rows each, covering: the flagged quota_exceeded/codex case, cross-provider exclusion, dual-reviewer-join and provider-model override label formats, substring false-positive safety, cross-feature scoping, and no double-counting. Final end-to-end check against all three fixed panels' exact query text confirms the expected row/count results.
|
Fixed in 910267b. This one was correct in substance but the reviewer's framing conflated two different statuses:
Fix: match Verified with two independent agents tracing the code paths separately, then two adversarial agents running the corrected predicate against seeded sqlite3 data (9-10 rows each pass) covering the flagged case, cross-provider exclusion, dual-reviewer/model-override label formats, substring false-positive safety, cross-feature scoping, and no double-counting — plus a final end-to-end check against the exact shipped query text. |
github-project-automation
Bot
moved this from Todo
to Done
in Self-Hosted Review Stack (LoopOver Orb)
1 participant
Summary
model LIKE '%codex%'and surfaced onlyestimated_neurons, even though migration0109_ai_usage_actual_tokens.sqladded realprovider/effort/input_tokens/output_tokens/total_tokens/cost_usdcolumns toai_usage_events— and refactor(ai): remove legacy workers-ai framing from live review features #3270 wired all four live AI features to actually populate them.codex-usage.jsonto filter onprovider = 'codex'instead of pattern-matching the model string.effort/input_tokens/output_tokens/total_tokens/cost_usdalongside the existingestimated_neurons(kept for reference, not as the only metric).grafana/dashboards/orb-ai-usage.json: a provider-neutral dashboard overai_usage_eventscovering every configured provider (Codex, Claude Code, OpenAI-compatible, Ollama, Anthropic BYOK), broken down by provider/model/effort/repo/PR, plus an "events missing real usage" counter to track which features still haven't threaded real usage throughrecordAiUsageEvent.claude-usage.json's description: it's Claude Code's own OTEL/session telemetry (a session-scoped cumulative counter vialast_over_time), not the durableai_usage_eventsDB attribution — cross-references the neworb-ai-usagedashboard for that.grafana/dashboards/(directory-mounted indocker-compose.yml:424, no config wiring needed for the new file) and validated byscripts/validate-observability-configs.mjs.scripts/export-grafana-reporting-db.shalready exports every column these panels reference (verified — no changes needed there).Scope
type(scope): short summaryConventional Commit format.CONTRIBUTING.mdand does not reintroduce GitHub Pages, VitePress,site/, orCNAME.Validation
git diff --checknpm run actionlint(vianpm run test:ci)node scripts/validate-observability-configs.mjs— dashboards and alert rules validnpm run typechecknpm run test:coverage— nosrc/**changes in this PR (JSON config only); full suite green (8807 passed), includingtest/unit/selfhost-grafana-dashboard.test.tsandtest/unit/selfhost-grafana-reporting.test.tsnpm run test:workersnpm run build:mcpnpm run test:mcp-packnpm run ui:openapi:checknpm run ui:lintnpm run ui:typechecknpm run ui:buildnpm audit --audit-level=moderate— 0 vulnerabilitiesnpm run test:cigate locally — greenSafety
gittensory-uichange (Grafana operator dashboards, not the app UI).UI Evidencesection. — N/A per the template's own scope (operator-only Grafana dashboards, notapps/gittensory-ui); did not stand up a Grafana instance to screenshot since no live datasource is available in this environment, but every panel query was validated against the actualai_usage_events/reporting-DB schema and the repo's own dashboard/alert-rule validator.