Skip to content

Add a Grafana dashboard for coding-agent-driver usage, cost, and attempt outcomes #5185

Description

@JSONbored

Context

grafana/dashboards/ currently has ORB's per-provider usage dashboards (e.g. claude-usage.json, codex-usage.json), but no file there mentions the miner at all. AMS runs three coding-agent providers (claude-cli, codex-cli, agent-sdk) and none of their attempt outcomes, token usage, or cost are visualized anywhere. This is distinct from #4838 (wiring the Prometheus renderer into a command) and #4839 (giving AMS its own metrics/tracing surface) — neither of those tickets is scoped to produce the actual dashboard artifact.

Dependencies

Requires the datasource issue in this same batch ("Provision a Grafana datasource for AMS's local SQLite ledgers (attempt log + prediction ledger)") to land first — this dashboard's panels query directly from that datasource, so there is nothing to wire up until it exists. Do not start until that datasource lands. Independent of #4838/#4839.

Requirements

  1. Author grafana/dashboards/miner-usage.json.
  2. Show per-provider (claude-cli, codex-cli, agent-sdk) attempt success/fail counts.
  3. Show per-provider token totals.
  4. Show per-provider cost.
  5. Query all panel data directly from the local ledgers via the new SQLite datasource added in the sibling datasource issue — do not scrape or invent a Prometheus metrics path for this.
  6. Add the same cross-reference convention the existing three ORB AI-usage dashboards use, so an operator running both ORB and AMS on one box can tell at a glance which dashboard answers which question.
  7. Merge all three coding-agent providers into this single dashboard (miner-usage.json) rather than mirroring ORB's one-dashboard-per-provider convention (claude-usage.json / codex-usage.json), because all three AMS drivers share one small local SQLite ledger with much lower data volume than ORB's three independent cloud reviewers — a single dashboard with a provider-filter variable is more useful to operators than three near-empty per-provider dashboards. Implement the provider distinction as a Grafana dashboard template variable (not three separate JSON files).
  8. Do not modify the existing ORB dashboards (claude-usage.json, codex-usage.json, etc.) except to add the cross-reference link called for in requirement 6.

Deliverables / Acceptance Criteria

  • grafana/dashboards/miner-usage.json added
  • Panels for per-provider attempt success/fail counts, token totals, and cost
  • Provider distinction implemented via a single templated variable, not separate per-provider dashboard files
  • Cross-reference convention added, matching the existing ORB AI-usage dashboards
  • Dashboard JSON validated (loads correctly in Grafana / passes any existing dashboard-lint CI check)
  • No unrelated changes to existing ORB dashboard files

Test Coverage Requirements

This PR must ship with full test coverage for every changed line and branch — the repo's Codecov patch gate requires 99%+ coverage and the house standard is to aim for 100%, including both sides of every conditional/nullish-coalescing branch introduced. The dashboard JSON itself lives under grafana/dashboards/, not src/**, so Codecov's gate does not directly instrument it — but any query-building or provider-filter helper logic this issue adds under src/** must still hit the 99%+ bar. Add: (1) unit tests covering the provider-filter logic's success path (a valid provider is selected and only its rows are returned) and failure/edge path (an unrecognized or empty provider filter falls back safely rather than silently returning all providers' data merged), (2) an invariant test asserting the dashboard never mixes ORB metrics and AMS metrics in the same panel, and (3) no regression test applies since this is new functionality, not a bug fix.

Expected Outcome

An operator running AMS can open a single Grafana dashboard to see attempt success/fail rates, token usage, and cost across all three coding-agent providers, filterable by provider, without hunting through three near-empty per-provider dashboards.

Links & Resources

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:featureGittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions