💡 Token Cost Observatory with Pre-Agentic Optimization

[don-petry]

Summary

Implement GitHub's Effective Tokens (ET) metric across all agentic workflows, add structured token-usage logging, and introduce pre-agentic context download steps that fetch issue/PR data via gh CLI before agent invocation. This reduces token consumption by 20-60% while providing cost visibility and budget alerting.

Market Signal

GitHub published its token efficiency framework with the Effective Tokens (ET) metric: ET = m × (1.0×I + 0.1×C + 4.0×O), showing 19-62% reductions across five internal workflows. TrueFoundry documented the "agentic token explosion" where costs grow O(n²) in the number of agent steps. Starting June 1, 2026, Copilot code review will consume GitHub Actions minutes under the new AI Credits billing model, adding a significant new cost vector. GitHub's approach of eliminating unused MCP tool registrations saved 8-12KB per turn (several thousand tokens per run).

User Signal

Issue #206 explicitly requests token usage monitoring and optimization. The org runs 8+ agentic workflows (claude-code, agent-shield, feature-ideation, compliance-audit, org-status, dependabot-automerge, pr-review-mention, dependency-audit). No token tracking infrastructure currently exists — costs are invisible. With Copilot's June 2026 billing change imminent, this gap is becoming a budget risk.

Technical Opportunity

Three implementation layers, each independently valuable:

1. Token logging — Claude Code's API proxy can emit per-call metrics (input tokens, output tokens, cache-read/write, model, timestamps) as a JSONL workflow artifact. No new infrastructure required.
2. ET metric calculation — A lightweight shell script applies the formula to the JSONL artifact, producing a weekly cost report that surfaces the top-3 optimization opportunities.
3. Pre-agentic context download — Add pre-fetch steps to claude-code-reusable.yml that download issue/PR context via gh CLI before invoking Claude. This eliminates MCP tool overhead and keeps credentials out of the LLM context window (a security co-benefit).

The weekly cost report can be appended to the existing daily-org-status report with minimal integration work.

Assessment

Dimension	Score	Rationale
Feasibility	high	Logging wrapper + shell script; no new infrastructure; each layer is independently deployable
Impact	high	20-60% cost reduction potential; directly addresses Issue #206; security co-benefit from credential isolation
Urgency	med	Issue #206 is an explicit request; Copilot billing change on June 1 creates a deadline

Adversarial Review

Strongest objection: The org is small with limited workflow runs. The cost isn't high enough to justify monitoring infrastructure. YAGNI applies.

Rebuttal: Issue #206 is an explicit user request, not speculative demand. The org runs 8+ agentic workflows, several daily. GitHub's own team achieved 62% reduction on their auto-triage workflow — similar in scope to this org's compliance-audit. The implementation is deliberately lightweight (logging wrapper + aggregation script, not a full observability platform). The pre-agentic download pattern provides a security benefit beyond cost savings by keeping GitHub tokens out of the LLM context.

Suggested Next Step

Add token-usage logging to claude-code-reusable.yml by capturing the Claude API response token counts and writing them to a JSONL artifact. Implement the ET metric formula in a small shell script (scripts/lib/token-metrics.sh). Create a weekly token report that aggregates costs by workflow and surfaces the top-3 optimization opportunities.

---

Moved from petry-projects/.github#271 — continued here as this repo hosts the agentic workflow infrastructure.

[don-petry]

Weekly Update — 2026-05-15

What Changed

Copilot code review will start consuming GitHub Actions minutes on June 1, 2026

GitHub announced (changelog) that Copilot code review — currently a "free" add-on — will begin consuming GitHub Actions minutes starting June 1, 2026. This is a significant cost signal for organizations like petry-projects that run Copilot review alongside Claude Code and CodeRabbit on every PR.

Impact on this org:

• Every PR currently triggers 3 AI review systems: Claude Code, CodeRabbit, and Copilot
• After June 1, Copilot reviews will consume Actions minutes in addition to the existing Claude Code and CodeRabbit costs
• Combined with the claude-code-action minutes usage and agent-shield runs, total Actions minute consumption per PR will increase materially

Code with Claude SF 2026 pricing changes:

• Anthropic doubled five-hour rate limits across Pro, Max, Team, and Enterprise plans
• Removed peak-hours throttling for Pro and Max accounts
• Increased Opus API rate limits "considerably" with per-tier specifications
• These changes may affect cost-per-review calculations for Claude Code in CI

GitHub Actions pricing context:

• GitHub reduced pricing for hosted runners effective January 1, 2026 (changelog)
• The combination of reduced runner pricing + new Copilot minute consumption creates a complex cost model that warrants monitoring

Updated Assessment

Dimension	Previous	Current	Delta
Feasibility	med	med	→
Impact	med	high	↑ — Copilot minutes add concrete, measurable cost pressure with a June 1 deadline
Urgency	med	high	↑ — June 1 deadline means cost increase hits in 17 days without visibility tooling

Recommendation

Advance — prioritize before June 1. At minimum, establish baseline metrics for current Actions minute consumption per PR (across all AI review tools) so the Copilot minutes impact can be measured when it hits. A lightweight first step: add a step to daily-org-status.yml that reports total Actions minutes consumed per repo in the past 24 hours.

[don-petry]

🔬 Implementation Analysis: pr-review & dev-lead Agents

Architecture Overview

Both agents share a single invocation layer in scripts/engine.sh, which exposes four functions:

Function	Used by	Tiers
run_triage	pr-review	triage (haiku 4.5)
run_agentic	pr-review	deep (sonnet 4.6), rubber-duck (o4-mini), audit (opus 4.7)
run_duck	pr-review	cross-engine rubber-duck adversarial
run_writer	dev-lead	all intents (fix-ci, fix-reviews, fix-issue, human)

This is the perfect single seam for token instrumentation — instrument engine.sh once and both agents are covered.

Token Capture Strategy

Claude (primary engine): The claude CLI supports --output-format json which returns a structured response including usage.input_tokens, usage.output_tokens, usage.cache_read_input_tokens, and usage.cache_creation_input_tokens. Currently all calls use --print (text mode). We'll add an optional instrumentation wrapper that:

1. Runs the CLI with --output-format json when TOKEN_LOG_FILE is set in the environment
2. Extracts the text response for existing callers (transparent to all downstream code)
3. Writes a JSONL token record to TOKEN_LOG_FILE

Gemini / Copilot fallback engines: Neither exposes token counts via CLI. We'll record null for token fields when running these engines — the ET metric still applies but counts are estimated from character count (÷4 as a rough tokenization ratio).

ET Metric Implementation

GitHub's formula: ET = m × (1.0×I + 0.1×C + 4.0×O)

Model cost multipliers (relative to haiku 4.5 = 1.0):

Model	Multiplier
claude-haiku-4-5	1.0
claude-sonnet-4-6	3.0
claude-opus-4-7	15.0
openai/o4-mini	2.0
gemini-2.0-flash	0.5
gemini-1.5-pro	2.0

Files to Create / Modify

New:

• scripts/lib/token-metrics.sh — ET formula, JSONL emitter, per-model multipliers
• tests/unit/test_token_metrics.bats — unit tests for all library functions
• .github/workflows/token-report.yml — weekly aggregation workflow

Modified:

• scripts/engine.sh — instrument run_triage, run_agentic, run_duck, run_writer with token capture
• tests/dev-lead/unit/test_engine_writer.bats — add token-capture assertions
• tests/dev-lead/fixtures/engines/stub-claude — emit JSON token stub output
• .github/workflows/pr-review.yml — set TOKEN_LOG_FILE artifact path
• .github/workflows/dev-lead-reusable.yml — set TOKEN_LOG_FILE artifact path

JSONL Record Schema

{
  "ts": "2026-05-20T19:45:00Z",
  "workflow": "pr-review",
  "tier": "triage",
  "engine": "claude",
  "model": "claude-haiku-4-5-20251001",
  "input_tokens": 2841,
  "cache_read_tokens": 1200,
  "output_tokens": 312,
  "et": 3865.2,
  "pr_url": "https://github.com/...",
  "run_id": "12345678"
}

Security Design

PR URLs are included for correlation but NO diff content, no prompt content, and no credentials appear in the JSONL. The log is a workflow artifact (ephemeral, scoped to the run) — not committed to the repo.

Estimated Token Reduction via Pre-Agentic Download

For the pr-review agent, the triage tier currently calls gh pr view from within the agent. If we pre-fetch this data in the workflow step and inject it into the prompt as a variable, we avoid the MCP tool call overhead. Rough estimate: 8-12KB saved per triage invocation (≈2000-3000 tokens at 4 chars/token), reducing triage cost by ~15-25% per PR.

Test Strategy (TDD)

1. Write test_token_metrics.bats tests first (red): emit_token_record, calculate_et, parse_claude_json_output, model_multiplier_for
2. Implement scripts/lib/token-metrics.sh (green)
3. Extend test_engine_writer.bats with token-capture assertions (red)
4. Instrument engine.sh (green)
5. Add workflow-level integration tests in CI

[github-actions[bot]]

Weekly Update

What Changed

GitHub published detailed results from their own agentic workflow token optimization program (May 2026):

1. Effective Tokens (ET) Metric — GitHub uses a normalized cost metric across model tiers:

   • Formula: ET = m × (I + 0.1×C + 4×O) where m is a model multiplier (Haiku 0.25×, Sonnet 1.0×, Opus 5.0×), I = input, C = cache-read, O = output tokens
   • Output tokens weighted 4× because they're the most expensive type across all providers
   • Enables apples-to-apples comparison across Haiku, Sonnet, and Opus runs

2. 62% ET Savings via Pre-Agentic Data Gathering — Moving deterministic reads (PR metadata, diffs, CI status) into workflow setup steps before agent execution eliminated them from the LLM reasoning loop:

   • Auto-Triage: 62% ET reduction (sustained across 109 post-fix runs)
   • Smoke Claude: 59% reduction
   • Security Guard: 43% reduction
   • Community Attribution: 37% reduction

3. MCP Tool Pruning — Removing unused MCP tool schemas saved 8-12KB per API call. A GitHub MCP server with 40 tools adds 10-15KB of schema per turn; workflows using only 2-3 tools waste the rest.

4. Daily Token Auditor + Optimizer — GitHub runs two meta-agents: an auditor that flags anomalous runs, and an optimizer that reads flagged workflows' source and proposes fixes via GitHub Issues.

Source: Improving token efficiency in GitHub Agentic Workflows

New Ideas discussions filed this week:

• 💡 Pre-Agentic Data Gathering Pipeline for 60%+ Token Savings #563: Pre-Agentic Data Gathering Pipeline (applies GitHub's 62% savings pattern to this project)
• 💡 Effective Token (ET) Anomaly Alerting with Runaway Session Detection #565: ET Anomaly Alerting with Runaway Session Detection (extends the Observatory with proactive alerting)

Updated Assessment

Dimension	Previous	Current	Delta
Feasibility	high	high	→
Impact	med	high	↑ — GitHub's 62% savings on comparable workloads validates the approach with hard data
Urgency	med	high	↑ — pre-agentic data gathering is now an industry-proven pattern; delay means continued 2-3x token waste

Recommendation

Advance — The ET metric should be adopted in token-metrics.sh for normalized cross-model reporting. Pre-agentic data gathering (#563) is the highest-ROI optimization available and should be prioritized. Anomaly alerting (#565) provides the proactive safety net the weekly report currently lacks.

[github-actions[bot]]

Enhancement: Token Cost Observatory — Current State Gap Analysis & Quick Wins

Sharpened problem & goal
The proposal is to implement GitHub's ET metric, add structured token-usage logging, and introduce pre-agentic context download steps. Looking at the current repo, significant portions already exist — the gap is narrower than the proposal implies. The most actionable first step is a gap audit: which workflows are not yet emitting compatible token-usage artifacts? Without this inventory, the initiative has an undefined backlog.

Context
What already exists in this repo:

• scripts/token_report.sh + token-report.yml — org-wide weekly ET aggregation is live; uses _fmt_usd, _fmt_int, and scripts/lib/model-pricing.tsv for effective-dated per-model pricing
• scripts/lib/token-metrics.sh + scripts/lib/model-pricing.sh — the ET computation library is in place
• AGENTS.md §"Cost reporting" — cents-precision standard (_fmt_usd) and "prices are data, not code" (pricing TSV) are org-wide standards
• scripts/dev-lead-preflight.sh — a natural home for pre-agentic gh context download

What is likely missing (to verify in the gap audit):

1. Per-workflow artifact emission (not all workflows may emit compatible token-usage artifacts)
2. Budget alerting when a single PR or workflow run exceeds a threshold
3. Provider/model breakdown in the weekly report (a model column in usage artifacts would enable per-vendor cost comparison, directly feeding Discussion Evaluate additional providers, models, and optimizations #631's vendor strategy)

Note: GitHub's ET paper's 19-62% reduction claims come from context pruning (removing irrelevant issue history), not just pre-downloading via gh CLI. Pre-agentic download reduces redundant tool calls during the agent run — a compatible but different optimization. Both are worth doing; they compound.

Impact / Effort

• Impact: HIGH — cost visibility is a prerequisite for every other cost-optimization proposal (Evaluate additional providers, models, and optimizations #631, RFC: Deterministic pre-agentic review gate to cut token burn (draft-until-green) #743, Idea: Adopt pi as Org-Wide Terminal Coding Agent Harness #283); without it, ROI of changes can't be measured
• Effort: S for the gap audit and adding model to the existing report; M for budget alerting; M for pre-agentic context download in all relevant workflows

Suggested acceptance criteria

• A gap audit (docs/token-observatory-gap.md) lists every workflow in the org and whether it emits a compatible token-usage artifact — the gaps become the prioritized backlog
• All "gap" workflows emit artifacts within one sprint, using the existing token-metrics.sh format (no new format invented)
• The weekly token-report.yml report breaks down ET and USD by model field — enabling per-provider cost comparison and feeding Discussion Evaluate additional providers, models, and optimizations #631's vendor-diversification data needs
• A budget-alert step is added to the two highest-cost workflows (dev-lead, pr-review): if a single run's ET exceeds a configurable threshold, a warning is posted to the PR or tracking issue
• Pre-agentic gh context download is added to scripts/dev-lead-preflight.sh and measured before/after — targeting ≥20% reduction in context-token usage for issue-linked PRs (reported via token-report.yml)

[Amar07Singh]

Hey @don-petry — how is Claude Code scaling on your .github-private repo?

Do things start lagging and forcing you into annoying loop-and-reprompt cycles as the code grows, or is it keeping up?

[don-petry]

Hey @Amar07Singh — good question, and the honest answer is "it keeps up, but because of architecture, not in spite of scale."

A key thing about this repo: .github-private is ~1,300 files / 62 MB, but Claude Code almost never sees the whole thing. It runs in two modes here:

1. Agentic CI (pr-review and dev-lead reusable workflows) — each invocation gets a bounded, task-scoped context: one PR diff, one issue, one CI failure. Repo growth doesn't linearly bloat the window because the agent isn't handed the repo — it's handed the task.
2. Interactive sessions — always in a git worktree on a scoped branch, with MCP tool schemas loaded lazily (deferred until needed) so idle tool definitions don't eat the context budget.

On the specific "loop-and-reprompt death spiral" you're describing — yes, that failure mode is real, and it's the one we engineered against directly rather than hoping the model would grow out of it:

• Timeouts escalate, they don't retry. A deep-implementation run that hits its wall gets exit 124 → reason=timeout → escalate to a human, not a blind re-prompt into the same wall. Blind retry is how you get the O(n²) token explosion.
• Phantom-spec / churn detection — when a task is grinding on files that don't exist or is oversized, it's rescoped and split, not looped.
• Pre-agentic data gathering — deterministic reads (PR metadata, diffs, CI status) are fetched in workflow setup before the agent runs, so they never re-enter the reasoning loop. GitHub measured ~62% token savings from this pattern; we're rolling the same thing here (tracked in 💡 Pre-Agentic Data Gathering Pipeline for 60%+ Token Savings #563).
• We also meter it: every agent call emits per-call token usage, and a weekly Token Cost Observatory report (this very discussion 😄) tells us if any workflow is quietly ballooning.

So: lag correlates with context-window fill and unbounded loops, not raw file count. Manage those two with scoped tasks + worktrees + escalate-don't-retry, and it scales fine. Opus 4.8's 1M window bought a lot of headroom too. Happy to go deeper on any of those.

[Amar07Singh]

One thing I've seen people experiment with lately for exactly this problem is maintaining a lightweight, persistent knowledge layer outside the active context window.

The pattern is basically: instead of relying on the model to repeatedly rediscover architecture decisions, conventions, prior investigations, failed attempts, etc., those get captured incrementally and surfaced back only when they're relevant to the current task.

That seems to complement the "bounded task context" approach you're describing pretty well because it attacks the same failure mode from another angle: context drift and rediscovery loops.

From what I've seen, tools like Bytebell are trying to solve that problem specifically for coding agents by giving them durable project memory rather than ever-larger windows. Curious whether you've experimented with anything in that direction, or if the worktree + scoped-agent model has made it unnecessary so far.

[don-petry]

✅ Delivered — closing as shipped

The Token Cost Observatory (the named deliverable of this idea) is live in production:

• Per-call token logging — both agentic engines emit token-usage-<run_id>.jsonl artifacts (dev-lead-reusable.yml, pr-review.yml) via TOKEN_LOG_FILE.
• ET metric library — scripts/lib/token-metrics.sh + scripts/lib/model-pricing.sh / model-pricing.tsv (prices are data, not code; effective-dated).
• Org-wide weekly cost report — scripts/token_report.sh + .github/workflows/token-report.yml runs Mondays 08:00 UTC, scanning every non-archived repo. Surfaces USD cost (cents-precision _fmt_usd), a cost-per-day stacked chart, by-workflow/tier/model breakdowns, and a most-expensive-PRs rollup. docs/token-report.md documents it.

Shipped across #334 → #456 → #460 → #465, with a reliability hardening pass in #955. Tests green (token_report.bats, model_pricing.bats, test_token_metrics.bats). The last scheduled runs (2026-06-27, 2026-06-29) succeeded.

Remaining work — promoted to dedicated discussions

The two optimization layers from the original 3-layer proposal now live as their own tracked ideas, so the parent is closing to avoid duplication:

• 💡 Pre-Agentic Data Gathering Pipeline for 60%+ Token Savings #563 — Pre-Agentic Data Gathering Pipeline (the "Pre-Agentic Optimization" half of the title; ~62% token savings pattern)
• 💡 Effective Token (ET) Anomaly Alerting with Runaway Session Detection #565 — ET Anomaly Alerting with Runaway Session Detection (proactive budget/anomaly alerting)

Closing #332 as delivered; follow-up optimization tracked in #563 and #565.