[don-petry]

Problem

The Gemini tier snapshot and its history are built from token-usage-* artifacts collected across the organisation. Those artifacts are untrusted input: a pull request run from a fork of a public repository can upload one under any name, with any content. That is harmless while the snapshot is informational, and stops being harmless the moment anything gates dispatch on it. "Gemini trickle under a budget pause" (#2147) makes this its first acceptance criterion: records feeding a dispatch decision must be traceable to trusted runs.

This is that criterion only. The trickle itself is not in this pull request and is not being built yet; see "What this does not do".

Design

What a record can prove. A record carries only what the job wrote, including its own run_id; all of it, and the artifact's name, can be forged. What cannot be forged is what the Actions API says about the run that uploaded the artifact.

The trust rule. An artifact is trusted when both hold:

1. The artifact listing's workflow_run.head_repository_id equals repository_id (not a fork).
2. Either its head branch is the repository's default branch, or a look-up of the run confirms the same run id and head repository and that neither the actor nor the triggering actor is Dependabot.

That is the set of runs GitHub gives the repository's secrets to. Whoever controls such a run's code can already call Gemini with the real keys, so a forged ledger gains them nothing; fork and Dependabot runs get no secrets and cannot have made a real call. Anything that cannot be established (a failed or mismatching look-up, no workflow_run, no actor) is untrusted.

"Default branch only" was rejected: on four days of real data it would drop about a fifth of genuine usage (269 of 1,353 artifacts), and missing usage makes Gemini look more available than it is.

Where it sits. scripts/token_report.sh classifies each artifact at listing time, looks up non-default-branch runs, and stamps every record with provenance {run_id, trusted, basis}, replacing anything the artifact wrote under that key. Only trusted records are handed to the tier report. Ignored and unreadable counts go to the log, a warning and the job summary.

"Cannot tell" is not "available". The snapshot gains a collection block (complete or not, counts, reason). A new reader for future dispatch decisions, gq_tier_decision, answers unknown when the snapshot is unreadable, stale (default one hour), from the future, for a quota day that has ended, built from an incomplete collection, not marked trusted-runs-only, or holding records for a model or key with no caps row. Nothing calls it yet. The existing informational gq_tier_can_run is unchanged.

The history cache key gains v2, so the store filled before the filter existed is not restored. The per-day history table loses up to seven earlier days, once.

What this does not do

• It cannot remove test-fixture records. About 8 of the 1,819 request-counting Gemini records in the sample were real; the rest were written by this repository's engine tests, run by dev-lead agent sessions that inherit the real ledger path, and uploaded by those same trusted runs. No provenance rule can separate them. Filed as The Gemini token ledger is mostly test fixtures: dev-lead agent sessions run the engine tests with the real TOKEN_LOG_FILE set, and the run uploads what the tests wrote #2304. This change detects the visibly fake ones (a model or key with no caps row), warns, and makes the decision reader answer unknown while any are present; 709 of them look exactly like real calls and cannot be counted.
• It does not build the trickle. The design work for Gemini trickle under a budget pause: let a bounded slice of cheap work run on Gemini while the Claude breaker has paused the fleet (default off) #2147 stopped on three findings, written up on that issue: the Copilot credential in dev-lead is the acting token and cannot be withheld from a job that must push and comment; the ledger above is unsound; and a job's own ledger starts empty, so capacity is not knowable at dispatch time and the caps file bounds one job, not the fleet.
• An agent session inside a trusted run can still be induced to append to its own ledger. Provenance authenticates the run, not each record.

Risk

• More API calls in the daily fleet monitor run: one run look-up per artifact that is not on the default branch, about a fifth of them (roughly 270 a day on the sample). The monitor uses the shared maintainer login, whose hourly quota was exhausted twice today by other reads; if that proves tight, the look-ups can be cached per run id.
• The listing and look-up field names were checked against the live API with a maintainer login, not with the monitor's own token.
• The JSON Schema change for the collection block was hand-checked; the schema test needs a Python package that is not installed locally, so CI is the first real validation.
• The collector's cost report is unfiltered on purpose: existing tests require artifacts without a workflow_run to still be collected for it.

Test plan

• tests/token_report_trust.bats (new, 27 tests), wired into lint.yml: the trust rule (fork, same-repository branch, default branch, Dependabot, an artifact declaring itself trusted, an unreadable or mismatching look-up), the hand-off, the snapshot's collection block, the decision reader's fail-closed cases, and the wiring.
• Existing suites unchanged and identical to baseline: token_report 44, test_gemini_quota 44, test_gemini_tier_view 48 (plus one that needs the missing Python package locally).
• Mutation-checked: 24 mutations, each turning a named test red (fork accepted, self-declared trust believed, unreadable look-up accepted, Dependabot accepted, hand-off unfiltered, stale snapshot accepted, incomplete collection accepted, unlisted records ignored, cache not renamed, and others).
• Re-run by me on this commit after rebasing: the new suite 27 of 27, token_report 44, test_gemini_quota 44, the fixed-path check 18; shellcheck --severity=warning -x and actionlint clean.
• Not run: the fleet monitor itself.

Rollback

Revert the squash commit. The tier report goes back to reading every collected record, and the old history cache is used again.

Monitoring

• The next "Actions Fleet Monitor" run's summary should have a "Gemini ledger provenance" section. Expect zero fork and Dependabot artifacts, and a warning about records for models or keys with no caps row until The Gemini token ledger is mostly test fixtures: dev-lead agent sessions run the engine tests with the real TOKEN_LOG_FILE set, and the run uploads what the tests wrote #2304 is fixed.
• That run's duration and any download failed or timed out warnings, for the added look-ups.

Refs #2147

🤖 Generated with Claude Code

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

[coderabbitai]

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 50 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: ASSERTIVE
• Plan: Advanced
• Run ID: 6dfe4c60-ac00-4ae2-af6d-6408b609eca7

---

📥 Commits

Reviewing files that changed from the base of the PR and between 04dc6f3 and 0d99896.

---

📒 Files selected for processing (8)

• .github/workflows/actions-fleet-monitor.yml
• .github/workflows/lint.yml
• AGENTS.md
• scripts/gemini_tier_report.sh
• scripts/lib/gemini-quota.sh
• scripts/lib/gemini-tier-snapshot.schema.json
• scripts/token_report.sh
• tests/token_report_trust.bats

---

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

---

_{Comment @coderabbitai help to get the list of available commands.}

[don-petry]

dev-lead is withholding action on this item.

It is labeled dev-lead:hands-off (deliberately kept off-limits to dev-lead (e.g. changes to the dev-lead automation itself)), so dev-lead will not pick it up while that label is present. This notice is posted once so the withhold is visible rather than looking like a stalled run.

To re-enable automated pickup: remove the dev-lead:hands-off label.

[don-petry]

QA Lead — test-risk advisory

Risk tier: MEDIUM. Nothing calls gq_tier_decision yet, so dispatch is not affected today. But this PR exists to make the snapshot's "complete, trusted-only" claim safe to gate on, and one path makes that claim untrue.

What I'd shore up (highest leverage first):

• A failed run look-up is counted as "untrusted", not as "failed". When _artifact_run_trusted hits a timeout, a 403 or the rate limit, the artifact is stamped run-not-verified and counted as ignored. No COLLECT_FAILED_DIR marker is written, so the snapshot still says complete: true. Real usage is silently dropped, and dropped usage is exactly what makes Gemini look more available than it is. The Risk section expects this to happen: the shared login's quota ran out twice today. Record a transport failure as a failed artifact, and add a bats case where the stub fails the look-up (as opposed to returning a mismatch) and the collection must read incomplete.
• Nothing tests the schema change, in CI or locally. The only schema test, test_gemini_tier_view.bats:554, validates raw gq_tier_snapshot output, which has no collection key. Add a contract test that validates the snapshot gemini_tier_report.sh writes, once with and once without a collection report.
• Look-up volume: add one test that runs many branch artifacts from the same run and asserts how many look-ups are made. That makes the per-run-id cache measurable before quota trouble starts.
• The rest is solid: a stubbed gh, a fixed clock, fail-closed cases, and a mutation check of 24 mutations. Nothing in it looks flaky to me.

Escalate? yes. Under the rate-limit condition this PR itself names, the "complete" flag fails open, so it can't be trusted to gate dispatch later.

Advisory only — I comment, I do not change code. Ground: BMAD Test Architecture
(bmad-tea v1.19.0). Opt out on this item with the qa-lead:hands-off label.

[donpetry-bot]

Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-10-11T19:39:15Z.

[gemini-code-assist]

Code Review

This pull request introduces a mechanism to verify the provenance of token-usage records, ensuring only records from trusted runs are used to build the Gemini tier snapshot and make dispatch decisions. It updates token_report.sh to determine trust via the GitHub Actions API, filters out untrusted records, and writes a collection report. gemini_tier_report.sh is updated to include collection metadata in the snapshot, and gemini-quota.sh introduces gq_tier_decision to validate snapshots before allowing dispatch. Comprehensive tests are added in token_report_trust.bats. The review feedback recommends three improvements: leveraging the existing EXIT trap by creating the temporary trusted_dir inside jsonl_dir to prevent orphaned directories, using jq's optional chaining operator (?) in gq_tier_decision to safely handle potentially missing nested properties, and piping the output of gq_tier_snapshot directly into jq to avoid creating a temporary file.

[codeant-ai]

CodeAnt Nitpicks

1 code suggestion

1. When a genuine same-repository branch run lookup fails, its records are dropped but no failure is counted. Dispatch may accept a complete snapshot missing that usage.

Incomplete implementation · scripts/token_report.sh:725-729

[codeant-ai]

CodeAnt PR Risk: Low Risk

• The PR appears safe to merge.
• Only records traceable to trusted runs reach the tier report, and incomplete collections make dispatch decisions unknown.
• Malformed record files are counted as failed reads, and the tests cover trust filtering and fail-closed decisions.

Assessed commit: 0d998963c200

[don-petry]

Review round 1, head 399b38dd.

• A failed run look-up is now a failed read, not an "untrusted" artifact (QA advisory, escalated; correct). The look-up has three answers: the run qualifies; the API answered and it does not; the API could not be asked or did not answer usably (non-zero exit, timeout, empty or unparseable body, an error body). The third is stamped run-lookup-failed, counted as run_lookups_failed, and makes the collection incomplete, so the decision reader answers unknown. A deleted run (404) cannot be told from a refusal at the CLI's exit code and also reads as a failed look-up, which is the fail-closed direction. I re-ran one mutation myself: treating a failed look-up as "answered no" turns four tests red.
• A malformed record file is a failed read too (CodeAnt).
• The schema change is now tested (QA): three tests validate the snapshot gemini_tier_report.sh actually writes, with and without a collection report, and reject an unknown or missing field. They fail in CI if the validator is missing. The validator is not installed where this was written, so CI on this head is the first real validation of the schema.
• Look-ups are shared per run (QA): both uploaders name the artifact after the run id, and a re-run attempt can add a second artifact under it, so the collector looks each run up once and a test asserts two look-ups for four artifacts of two runs.
• The working-file and temporary-directory points are answered on their threads.

Suites: token_report_trust 36 (3 of them the schema tests, skipped locally), token_report 44, canary_report 46, the fixed-path check 18; the negation lint and shellcheck clean. main is merged in.

[donpetry-bot]

Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-10-11T22:27:32Z.

[don-petry]

Review round 2, head 0d998963. Both points from CodeAnt's read of the previous head are right and fixed:

• An artifact with missing or incomplete workflow_run metadata was treated like a fork's: untrusted, nothing missing. A fork is known to be a fork and its run got no secrets; an artifact whose run the listing does not name may be real usage. It is now stamped run-unknown, warned about, and counted with the failed look-ups, so the collection is incomplete and the decision reader answers unknown. New test: "an artifact the listing names no run for is a failed read; a fork's is not". Classifying it as a fork again turns it red.
• A repository whose listing file could not be created is now counted as a failed listing, not only warned about.

Suites after merging main: token_report_trust and token_report pass, canary_report 46; shellcheck, the negation lint and the fixed-path check clean. The schema tests passed in CI on the previous head (unit and bats were green there), which was the first real validation of the schema change.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[donpetry-bot]

Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-10-12T02:46:20Z.