Skip to content

[Fleet Monitor] combsmith/.github-private — .github/workflows/sync-scope-guard.yml #2373

Closed
#2378
Closed
[Fleet Monitor] combsmith/.github-private — .github/workflows/sync-scope-guard.yml#2373
#2378
Labels
dev-leadFor dev-lead agent pickupfleet-trackerTracked by the Actions Fleet Monitorhealth-checkAutomated health check report

Description

@don-petry
don-petry
opened on Oct 20, 2026

🟠 DEGRADED: combsmith/.github-private / .github/workflows/sync-scope-guard.yml

Metric Value
Failure Rate 25% (2 / 8 runs)
Status DEGRADED
p50 Duration 25s
p95 Duration 1380s
Successful Runs 4
Cancelled Runs 2

Workflow: .github/workflows/sync-scope-guard.yml

Last updated by Fleet Monitor run on 2026-10-20.
Threshold: failure rate > 10% over the monitored window.

Activity

  1. don-petry
    added
    health-checkAutomated health check report
    dev-leadFor dev-lead agent pickup
    fleet-trackerTracked by the Actions Fleet Monitor
    on Oct 20, 2026

  2. don-petry commented on Oct 20, 2026

    @don-petry
    don-petry
    on Oct 20, 2026
    CollaboratorAuthor

    Dev-Lead: issue #2373 implementation failed — will retry

    All engines were rate-limited. This was attempt 1 of 3; dev-lead will retry automatically. You can also re-apply the dev-lead label to retry now.

    Last 40 lines of engine output (redacted) 
    ::notice::[gemini] model gemini-3.8-flash: API key 2 of 4 throttled — trying the next key
::notice::[gemini] model gemini-3.8-flash: key index 3 cooling down for 68056s (daily quota wording → next reset)
::notice::[gemini] model gemini-3.8-flash: API key 3 of 4 throttled — trying the next key
::notice::[gemini] model gemini-3.8-flash: key index 4 cooling down for 68054s (daily quota wording → next reset)
::warning::[gemini] model gemini-3.8-flash throttled (rc=173) — trying next in chain
Warning: 256-color support not detected. Using a terminal with at least 256-color support is recommended for a better visual experience.
Both GOOGLE_API_KEY and GEMINI_API_KEY are set. Using GOOGLE_API_KEY.
Ripgrep is not available. Falling back to GrepTool.
Both GOOGLE_API_KEY and GEMINI_API_KEY are set. Using GOOGLE_API_KEY.
Both GOOGLE_API_KEY and GEMINI_API_KEY are set. Using GOOGLE_API_KEY.
Error when talking to Gemini API Full report available at: /tmp/gemini-client-error-Turn.run-sendMessageStream-2026-10-20T12-05-46-238Z.json TerminalQuotaError: You have exhausted your daily quota on this model.
    at classifyGoogleError (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-NR7L3FXV.js:308059:16)
    at retryWithBackoff (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-NR7L3FXV.js:308777:29)
    at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
    at async GeminiChat.makeApiCallAndProcessStream (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-NR7L3FXV.js:333648:28)
    at async GeminiChat.streamWithRetries (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-NR7L3FXV.js:333343:29)
    at async Turn.run (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-NR7L3FXV.js:334387:24)
    at async GeminiClient.processTurn (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-NR7L3FXV.js:348322:22)
    at async GeminiClient.sendMessageStream (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-NR7L3FXV.js:348419:14)
    at async file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/gemini-Q2GRYIAK.js:12199:26
    at async main (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/gemini-Q2GRYIAK.js:17797:5) {
  cause: {
    code: 429,
    message: 'You exceeded your current quota, please check your plan and billing details. For more information on this error, head to: https://ai.google.dev/gemini-api/docs/rate-limits. To monitor your current usage, head to: https://ai.dev/rate-limit. \n' +
      '* Quota exceeded for metric: generativelanguage.googleapis.com/generate_content_free_tier_requests, limit: 20, model: gemini-3.8-flash\n' +
      'Please retry in 18h54m13.796733982s.',
    details: [ [Object], [Object], [Object] ]
  },
  retryDelayMs: undefined,
  reason: undefined,
  status: 429
}
{
  "session_id": "0110e3ee-7737-4c51-98a9-43b35d21919a",
  "error": {
    "type": "Error",
    "message": "You have exhausted your daily quota on this model.",
    "code": 429
  }
}

  3. don-petry commented on Oct 20, 2026

    @don-petry
    don-petry
    on Oct 20, 2026
    CollaboratorAuthor

    Dev-Lead Implementation Plan

    Issue: #2373 — [Fleet Monitor] combsmith/.github-private — .github/workflows/sync-scope-guard.yml

    Scope

    Diagnosis only. No change in this repo. The guard is working as designed; the defect is upstream in combsmith/.github.

    Root cause

    Implementation Checklist

    Recommended follow-ups (outside this repo)

    • combsmith/.github: have deploy-standard-workflows.sh emit <!-- standards-sync:declared-paths\n.github/workflows/<stub>.yml …\n--> with every stub path in its batched PR body. scripts/aw-standards-sync.sh in this repo already does this (see its declared_block) and can serve as the reference.
    • chore: sync 3 org-standard workflow stub(s) from combsmith/.github #2366: close it so the next sweep regenerates it with the manifest, or have a maintainer add the 3 declared paths to its body.

    Test Command

    bats tests/test_sync_scope_check.bats tests/test_sync_scope_guard.bats (existing; nothing changes here)

  4. don-petry commented on Oct 20, 2026

    @don-petry
    don-petry
    on Oct 20, 2026
    CollaboratorAuthor

    Dev-Lead Implementation Plan

    Issue: #2373 — [Fleet Monitor] combsmith/.github-private — .github/workflows/sync-scope-guard.yml

    Diagnosis

    Scope

    This repo needs no code change. Relaxing the guard to pass manifest-less sync PRs would reopen #1523. The fix belongs in combsmith/.github:

    • batch_pr_body() should append a block listing the stub paths it syncs, for example <!-- standards-sync:declared-paths followed by .github/workflows/<name>.yml, one per line, then -->.
    • After that, close and regenerate chore: sync 3 org-standard workflow stub(s) from combsmith/.github #2366, or add the manifest to its body by hand. Its three changed files are exactly the three stubs listed in its body.

    Implementation Checklist

    • (no change in this repo) — the fix is tracked as a follow-up in combsmith/.github scripts/deploy-standard-workflows.sh

    Tests to Write

    • None in this repo. Phase 2 is skipped as an exception because there is no behavioral change here. The fix in .github should come with a test that batch_pr_body output passes sync_extract_declared_paths and covers the changed files.

    Test Command

    bash .dev-lead/scripts/dev-lead-lint.sh (passes locally on an unchanged tree)

  5. don-petry commented on Oct 21, 2026

    @don-petry
    don-petry
    on Oct 21, 2026
    CollaboratorAuthor

    Dev-Lead Implementation Plan

    Issue: #2373 — [Fleet Monitor] combsmith/.github-private — .github/workflows/sync-scope-guard.yml

    Root cause

    Both real failures (runs 39279060061, 39281325441) are on PR #2366 (standards-sync/workflows-20261019). The guard failed closed with: PR #2366 is a sync PR but declares no path manifest. PR #2366 carries the standards-sync label, but its body has no <!-- standards-sync:declared-paths … --> manifest. The body comes from scripts/deploy-standard-workflows.sh in combsmith/.github, which does not write that manifest. So the guard is doing its job. The workflow is not broken. The other "failures" are cancelled runs from cancel-in-progress concurrency. The 1380s p95 comes from updatedAt on those runs, not job runtime (the jobs ran for about 10s).

    Scope

    sync-scope-guard.yml is a gate workflow whose failure is intentional enforcement, like test-deletion-guard.yml and holdout-guard.yml. Add it to FLEET_GATE_WORKFLOWS so the Fleet Monitor stops opening false-positive trackers for it. AGENTS.md says: "Add new gate workflows of this class to that list."

    Implementation Checklist

    • scripts/fleet_report.sh: add sync-scope-guard.yml to the default FLEET_GATE_WORKFLOWS
    • AGENTS.md: add sync-scope-guard.yml to the exempt gate list in the test-deletion-guard exception

    Tests to Write

    • tests/fleet_report.bats: a sync-scope-guard row over 10% is excluded from high-failure tracking

    Test Command

    bats tests/fleet_report.bats (full: bats tests/), plus bash .dev-lead/scripts/dev-lead-lint.sh

    Out of scope (follow-up)

    PR #2366 itself, and the generator in combsmith/.github (deploy-standard-workflows.sh labels PRs standards-sync but does not write the declared-paths manifest). That needs a change in the other repo. Until it lands, every stub-sync PR it opens here will fail this required check. #2366 stays blocked until it is regenerated with a manifest or closed.

  6. don-petry commented on Oct 21, 2026

    @don-petry
    don-petry
    on Oct 21, 2026
    CollaboratorAuthor

    Dev-Lead: Implementation Complete — PR #2378

    The implementation for issue #2373 is durable: commits are pushed and a pull request is open. This record is backed by verifiable artifacts you can confirm in one click.

    Review happens on the PR. Acceptance criteria and test results are evidenced by the PR's diff and CI checks — not asserted here. If a later stage fails, this claim is retracted in place.

  7. don-petry
    closed this as completedin #2378on Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dev-leadFor dev-lead agent pickupfleet-trackerTracked by the Actions Fleet Monitorhealth-checkAutomated health check report

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions