[Phase 2] Add scheduled-run reliability analysis to the fleet health daily check

[github-actions]

Story

As a fleet reliability owner,
I want extend the daily Actions Fleet Monitor to measure and report per-workflow scheduled-run reliability -- expected scheduled ticks vs. actual schedule-triggered runs in the window,
so that a cron that silently never fires becomes visible and distinguishable from a healthy one, so 'successful but never ran' stops hiding stalled backlogs.

Acceptance Criteria

1. For each workflow that has a schedule trigger, fleet_monitor.sh/fleet_report.sh compute the number of expected scheduled ticks within the lookback window and the number of actual runs whose triggering event is schedule, plus a reliability/hit-rate metric (and/or a missed-tick count).
2. The per-workflow runs query captures the run's event field so schedule-triggered runs are isolated from workflow_dispatch / push / repository_dispatch runs.
3. The workflow's cron expression(s) are obtained per workflow (parsed from the workflow file YAML, since the actions/workflows API does not expose cron) and expanded to the count of expected ticks over the window; multi-cron workflows and the default-branch-only scheduling caveat are handled or explicitly documented.
4. The daily fleet report (both the GITHUB_STEP_SUMMARY and the issue-body report) gains a schedule-reliability section, and workflows whose reliability falls below a threshold are surfaced in a style consistent with the existing high-failure tracking.
5. Best-effort delay is not falsely reported as a miss: a tolerance window is applied (a run that fired late but within tolerance counts as present), and the existing caveat that GitHub caps run queries at 1000 results per window is respected/noted.
6. bats coverage is added (in tests/fleet_report.bats or a new bats file appended to the lint.yml bats list) for the cron-to-expected-ticks expansion and the reliability classification, driven by fixture run data.

Tasks / Subtasks

• Capture the run event in the per-workflow runs query in fleet_monitor.sh and thread it through the metrics (AC: test issue from agent #1, Go-live improvements for PR review agent #2)
• Add a cron-parse + expected-tick helper that reads schedule.cron from the workflow file and counts expected ticks over LOOKBACK_DAYS (AC: test issue from agent #1, Add @claude delegation, auto-merge, and rebase handling #3)
  • Fetch workflow file content (or read locally for this repo) to extract cron; the workflows API does not return cron
  • Handle multiple cron entries per workflow; document day-of-week/day-of-month interaction handling
• Compute reliability (actual schedule runs vs expected, with tolerance) and a classification/threshold (AC: test issue from agent #1, feat: add Copilot engine support via REVIEW_ENGINE toggle #5)
• Render a schedule-reliability section in fleet_report.sh generate_report (step summary + issue body) (AC: Optimize review: small-PR and incremental fast paths #4)
• Add bats tests with fixture run data covering tick expansion and classification; append the test file to the lint.yml bats list if new (AC: feat: feature ideation framework — adversarial proposer/challenger/synthesizer #6)

Dev Notes

• Entry points: fleet_monitor.sh fetches per-workflow runs (~lines 100-120) selecting run_number/conclusion/created_at/html_url/duration_s -- it does NOT currently request .event; add it there. Metrics are emitted as a 12-field TSV (~lines 154-160) consumed by fleet_report.sh generate_report; extend the row or emit a parallel schedule-metrics file rather than breaking the existing 12-field contract relied on by the high-failure JSON export (~lines 227-240).
• The actions/workflows API (used at ~line 85) returns id/path/state but NOT the cron schedule -- cron must be parsed from the workflow YAML. For org-wide scanning that means fetching file content per workflow; keep it bounded and tolerate parse failures (record as unknown rather than failing the monitor, consistent with the existing ERROR-sentinel pattern).
• Scheduled events are best-effort by design (the whole point of Improving reliability of scheduled GitHub Actions runs (cron is best-effort) #656): a late-but-present run must NOT be scored as a miss. Apply a tolerance window and only flag workflows where actual scheduled runs fall materially below expected. Respect the documented 1000-results-per-window cap (comment near line 98).
• 'Fleet health daily check' = the Actions Fleet Monitor (actions-fleet-monitor.yml, daily) driven by scripts/fleet_monitor.sh + scripts/fleet_report.sh -- that is the surface the owner asked to incorporate this into, not daily-pr-review-health.yml.
• Testing standard: this repo uses bats with an explicit allow-list in lint.yml; tests/fleet_report.bats already exists and is the natural home for report-rendering and math assertions. python3 is available in CI (lint.yml installs pyyaml/jsonschema) if a small parser helper is cleaner than pure bash/awk.

Project Structure Notes

Changes are confined to scripts/fleet_monitor.sh, scripts/fleet_report.sh, and tests (tests/fleet_report.bats or a new bats file added to lint.yml's bats list). actions-fleet-monitor.yml itself likely needs no change since it just runs fleet_monitor.sh; confirm no new step is required to surface the section.

References

• scripts/fleet_monitor.sh#per-workflow-runs-query
• scripts/fleet_monitor.sh#metrics-tsv
• scripts/fleet_monitor.sh#high-failure-json-export
• scripts/fleet_report.sh#generate_report
• .github/workflows/actions-fleet-monitor.yml
• tests/fleet_report.bats
• .github/workflows/lint.yml#bats

Likely target surface

• scripts/fleet_monitor.sh
• scripts/fleet_report.sh
• tests/fleet_report.bats
• .github/workflows/lint.yml

Story prepared by the BMAD Scrum Master (Bob) for epic #722. Status: ready-for-dev.

[don-petry]

Dev-Lead Implementation Plan

Issue: #725 — [Phase 2] Add scheduled-run reliability analysis to the fleet health daily check

Scope

Extend the Actions Fleet Monitor to measure per-workflow scheduled-run reliability (expected cron ticks vs. actual schedule-triggered runs, with tolerance) and surface stalled crons in the daily report — without breaking the existing 12-field metrics TSV contract (a parallel schedule-metrics file is used instead).

Implementation Checklist

• scripts/fleet_report.sh: add pure helpers count_cron_ticks, extract_crons, workflow_expected_ticks, classify_schedule_reliability, generate_schedule_reliability_report; extend label_to_icon with STALLED/STABLE.
• scripts/fleet_monitor.sh: capture .event in the per-workflow runs query; compute window epochs; per workflow fetch file content, parse cron, count expected ticks, count actual schedule runs, classify, and write a parallel schedule_metrics_file; render the section in both the step summary and the issue-body report; clean up temp file.
• Handle multi-cron (summed ticks), standard dom/dow OR-interaction, and document best-effort/tolerance/default-branch-only/1000-cap caveats.

Tests to Write (tests/fleet_report.bats — already in lint.yml bats list)

• count_cron_ticks: daily, hourly, */15, and no-fire-in-window cases
• extract_crons: single cron, multi-cron, no-schedule, on:-as-boolean YAML gotcha
• workflow_expected_ticks: multi-cron sum and no-schedule → 0
• classify_schedule_reliability: STABLE / STALLED / DEGRADED / within-tolerance / hit-rate cap
• generate_schedule_reliability_report: renders STALLED row + caveat note; empty file → no output

Test Command

scripts/run-bats.sh tests/fleet_report.bats and bash .dev-lead/scripts/dev-lead-lint.sh

[don-petry]

Dev-Lead: Implementation Complete

Plan Execution

• scripts/fleet_report.sh: added pure helpers count_cron_ticks, extract_crons, workflow_expected_ticks, classify_schedule_reliability, generate_schedule_reliability_report; extended label_to_icon with STALLED/STABLE and added SCHEDULE_TOLERANCE_TICKS / SCHEDULE_RELIABILITY_THRESHOLD tunables.
• scripts/fleet_monitor.sh: captured .event in the per-workflow runs query; computed [CUTOFF, now) window epochs; per workflow fetch file content (raw), parse cron, count expected ticks, count actual schedule-event runs, classify, and write a parallel schedule_metrics_file (12-field TSV contract untouched); rendered the section in both the step summary and the issue-body report; cleaned up the temp file.
• Multi-cron ticks summed; standard dom/dow OR-interaction implemented; best-effort / tolerance / default-branch-only / 1000-cap caveats documented in the rendered note.
• bats coverage added to tests/fleet_report.bats (already on the lint.yml bats list — no workflow change needed).

Test Results

# bats tests/fleet_report.bats  → 81/81 pass (23 new)
ok 59 cron ticks: daily 6am fires once over a 24h window
ok 60 cron ticks: hourly fires 24 times over a 24h window
ok 61 cron ticks: */15 fires 4 times over a 1h window
ok 62 cron ticks: comma+range minute list fires the expected count
ok 63 cron ticks: day-of-week Monday fires once over a week
ok 64 cron ticks: a cron that never matches in the window yields 0
ok 65 cron ticks: half-open window does not double-count the end boundary
ok 66 cron ticks: empty cron expression yields 0
ok 67 extract crons: single cron entry is printed
ok 68 extract crons: multiple cron entries are each printed
ok 69 extract crons: workflow with no schedule prints nothing
ok 70 extract crons: handles YAML 'on:' parsed as boolean true key
ok 71 expected ticks: multi-cron workflow sums ticks across entries
ok 72 expected ticks: workflow with no schedule yields 0
ok 73 reliability: expected==actual is STABLE at 100%
ok 74 reliability: zero actual against many expected is STALLED
ok 75 reliability: actual materially below threshold is DEGRADED
ok 76 reliability: a single missed tick within tolerance stays STABLE
ok 77 reliability: hit rate is capped at 100% when actual exceeds expected
ok 78 reliability: whole-number rate renders without a decimal
ok 79 schedule report: renders a STALLED workflow and the caveat note
ok 80 schedule report: STALLED sorts above STABLE
ok 81 schedule report: empty file produces no output

# shellcheck --severity=warning -x scripts/fleet_monitor.sh scripts/fleet_report.sh → clean
# bash .dev-lead/scripts/dev-lead-lint.sh → all checks passed

Files Changed

• scripts/fleet_report.sh: pure, sourced, testable schedule-reliability helpers + report renderer (+225 lines).
• scripts/fleet_monitor.sh: .event capture, cron fetch/parse per workflow, parallel schedule-metrics file, section wiring, cleanup (+53 lines).
• tests/fleet_report.bats: 23 new tests covering cron→tick expansion, cron extraction, multi-cron summing, reliability classification, and section rendering (+224 lines).

Notes

• Contract preserved: schedule metrics live in a separate TSV, so the 12-field high-failure JSON export is unchanged. .github/workflows/actions-fleet-monitor.yml needs no change — it just runs fleet_monitor.sh, and the new section is appended to both the step summary and the issue body.
• False-miss avoidance (AC#5): a schedule run is counted regardless of conclusion (incl. in-progress), and a tolerance of 1 tick absorbs boundary/late ticks. Consequence: STALLED only triggers when actual==0 and expected ≥ 2, so a single daily tick missed in a 1-day window is intentionally not flagged (indistinguishable from a delayed-past-window run). Sub-daily crons (hourly, */15) and low-frequency crons over a longer LOOKBACK_DAYS are flagged as designed. Thresholds are tunable via SCHEDULE_TOLERANCE_TICKS / SCHEDULE_RELIABILITY_THRESHOLD.
• Cost: one extra contents API read per workflow (unavoidable — the workflows API does not expose cron). Bounded by workflow count; read/parse failures are non-fatal ::warning::s consistent with the ERROR-sentinel pattern.
• dom/dow: standard Vixie-cron rule — when both day-of-month and day-of-week are restricted, a tick fires if either matches.
• No follow-up issues required.