[Phase 2] Add scheduled-run reliability analysis to the fleet health daily check #725

Activity

1. github-actions

   added

   initiativeEpic / initiative tracking issueEpic / initiative tracking issue

   on Jun 14, 2026
2. github-actions

   added a parent issue on Jun 14, 2026

   • Reliable scheduled Actions: off-peak cron offsets, scheduling standard, and schedule-reliability reporting #722
3. don-petry

   added

   dev-leadFor dev-lead agent pickupFor dev-lead agent pickup

   on Jul 14, 2026
4. don-petry commented on Jul 14, 2026

   

   don-petry

   on Jul 14, 2026

   Collaborator

   More actions

   Dev-Lead Implementation Plan

   Issue: #725 — [Phase 2] Add scheduled-run reliability analysis to the fleet health daily check

   Scope

   Extend the Actions Fleet Monitor to measure per-workflow scheduled-run reliability (expected cron ticks vs. actual schedule-triggered runs, with tolerance) and surface stalled crons in the daily report — without breaking the existing 12-field metrics TSV contract (a parallel schedule-metrics file is used instead).

   Implementation Checklist

   • scripts/fleet_report.sh: add pure helpers count_cron_ticks, extract_crons, workflow_expected_ticks, classify_schedule_reliability, generate_schedule_reliability_report; extend label_to_icon with STALLED/STABLE.
   • scripts/fleet_monitor.sh: capture .event in the per-workflow runs query; compute window epochs; per workflow fetch file content, parse cron, count expected ticks, count actual schedule runs, classify, and write a parallel schedule_metrics_file; render the section in both the step summary and the issue-body report; clean up temp file.
   • Handle multi-cron (summed ticks), standard dom/dow OR-interaction, and document best-effort/tolerance/default-branch-only/1000-cap caveats.

   Tests to Write (tests/fleet_report.bats — already in lint.yml bats list)

   • count_cron_ticks: daily, hourly, */15, and no-fire-in-window cases
   • extract_crons: single cron, multi-cron, no-schedule, on:-as-boolean YAML gotcha
   • workflow_expected_ticks: multi-cron sum and no-schedule → 0
   • classify_schedule_reliability: STABLE / STALLED / DEGRADED / within-tolerance / hit-rate cap
   • generate_schedule_reliability_report: renders STALLED row + caveat note; empty file → no output

   Test Command

   scripts/run-bats.sh tests/fleet_report.bats and bash .dev-lead/scripts/dev-lead-lint.sh
5. don-petry commented on Jul 14, 2026

   

   don-petry

   on Jul 14, 2026

   Collaborator

   More actions

   Dev-Lead: Implementation Complete

   Plan Execution

   • scripts/fleet_report.sh: added pure helpers count_cron_ticks, extract_crons, workflow_expected_ticks, classify_schedule_reliability, generate_schedule_reliability_report; extended label_to_icon with STALLED/STABLE and added SCHEDULE_TOLERANCE_TICKS / SCHEDULE_RELIABILITY_THRESHOLD tunables.
   • scripts/fleet_monitor.sh: captured .event in the per-workflow runs query; computed [CUTOFF, now) window epochs; per workflow fetch file content (raw), parse cron, count expected ticks, count actual schedule-event runs, classify, and write a parallel schedule_metrics_file (12-field TSV contract untouched); rendered the section in both the step summary and the issue-body report; cleaned up the temp file.
   • Multi-cron ticks summed; standard dom/dow OR-interaction implemented; best-effort / tolerance / default-branch-only / 1000-cap caveats documented in the rendered note.
   • bats coverage added to tests/fleet_report.bats (already on the lint.yml bats list — no workflow change needed).

   Test Results

   # bats tests/fleet_report.bats  → 81/81 pass (23 new)
   ok 59 cron ticks: daily 6am fires once over a 24h window
   ok 60 cron ticks: hourly fires 24 times over a 24h window
   ok 61 cron ticks: */15 fires 4 times over a 1h window
   ok 62 cron ticks: comma+range minute list fires the expected count
   ok 63 cron ticks: day-of-week Monday fires once over a week
   ok 64 cron ticks: a cron that never matches in the window yields 0
   ok 65 cron ticks: half-open window does not double-count the end boundary
   ok 66 cron ticks: empty cron expression yields 0
   ok 67 extract crons: single cron entry is printed
   ok 68 extract crons: multiple cron entries are each printed
   ok 69 extract crons: workflow with no schedule prints nothing
   ok 70 extract crons: handles YAML 'on:' parsed as boolean true key
   ok 71 expected ticks: multi-cron workflow sums ticks across entries
   ok 72 expected ticks: workflow with no schedule yields 0
   ok 73 reliability: expected==actual is STABLE at 100%
   ok 74 reliability: zero actual against many expected is STALLED
   ok 75 reliability: actual materially below threshold is DEGRADED
   ok 76 reliability: a single missed tick within tolerance stays STABLE
   ok 77 reliability: hit rate is capped at 100% when actual exceeds expected
   ok 78 reliability: whole-number rate renders without a decimal
   ok 79 schedule report: renders a STALLED workflow and the caveat note
   ok 80 schedule report: STALLED sorts above STABLE
   ok 81 schedule report: empty file produces no output
   
   # shellcheck --severity=warning -x scripts/fleet_monitor.sh scripts/fleet_report.sh → clean
   # bash .dev-lead/scripts/dev-lead-lint.sh → all checks passed
   

   Files Changed

   • scripts/fleet_report.sh: pure, sourced, testable schedule-reliability helpers + report renderer (+225 lines).
   • scripts/fleet_monitor.sh: .event capture, cron fetch/parse per workflow, parallel schedule-metrics file, section wiring, cleanup (+53 lines).
   • tests/fleet_report.bats: 23 new tests covering cron→tick expansion, cron extraction, multi-cron summing, reliability classification, and section rendering (+224 lines).

   Notes

   • Contract preserved: schedule metrics live in a separate TSV, so the 12-field high-failure JSON export is unchanged. .github/workflows/actions-fleet-monitor.yml needs no change — it just runs fleet_monitor.sh, and the new section is appended to both the step summary and the issue body.
   • False-miss avoidance (AC#5): a schedule run is counted regardless of conclusion (incl. in-progress), and a tolerance of 1 tick absorbs boundary/late ticks. Consequence: STALLED only triggers when actual==0 and expected ≥ 2, so a single daily tick missed in a 1-day window is intentionally not flagged (indistinguishable from a delayed-past-window run). Sub-daily crons (hourly, */15) and low-frequency crons over a longer LOOKBACK_DAYS are flagged as designed. Thresholds are tunable via SCHEDULE_TOLERANCE_TICKS / SCHEDULE_RELIABILITY_THRESHOLD.
   • Cost: one extra contents API read per workflow (unavoidable — the workflows API does not expose cron). Bounded by workflow count; read/parse failures are non-fatal ::warning::s consistent with the ERROR-sentinel pattern.
   • dom/dow: standard Vixie-cron rule — when both day-of-month and day-of-week are restricted, a tick fires if either matches.
   • No follow-up issues required.
6. don-petry

   closed this as completedin #1217on Jul 14, 2026
7. don-petry

   added a commit that references this issue on Jul 14, 2026

   feat: implement issue #725 — [Phase 2] Add scheduled-run reliability …

   ...

   2548a42