Skip to content

engine-liveness monitor reports stale pre-incident failures as a live OUTAGE — ContentTwin flagged from 3-day-old runs while its preflight passes #1600

New issue
New issue
Closed
#1601
Closed
engine-liveness monitor reports stale pre-incident failures as a live OUTAGE — ContentTwin flagged from 3-day-old runs while its preflight passes#1600
#1601
Labels
bugBug reportsdev-leadFor dev-lead agent pickup

Description

@don-petry
don-petry
opened on Aug 31, 2026

Follow-up to #1587 (PR #1588), found by running the monitor rather than reading it.

Defect

repo_liveness_state classifies a repo from its count of consecutive preflight failures with no recency bound. A low-traffic repo whose last runs predate a resolved incident is reported as a sustained OUTAGE indefinitely — until unrelated traffic happens to arrive.

Reproduction (first real run of the monitor, 2026-08-31T03:19Z)

petry-projects/ContentTwin / dev-lead.yml: OUTAGE (2 consecutive preflight failure(s))
Inspected 16 stub(s); API errors: 0; sustained-outage repos: 1; failing repos: 1.
##[error]Process completed with exit code 1

ContentTwin's two failures are dated 2026-08-25T07:21Z and 2026-08-28T18:57Z — both inside the 08-23 → 08-30 token outage window. It has had no dev-lead runs since, so the streak never cleared.

Verified it is not a live outage: a probe dispatch at 03:20:40Z (run 33353551272) completed success with Engine token preflight: success. The org secret is confirmed shared with the repo. The engine is healthy there; the monitor was reporting three-day-old history as a current outage.

Why this matters rather than being cosmetic

#1587 exists because a real 7-day outage went unnoticed while Fleet Monitor reports (#1577, #1586) accumulated unactioned. A monitor that reports resolved history as a live outage trains people to ignore it — which is precisely the failure mode #1587 was filed to end. The first run already produced one false OUTAGE out of one, and it fails the workflow (exit 1), so every scheduled run stays red until unrelated traffic clears it.

Suggested fix

Bound the signal by recency as well as by streak:

  • Ignore runs older than a configurable window (e.g. ENGINE_LIVENESS_LOOKBACK_DAYS, default 3–7) when computing the streak.
  • Distinguish the states explicitly rather than collapsing them:
    • OUTAGE — ≥2 consecutive preflight failures within the lookback window.
    • STALE / NO RECENT ACTIVITY — the newest run is older than the window. Report it, do not fail the workflow on it, and say when the last run was.
    • OK — recent runs, no preflight failures.
  • Consider treating a repo with no runs in the window as unverified rather than healthy — the honest state — but keep it non-failing so it cannot mask a real outage elsewhere.

Optional but useful: when a repo is STALE, the monitor could fire the same side-effect-free probe used to verify ContentTwin (a dev-lead-reviews-retry dispatch with no pr_number runs the preflight and then cleanly skips at the intent classifier). That converts an unverified repo into a verified one instead of waiting for organic traffic.

Acceptance criteria

  • Streak computation ignores runs outside a configurable lookback window; default documented.
  • STALE / no-recent-activity is a distinct reported state and does not fail the workflow.
  • A repo whose only failures predate the window reports STALE, not OUTAGE — regression test using ContentTwin's exact shape (two failures 3–6 days old, no runs since).
  • A repo with ≥2 consecutive failures inside the window still reports OUTAGE and still fails — the #1587 behaviour must not regress; assert both directions in one test.
  • The report states each repo's last-run timestamp, so a reader can tell "healthy" from "not recently exercised" without opening Actions.

Activity

  1. don-petry
    added
    bugBug reports
    dev-leadFor dev-lead agent pickup
    on Aug 31, 2026
  2. github-actions
    added
    needs-human-reviewFlagged by automated PR review agent
    on Aug 31, 2026

  3. github-actions commented on Aug 31, 2026

    @github-actions
    github-actionsBot
    on Aug 31, 2026 – with GitHub Actions
    Contributor

    Thanks for the detailed and well-reproduced follow-up — including the exact run timestamps and the probe verification, this is exactly the kind of report that makes a fix straightforward to scope. The root cause and suggested fix are clear; one question before we start: is there a preferred default for ENGINE_LIVENESS_LOOKBACK_DAYS (the issue suggests 3–7 days), or should we match the existing ENGINE_LIVENESS_GRACE_PERIOD_DAYS value from #1588 for consistency?

  4. don-petry commented on Aug 31, 2026

    @don-petry
    don-petry
    on Aug 31, 2026
    CollaboratorAuthor

    Dev-Lead Implementation Plan

    Issue: #1600 — engine-liveness monitor reports stale pre-incident failures as a live OUTAGE

    Scope

    Bound the engine-token liveness signal by recency: ignore preflight runs older than a configurable lookback window when computing the failure streak, add a distinct non-failing STALE/no-recent-activity state, surface each repo's last-run timestamp, and keep a within-window OUTAGE failing the workflow.

    Implementation Checklist

    • scripts/lib/engine_liveness_detect.sh — add ENGINE_LIVENESS_DEFAULT_LOOKBACK_DAYS (default 3), within_lookback, within_window_streak, newest_run_recent, repo_liveness_from_runs (returns STALE/OUTAGE/FAIL/OK), extend repo_liveness_state with a recency flag, add run_should_fail, and add a "Last run" column to render_liveness_table.
    • scripts/engine_liveness.sh — gather each run's timestamp, compute state via repo_liveness_from_runs, count stale_repos, write last-run column + ENGINE_LIVENESS_SHOULD_FAIL/ENGINE_LIVENESS_STALE_REPOS to GITHUB_ENV, document ENGINE_LIVENESS_LOOKBACK_DAYS.
    • .github/workflows/engine-token-liveness.yml — add a final always() step that fails the run only on a within-window OUTAGE (ENGINE_LIVENESS_SHOULD_FAIL == 'true'), after the fleet-alert step.
    • AGENTS.md — note the STALE state + lookback default in the Engine-token liveness monitoring: a 7-day fleet-wide dev-lead outage went unalerted, and pr-review fails open #1587 exception paragraph.

    Tests to Write (tests/test_engine_liveness_detect.bats)

    • within_lookback includes recent / excludes old runs (+ boundary).
    • ContentTwin shape (two failures 3–6 days old, no runs since) → STALE, not OUTAGE.
    • ≥2 consecutive failures inside the window → OUTAGE (both directions asserted together).
    • Recent failures ignored when older ones fall outside the window (streak counts within-window only).
    • run_should_fail: OUTAGE fails, STALE/OK do not.
    • render_liveness_table includes the last-run timestamp column.

    Test Command

    bats tests/test_engine_liveness_detect.bats + bash .dev-lead/scripts/dev-lead-lint.sh

  5. don-petry commented on Aug 31, 2026

    @don-petry
    don-petry
    on Aug 31, 2026
    CollaboratorAuthor

    Dev-Lead: Progress (in progress — not yet durable)

    Provisional. Nothing is pushed yet; the automation posts the authoritative
    completion record (with PR number + head SHA) after the push. Treat the notes
    below as intent, not delivered results.

    Plan Execution (intended)

    • Bounded the engine-liveness signal by recency in the pure detector (scripts/lib/engine_liveness_detect.sh): ENGINE_LIVENESS_LOOKBACK_DAYS (default 3), within_lookback, within_window_streak, newest_run_recent, repo_liveness_from_runs, a recency flag on repo_liveness_state, and run_should_fail.
    • New distinct STALE state: newest run older than the window → reported as unverified, not OUTAGE, and does not fail the run.
    • Streak now ignores runs outside the window; a within-window OUTAGE (≥2) still reports OUTAGE and still fails the run (ENGINE_LIVENESS_SHOULD_FAIL → new always() workflow step, ordered after the fleet-alert step).
    • Report/table now surface each stub's last-run timestamp so a reader can distinguish "healthy" from "not recently exercised".
    • Documented the default + STALE behaviour in the AGENTS.md Engine-token liveness monitoring: a 7-day fleet-wide dev-lead outage went unalerted, and pr-review fails open #1587 exception.

    Local check output (provisional — not the pushed run)

    bats tests/test_engine_liveness_detect.bats  -> 50 ok / 0 not ok (13 new)
bash .dev-lead/scripts/dev-lead-lint.sh      -> all checks passed (shellcheck 115 files)
Smoke test (mocked gh): StaleRepo(6-7d failures)=STALE non-failing; LiveRepo(<1d failures)=OUTAGE, SHOULD_FAIL=true

    Files touched (this workspace, not yet pushed)

    • scripts/lib/engine_liveness_detect.sh: recency-bounded detector + STALE + run_should_fail + last-run column.
    • scripts/engine_liveness.sh: gather run timestamps, classify via repo_liveness_from_runs, count stale repos, emit ENGINE_LIVENESS_SHOULD_FAIL/ENGINE_LIVENESS_STALE_REPOS.
    • .github/workflows/engine-token-liveness.yml: fail-red step gated on a within-window OUTAGE only.
    • tests/test_engine_liveness_detect.bats: recency/STALE/OUTAGE regression tests (incl. ContentTwin's exact shape, both directions).
    • AGENTS.md: documented lookback default + STALE.

    Notes

    The ContentTwin-shape regression (two failures 3–6 days old, no runs since) now resolves to STALE, not OUTAGE; a genuine within-window sustained outage still goes red. Did not implement the optional STALE→probe dispatch (out of the stated acceptance criteria).

  6. don-petry commented on Aug 31, 2026

    @don-petry
    don-petry
    on Aug 31, 2026
    CollaboratorAuthor

    Dev-Lead: Implementation Complete — PR #1601

    The implementation for issue #1600 is durable: commits are pushed and a pull request is open. This record is backed by verifiable artifacts you can confirm in one click.

    Review happens on the PR. Acceptance criteria and test results are evidenced by the PR's diff and CI checks — not asserted here. If a later stage fails, this claim is retracted in place.

  7. don-petry
    removed
    needs-human-reviewFlagged by automated PR review agent
    on Aug 31, 2026
  8. don-petry
    added 2 commits that reference this issue on Aug 31, 2026

    fix(liveness): anchor the ISO normalization so a 'UTC' suffix survives

    b2df87d

    feat: implement issue #1600 — engine-liveness monitor reports stale p…

    82c1f76
  9. don-petry
    closed this as completedin #1601on Aug 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugBug reportsdev-leadFor dev-lead agent pickup

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions