ci(e2e): classify hosted-runner resource pressure and infrastructure loss

[apurvvkumaria]

Parent epic: #7140

Related dependency: #7101 owns general secret-safe phase heartbeats and progress coverage across E2E targets. This issue owns accurate resource attribution, failure classification, and retry policy.

Problem

Current host-memory snapshots can make healthy Linux page cache look like memory exhaustion, while a GitHub-hosted VM can disappear before cleanup, logs, or artifacts identify the cause. Without cgroup, pressure, process, and Docker evidence, maintainers cannot reliably distinguish:

• application assertion failures;
• a process or container OOM kill;
• disk or inode exhaustion;
• a stalled Docker build; and
• hosted-runner infrastructure loss.

That uncertainty encourages broad retries, which can hide deterministic regressions.

Scope

• Extend the secret-safe heartbeat contract with accurate host measurements where available:
  • MemAvailable, Cached, SReclaimable, swap use, and load;
  • cgroup memory.current, peak/limit, memory.events, and OOM/OOM-kill counters;
  • memory and I/O pressure stall information;
  • top process RSS/PSS consumers;
  • Docker container stats, image/build-cache usage, workspace free space, and inode availability; and
  • kernel OOM evidence where the hosted environment permits access.
• Emit bounded snapshots before and after expensive image/install/rebuild phases and periodically while a phase is active.
• Produce a machine-readable terminal classification for ordinary failures: assertion, timeout, process OOM, container OOM, disk pressure, or unknown.
• Define workflow-level signatures for hosted-runner loss when the runner disappears before test cleanup can execute.
• Permit at most one retry only for confirmed infrastructure-loss signatures. Never retry assertions, deterministic command failures, policy violations, or classified OOM failures.
• Keep all streamed and uploaded evidence free of command payloads, credentials, tokens, and environment-variable values.

Acceptance criteria

• Low raw MemFree alone is never classified as OOM.
• Tests cover classification of application failure, timeout, cgroup OOM, disk exhaustion, runner loss, and ambiguous failure.
• An ordinary assertion receives zero automatic retries.
• A confirmed hosted-runner-loss result receives no more than one retry and links the two attempts for diagnosis.
• The original failure evidence remains available when a retry succeeds.
• Representative rebuild-hermes and other heavy lanes emit enough evidence to identify the largest host/container memory consumers and Docker disk use.
• The implementation interoperates with Add secret-safe phase progress diagnostics to every E2E test #7101 rather than creating a second progress-heartbeat framework.

Non-goals

• Increasing test timeouts.
• Retrying failures that lack a positive infrastructure-loss classification.
• Treating larger runners as a substitute for diagnostics.

[souvikDevloper]

will take a look.

[apurvvkumaria]

Reopening is required based on live evidence from #7391's exact-SHA E2E run 29964501381.

Hermes security-posture job 89073364455 lost GitHub-hosted runner 1021276374 during phase 2/8. The log ended with “The runner has received a shutdown signal” followed by cancellation; there was no assertion failure, artifact upload, or cleanup.

The current #7168 detector did not classify or retry it because GitHub now reports this shutdown as a completed/failed job with one completed/cancelled live step and subsequent completed/skipped cleanup steps. The implementation only accepts the older shape where one step remains in_progress with a null conclusion. The coordination check therefore emitted no runner-loss policy or retry marker.

A second gap is that the existing marker merely permits a later CI event to replace the check; it does not automatically schedule the single retry required by this issue.

Follow-up work is being prepared to:

1. recognize both strict GitHub-hosted shutdown API shapes without retrying assertions or ordinary cancellations; and
2. dispatch one fresh, exact-identity replacement child automatically, preserve/link attempt 1, and make attempt 2 terminal.