Skip to content

A publish workflow that fails identically forever should surface once, loudly, not retry quietly #9951

Description

@JSONbored

Fallout from #9946: the MCP and Miner publishes had failed on every main commit for as far back as the run history goes — three attempts per push, never one success — and nobody noticed. Neither package had been publishing.

The failure itself was a one-line omission. What made it a long-running outage rather than a five-minute fix was that nothing ever escalated.

Why it stayed invisible

  • The failure is in a publish workflow, not validate-*, so PR CI stays green and the gate is unaffected. Main’s check rollup shows red, but it reads as release noise.
  • It retries three times per commit and fails identically each time, which makes the signal look like flakiness rather than a hard, permanent break.
  • Nothing distinguishes "this attempt failed" from "this has now failed on 40 consecutive commits with the same error".

Suggested direction

  • Track consecutive failures per publish workflow. A run that fails with the same step + error as the previous run on a different commit is not a retry — it is a standing outage.
  • Escalate once when that count crosses a small threshold (2–3 distinct commits): an issue, a PagerDuty route, or whatever the deployment already uses — not a repeated notification per attempt.
  • Stop retrying at that point. Three attempts against a deterministic TS2307 buys nothing; it only adds noise that hides the signal.
  • Surface "last successful publish" per package somewhere durable. @loopover/mcp silently not shipping for many commits should be answerable at a glance.

The general principle is the same one behind #9876 and #9939: a state that is supposed to be transient must be bounded, and when it turns out not to be transient, that fact has to become visible on its own rather than waiting for someone to go looking.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions