Skip to content

Parallel-run validation window + explicit go/no-go criteria before cutover #4333

Description

@JSONbored

Part of #4325.

Context

Per the epic's hard constraint, the current host stays authoritative until the new host is independently validated. This issue defines what "validated" concretely means, so cutover is a decision against explicit criteria, not a vibe call.

Requirements

  • Define concrete go/no-go criteria before the validation window starts, covering at minimum: health-check stability over the window (/ready, container health), embedding latency matching or beating the GPU benchmark sub-issue's result under real (not synthetic) load, no unexplained job-queue backlog growth, no error-rate regression vs. the current host's baseline.
  • Run the new host through the window handling shadow/non-authoritative traffic (e.g. a duplicate webhook feed, or a manual replay of recent real events) — not just synthetic smoke tests — before it ever becomes authoritative.
  • The window's length should be stated explicitly up front (e.g. covering at least one full cycle of every scheduled maintenance job's cadence — see the job cadences in src/selfhost/maintenance-admission.ts and src/index.ts's scheduled-job enqueue logic — so the validation actually observes the periodic jobs, not only request-driven behavior).

Deliverables

  • Go/no-go criteria written down before the window starts
  • Validation window run against shadow/replayed real traffic, not only synthetic checks
  • Each criterion's actual result recorded (pass/fail), not just an overall impression
  • Explicit go/no-go decision recorded, with reasoning if "no-go"

Expected outcome

Cutover only happens once against measured, pre-agreed criteria — the epic never has to rely on "it seemed fine."

Effort

M

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions