Skip to content

test(selfhost): add beta smoke matrix and release checklist - #2657

Merged
JSONbored merged 3 commits into
mainfrom
docs/selfhost-beta-release-checklist
Jul 3, 2026
Merged

test(selfhost): add beta smoke matrix and release checklist#2657
JSONbored merged 3 commits into
mainfrom
docs/selfhost-beta-release-checklist

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Closes #1944

Summary

  • The existing CI "build + boot smoke test" only exercises one fixed configuration (SQLite + Redis + direct-App mode) — there was no repeatable way to validate brokered mode, air-gapped mode, or any AI provider before cutting an RC image.
  • scripts/smoke-selfhost.sh: a portable, mode-agnostic smoke script. Boots one container against a fresh Redis on an isolated network, waits for health, and asserts on /health, /ready, /metrics plus caller-supplied expected/forbidden startup log events (SELFHOST_SMOKE_EXPECT_EVENTS / _FORBID_EVENTS). Always asserts selfhost_migrations_applied regardless of mode.
  • New docs page "Beta release checklist" walks through all six required scenarios (direct App, brokered, air-gapped, each AI provider, SQLite vs. Postgres, Redis + optional Qdrant RAG) with the exact env + smoke-script invocation for each, an expected-events table, and a known-warnings table distinguishing acceptable-in-beta from release-blocking (e.g. a failed relay registration is a warning in pull mode but fatal in push mode).
  • Linked from the Releases & images page's prerelease section and added to the self-hosting nav under "Self-hosting: release & security".
  • Drift-guard test (mirrors docs(ops): finish self-host rate-limit dashboards and runbooks #1943's pattern): every selfhost_* event name cited in the new doc and script is verified against the real console.log/error call sites in src/server.ts and src/selfhost/ai.ts.

No dedicated tests for the shell script

scripts/smoke-selfhost.sh is a manually-invoked operator tool (like the existing deploy-selfhost-image.sh/deploy-selfhost-prebuilt.sh, which also have no unit tests) — it boots real Docker containers and isn't unit-testable. Verified with shellcheck (clean) and the drift-guard test above, which does cover its cited event names.

Test plan

  • shellcheck scripts/smoke-selfhost.sh — clean
  • npm run ui:typecheck, npm run ui:lint, npm run ui:test, npm run ui:build — all clean
  • npm run test:ci (full local gate, unsharded) — green, including the new drift-guard test
  • npm audit --audit-level=moderate — 0 vulnerabilities
  • Manually verified in a running dev server: new page renders all 6 scenario sections + expected-events + known-warnings tables, docs nav shows "Beta release checklist" under "Self-hosting: release & security", Releases & images page links to it

The existing CI "build + boot smoke test" only exercises one fixed configuration
(SQLite + Redis + direct-App mode) -- there was no repeatable way to validate
brokered mode, air-gapped mode, or any AI provider before cutting an RC image.

- scripts/smoke-selfhost.sh: a portable, mode-agnostic smoke script. Boots one
  container against a fresh Redis on an isolated network, waits for health, and
  asserts on /health, /ready, /metrics plus caller-supplied expected/forbidden
  startup log events (SELFHOST_SMOKE_EXPECT_EVENTS / _FORBID_EVENTS). Always
  asserts selfhost_migrations_applied regardless of mode.
- New docs page "Beta release checklist" walks through all six required
  scenarios (direct App, brokered, air-gapped, each AI provider, SQLite vs.
  Postgres, Redis + optional Qdrant RAG) with the exact env + smoke-script
  invocation for each, an expected-events table, and a known-warnings table
  distinguishing acceptable-in-beta from release-blocking (e.g. a failed relay
  registration is a warning in pull mode but fatal in push mode).
- Linked from the Releases & images page's prerelease section and added to the
  self-hosting nav under "Self-hosting: release & security".
- Drift-guard test (mirrors #1943's pattern): every selfhost_* event name cited
  in the new doc and script is verified against the real console.log/error call
  sites in src/server.ts and src/selfhost/ai.ts.
@dosubot dosubot Bot added the size:L label Jul 3, 2026
@loopover-orb

loopover-orb Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Important

🟪🟪🟪🟪🟪🟪🟪🟪🟪🟪🟪🟪

🔍 Gittensory is reviewing…

AI analysis is in progress. This comment will update when the review is complete.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed · 🟪 Reviewing

@loopover-orb loopover-orb Bot added gittensor gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. labels Jul 3, 2026
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jul 3, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
gittensory-ui 92b06c0 Commit Preview URL

Branch Preview URL
Jul 03 2026, 01:18 AM

@codecov

codecov Bot commented Jul 3, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.09%. Comparing base (6ff28c4) to head (92b06c0).
⚠️ Report is 5 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #2657   +/-   ##
=======================================
  Coverage   96.09%   96.09%           
=======================================
  Files         237      237           
  Lines       26472    26472           
  Branches     9602     9602           
=======================================
  Hits        25438    25438           
  Misses        425      425           
  Partials      609      609           
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…t.sh

Gate review caught two real bugs: (1) the script unconditionally created the
network even when SELFHOST_SMOKE_NETWORK was set, breaking the doc's own
Postgres/Qdrant examples (network already exists); cleanup then unconditionally
removed it too, which would have torn down the caller's network out from under
their other containers. (2) the Redis readiness loop fell through silently on
timeout instead of failing with a clear error. Also pinned the Qdrant image tag
to match docker-compose.yml instead of :latest.

Verified live with Docker: a pre-created network now survives the full
create-skip + cleanup-skip path, and the default (owned) path still cleans up
with no orphaned networks.
…ke-selfhost.sh

Gate review caught a real bug: SELFHOST_SMOKE_EXTRA_ENV is line-delimited, so a
multiline PEM private key passed through it (exactly what the direct-App
checklist scenario documented) gets truncated at the first internal newline and
the remaining PEM lines become bogus separate env entries -- the container ends
up without a valid private key for the one scenario that most needs it.

Added SELFHOST_SMOKE_EXTRA_VOLUMES (same line-delimited -v passthrough pattern)
and switched the direct-App doc example to mount the key file + set
GITHUB_APP_PRIVATE_KEY_FILE, which src/server.ts already resolves into
GITHUB_APP_PRIVATE_KEY at startup like every other *_FILE variable. Verified
live with Docker that a multiline file mounts with content intact.
@JSONbored JSONbored self-assigned this Jul 3, 2026
@JSONbored
JSONbored merged commit 52f770f into main Jul 3, 2026
10 of 11 checks passed
@JSONbored
JSONbored deleted the docs/selfhost-beta-release-checklist branch July 3, 2026 01:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test(selfhost): add beta smoke matrix and release checklist

1 participant