test(selfhost): add beta smoke matrix and release checklist - #2657
Merged
Conversation
The existing CI "build + boot smoke test" only exercises one fixed configuration (SQLite + Redis + direct-App mode) -- there was no repeatable way to validate brokered mode, air-gapped mode, or any AI provider before cutting an RC image. - scripts/smoke-selfhost.sh: a portable, mode-agnostic smoke script. Boots one container against a fresh Redis on an isolated network, waits for health, and asserts on /health, /ready, /metrics plus caller-supplied expected/forbidden startup log events (SELFHOST_SMOKE_EXPECT_EVENTS / _FORBID_EVENTS). Always asserts selfhost_migrations_applied regardless of mode. - New docs page "Beta release checklist" walks through all six required scenarios (direct App, brokered, air-gapped, each AI provider, SQLite vs. Postgres, Redis + optional Qdrant RAG) with the exact env + smoke-script invocation for each, an expected-events table, and a known-warnings table distinguishing acceptable-in-beta from release-blocking (e.g. a failed relay registration is a warning in pull mode but fatal in push mode). - Linked from the Releases & images page's prerelease section and added to the self-hosting nav under "Self-hosting: release & security". - Drift-guard test (mirrors #1943's pattern): every selfhost_* event name cited in the new doc and script is verified against the real console.log/error call sites in src/server.ts and src/selfhost/ai.ts.
Contributor
|
Important 🟪🟪🟪🟪🟪🟪🟪🟪🟪🟪🟪🟪 🔍 Gittensory is reviewing…AI analysis is in progress. This comment will update when the review is complete. 🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed · 🟪 Reviewing |
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
gittensory-ui | 92b06c0 | Commit Preview URL Branch Preview URL |
Jul 03 2026, 01:18 AM |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2657 +/- ##
=======================================
Coverage 96.09% 96.09%
=======================================
Files 237 237
Lines 26472 26472
Branches 9602 9602
=======================================
Hits 25438 25438
Misses 425 425
Partials 609 609 🚀 New features to boost your workflow:
|
…t.sh Gate review caught two real bugs: (1) the script unconditionally created the network even when SELFHOST_SMOKE_NETWORK was set, breaking the doc's own Postgres/Qdrant examples (network already exists); cleanup then unconditionally removed it too, which would have torn down the caller's network out from under their other containers. (2) the Redis readiness loop fell through silently on timeout instead of failing with a clear error. Also pinned the Qdrant image tag to match docker-compose.yml instead of :latest. Verified live with Docker: a pre-created network now survives the full create-skip + cleanup-skip path, and the default (owned) path still cleans up with no orphaned networks.
…ke-selfhost.sh Gate review caught a real bug: SELFHOST_SMOKE_EXTRA_ENV is line-delimited, so a multiline PEM private key passed through it (exactly what the direct-App checklist scenario documented) gets truncated at the first internal newline and the remaining PEM lines become bogus separate env entries -- the container ends up without a valid private key for the one scenario that most needs it. Added SELFHOST_SMOKE_EXTRA_VOLUMES (same line-delimited -v passthrough pattern) and switched the direct-App doc example to mount the key file + set GITHUB_APP_PRIVATE_KEY_FILE, which src/server.ts already resolves into GITHUB_APP_PRIVATE_KEY at startup like every other *_FILE variable. Verified live with Docker that a multiline file mounts with content intact.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1944
Summary
scripts/smoke-selfhost.sh: a portable, mode-agnostic smoke script. Boots one container against a fresh Redis on an isolated network, waits for health, and asserts on/health,/ready,/metricsplus caller-supplied expected/forbidden startup log events (SELFHOST_SMOKE_EXPECT_EVENTS/_FORBID_EVENTS). Always assertsselfhost_migrations_appliedregardless of mode.selfhost_*event name cited in the new doc and script is verified against the realconsole.log/errorcall sites insrc/server.tsandsrc/selfhost/ai.ts.No dedicated tests for the shell script
scripts/smoke-selfhost.shis a manually-invoked operator tool (like the existingdeploy-selfhost-image.sh/deploy-selfhost-prebuilt.sh, which also have no unit tests) — it boots real Docker containers and isn't unit-testable. Verified withshellcheck(clean) and the drift-guard test above, which does cover its cited event names.Test plan
shellcheck scripts/smoke-selfhost.sh— cleannpm run ui:typecheck,npm run ui:lint,npm run ui:test,npm run ui:build— all cleannpm run test:ci(full local gate, unsharded) — green, including the new drift-guard testnpm audit --audit-level=moderate— 0 vulnerabilities