Skip to content

deploy-selfhost-prebuilt.sh reports complete without waiting for container health, unlike its sibling deploy-selfhost-image.sh #8395

Description

@JSONbored

Context

scripts/deploy-selfhost-image.sh and scripts/deploy-selfhost-prebuilt.sh are documented as parallel
deploy paths — both headers say "After a successful run, verify with:
./scripts/selfhost-post-update-check.sh" and both are supported as standalone, directly-invoked
entry points (each shows its own bare ./scripts/deploy-selfhost-*.sh usage example in its header).

deploy-selfhost-image.sh blocks on the deployed container becoming healthy before reporting success:
it calls wait_for_healthy (defined in that file), which polls docker inspect's health status for up
to SELFHOST_HEALTH_TIMEOUT_SECONDS (default 180s) and, on timeout, prints the container's status +
last 80 log lines to stderr and exits 1 — the script only ever prints "selfhost image deploy:
complete" after a real, confirmed-healthy container.

scripts/deploy-selfhost-prebuilt.sh has no equivalent. Its run_compose_deploy function ends with:

echo "selfhost deploy: restarting $SERVICE"
maybe_infisical_run docker compose "${compose_args[@]}" up -d --no-deps "$SERVICE"

and the top-level script immediately proceeds to echo "selfhost deploy: complete ($SENTRY_RELEASE)"
and exits 0 — docker compose up -d only confirms the container was created and started, not that it
became healthy. An operator running ./scripts/deploy-selfhost-prebuilt.sh directly (its own
documented usage) gets a false "complete" success signal even if the freshly-built image crash-loops or
never passes its health check, unlike an operator running deploy-selfhost-image.sh.

Note: scripts/selfhost-update.sh (the higher-level git-pull-and-update orchestrator) happens to call
deploy-selfhost-prebuilt.sh and then separately run selfhost-post-update-check.sh afterward by
default — so the orchestrated path is covered. This issue is about the asymmetry when
deploy-selfhost-prebuilt.sh is run on its own, exactly as its own header documents.

wait_for_healthy currently lives only inside deploy-selfhost-image.sh, not in the shared
scripts/lib/selfhost-deploy-common.sh (which already holds require_cmd, env_get/env_put,
maybe_infisical_run, compose_file_args — the helpers both deploy scripts already share).

Requirements

  • Move wait_for_healthy (and the SERVICE/HEALTH_TIMEOUT_SECONDS variables it depends on, or
    parameterize them as arguments) from deploy-selfhost-image.sh into
    scripts/lib/selfhost-deploy-common.sh, so both deploy scripts can call the same implementation
    rather than one hand-copying the other.
  • Update deploy-selfhost-image.sh to call the shared version (no behavior change for that script).
  • Call the shared wait_for_healthy from deploy-selfhost-prebuilt.sh's run_compose_deploy (or
    immediately after it in the top-level flow) after docker compose up -d --no-deps "$SERVICE",
    respecting the same SELFHOST_SERVICE/SELFHOST_HEALTH_TIMEOUT_SECONDS env var overrides
    deploy-selfhost-image.sh already supports, so the two scripts' health-check behavior is consistent.
  • deploy-selfhost-prebuilt.sh must exit non-zero (with the same diagnostic docker compose ps/logs --tail=80 output wait_for_healthy already produces) if the container never becomes healthy within
    the timeout, matching deploy-selfhost-image.sh's existing behavior exactly.
  • Do not change docker-compose.yml's health check definition itself, and do not change
    selfhost-post-update-check.sh/selfhost-update.sh — this is scoped to closing the gap between the
    two deploy scripts' own inline behavior.

Deliverables

  • wait_for_healthy lives in scripts/lib/selfhost-deploy-common.sh, shared by both deploy
    scripts.
  • deploy-selfhost-prebuilt.sh blocks on and reports the same pass/fail health-check outcome
    deploy-selfhost-image.sh already does, instead of declaring "complete" immediately after
    docker compose up -d.

Test Coverage Requirements

These are self-host operator shell scripts with no Docker daemon available in CI to exercise them
end-to-end, and scripts/** is excluded from Codecov's coverage.include (codecov.yml) regardless.
Verification: shellcheck (if configured locally; this repo's scripts already use
# shellcheck source=... pragmas) on both edited files plus scripts/lib/selfhost-deploy-common.sh
with no new warnings, and a manual local run of both ./scripts/deploy-selfhost-image.sh and
./scripts/deploy-selfhost-prebuilt.sh against a real (or intentionally broken, to exercise the
timeout/failure path) self-host docker-compose stack, confirming both now exit non-zero with the same
diagnostic output shape when the service never becomes healthy, and exit 0 only once it does.

Expected Outcome

Running ./scripts/deploy-selfhost-prebuilt.sh directly gives an operator the same blocking,
fail-loud-with-diagnostics health confirmation ./scripts/deploy-selfhost-image.sh already gives,
instead of a false "complete" the moment the container starts.

Links & Resources

  • scripts/deploy-selfhost-image.sh (wait_for_healthy, the implementation to share)
  • scripts/deploy-selfhost-prebuilt.sh (run_compose_deploy, where the check is missing)
  • scripts/lib/selfhost-deploy-common.sh (the shared-helpers file to add wait_for_healthy to)
  • scripts/selfhost-post-update-check.sh (the separate, already-existing manual/orchestrated health
    verification — not itself changed by this issue)

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions