Skip to content

web: hosted instance upgrades stay in progress after they fail #2364

Open

Description

@Asuka109

Situation

A version upgrade started from the web dashboard reportedly left the current release running when disk usage was high. The dashboard gave no reason for the failure. The report concerns an instance hosted by Rome Cloud.

Symptom

After Rome Cloud accepts the request, the sidebar shows Updating to 1.2.0… while the old release continues to answer. After 40 minutes it shows Update failed with Retry. The Settings version row also shows a generic timeout. The next upgrade confirmation provides no record of the failure.

Expected behavior

  • Every WebUI client showing the attempt as Updating switches to Update failed and shows the reason in the same view on its next successful status check. A client that reconnects during or after the restart reads the same result.
  • The sidebar icon for an available upgrade is yellow in light and dark themes. If the most recent completed attempt failed, the confirmation dialog shows its target version and failure reason before the user confirms again. The dialog reads the result even after a reload or from another client.
  • If an upgrade fails for lack of disk space, the job cleans up the temporary files it owns. Repeated failed attempts leave no accumulating temporary files from those attempts.

How to reproduce

  1. Use an isolated fixture with an instance on 1.1.0 and an upgrade offer for 1.2.0.
  2. Start the upgrade from the sidebar and select Upgrade and restart. Return HTTP 202 with {"status":"upgrading","target":"1.2.0"} for POST /api/instance/upgrade.
  3. Simulate an upgrade that fails after acceptance by keeping GET /api/build-info reachable on 1.1.0.
  4. Advance the clock to 39 minutes and 57 seconds, then to 40 minutes. Check that the dashboard keeps waiting until the timeout and that it shows no failure reason.

The existing sidebar test covers the timeout with fake time.

pnpm --filter rome-web test src/shell/UpgradeIndicator.test.tsx -t 'fails after 40 minutes'

Initial triage

  • An isolated run with the Core route and Web upgrade hook reproduced the wait and timeout. A simulated HTTP 507 with a JSON error reached the hook immediately. Core route, Web timeout test
  • System Diagnosis measures the instance's profile volume. Its low-space warning describes that volume. The upgrade job logs are needed to identify which filesystem and operation failed in the reported incident. Disk usage reader

Suspected root cause

The feedback gap is confirmed. After acceptance, the Web upgrade hook polls only the running build version. The old release can answer both while an upgrade is underway and after it fails. The dashboard has no upgrade outcome that distinguishes those states. Web upgrade hook, background upgrade PR

Possible fixes

  1. Check available space on the filesystems the upgrade uses before accepting it. If space is insufficient, record the attempt as failed and return a structured error with a recovery action.
  2. Store the outcome in Rome Cloud under the upgrade attempt ID. Expose the result through the existing upgrade check so each watching client can read it. Include the most recent completed attempt when the dashboard opens the confirmation dialog.
  3. Use the existing theme's warning color for the available-upgrade icon. Use the upgrade status and confirmation views to present the failure reason as text.
  4. Give each attempt ownership of its staging files. Record disk-space errors promptly and run cleanup even when the attempt fails partway through. Resume interrupted cleanup before staging another attempt. Check the staging area after repeated failures to verify that cleanup reclaims the attempt's space.

Environment

Isolated reproduction on rome-os/rome main at 074ab6359. The production upgrade logs, affected filesystem, and release pair remain to be verified.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Fix in the normal course of workbugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions