Repository navigation
Description
Situation
A version upgrade started from the web dashboard reportedly left the current release running when disk usage was high. The dashboard gave no reason for the failure. The report concerns an instance hosted by Rome Cloud.
Symptom
After Rome Cloud accepts the request, the sidebar shows Updating to 1.2.0… while the old release continues to answer. After 40 minutes it shows Update failed with Retry. The Settings version row also shows a generic timeout. The next upgrade confirmation provides no record of the failure.
Expected behavior
- Every WebUI client showing the attempt as
Updatingswitches toUpdate failedand shows the reason in the same view on its next successful status check. A client that reconnects during or after the restart reads the same result. - The sidebar icon for an available upgrade is yellow in light and dark themes. If the most recent completed attempt failed, the confirmation dialog shows its target version and failure reason before the user confirms again. The dialog reads the result even after a reload or from another client.
- If an upgrade fails for lack of disk space, the job cleans up the temporary files it owns. Repeated failed attempts leave no accumulating temporary files from those attempts.
How to reproduce
- Use an isolated fixture with an instance on
1.1.0and an upgrade offer for1.2.0. - Start the upgrade from the sidebar and select Upgrade and restart. Return HTTP
202with{"status":"upgrading","target":"1.2.0"}forPOST /api/instance/upgrade. - Simulate an upgrade that fails after acceptance by keeping
GET /api/build-inforeachable on1.1.0. - Advance the clock to 39 minutes and 57 seconds, then to 40 minutes. Check that the dashboard keeps waiting until the timeout and that it shows no failure reason.
The existing sidebar test covers the timeout with fake time.
pnpm --filter rome-web test src/shell/UpgradeIndicator.test.tsx -t 'fails after 40 minutes'Initial triage
- An isolated run with the Core route and Web upgrade hook reproduced the wait and timeout. A simulated HTTP
507with a JSON error reached the hook immediately. Core route, Web timeout test - System Diagnosis measures the instance's profile volume. Its low-space warning describes that volume. The upgrade job logs are needed to identify which filesystem and operation failed in the reported incident. Disk usage reader
Suspected root cause
The feedback gap is confirmed. After acceptance, the Web upgrade hook polls only the running build version. The old release can answer both while an upgrade is underway and after it fails. The dashboard has no upgrade outcome that distinguishes those states. Web upgrade hook, background upgrade PR
Possible fixes
- Check available space on the filesystems the upgrade uses before accepting it. If space is insufficient, record the attempt as failed and return a structured error with a recovery action.
- Store the outcome in Rome Cloud under the upgrade attempt ID. Expose the result through the existing upgrade check so each watching client can read it. Include the most recent completed attempt when the dashboard opens the confirmation dialog.
- Use the existing theme's warning color for the available-upgrade icon. Use the upgrade status and confirmation views to present the failure reason as text.
- Give each attempt ownership of its staging files. Record disk-space errors promptly and run cleanup even when the attempt fails partway through. Resume interrupted cleanup before staging another attempt. Check the staging area after repeated failures to verify that cleanup reclaims the attempt's space.
Environment
Isolated reproduction on rome-os/rome main at 074ab6359. The production upgrade logs, affected filesystem, and release pair remain to be verified.
Activity
fix(web): join an upgrade already under way, and keep proxy HTML out …