Skip to content

fix: restart the server after an unexpected crash, with backoff - #979

Draft
ianalitis wants to merge 1 commit into
ddalcu:mainfrom
ianalitis:fix/265-crash-recovery
Draft

ianalitis wants to merge 1 commit into
ddalcu:mainfrom
ianalitis:fix/265-crash-recovery

Conversation

@ianalitis

@ianalitis ianalitis commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #265. When the server process died on its own the app raised a modal alert and stayed down, so a model that hit an out-of-memory kill or a transient driver failure needed a human to notice and press Start. Settings > Server gains one row, "If the server exits on its own", with three choices: Ask (today's alert, default), Restart, and Restart + Notify.

What changed

  • app/Sources/MLXServe/Services/CrashRecovery.swift (new) holds the whole policy as pure values, so the behaviour is testable without a real process:
    • CrashRecoveryMode (.ask / .restartQuietly / .restartAndTellMe) is a display preference in UserDefaults (crashRecoveryMode), deliberately not a ServerOptions field, since there is no CLI flag for it and it must not make the launch form dirty.
    • CrashRestartPolicy waits 1, 2, 4, 8, 16 then 30 seconds, six attempts in total. A run that stayed up for two minutes counts as healthy and clears the streak, so a crash loop still stops and shows the model's own error instead of restarting forever. reaction(mode:canRelaunch:uptime:) folds the streak reset into the decision, which is why it is the only thing ServerManager has to trust.
    • CrashRecoveryNotifier posts a soundless local notification for the + Notify mode, bundle-guarded the same way TaskNotifier is so swift test never touches UNUserNotificationCenter.
  • ServerManager:
    • The launch path records the exact arguments it used, and a relaunch replays them. Re-deriving them from the Settings form would bring the server back with flags other than the ones that died, which is a worse bug than the crash it recovered from.
    • transitionToRunning() stamps runningSince, so the backoff counts from the moment the server was actually healthy rather than from exec, keeping a slow model load from being read as an instant crash.
    • handleTermination now distinguishes four cases: unexpected exit while running, a recovery relaunch that died before reporting healthy (same streak, keep backing off), a first start that failed (still "Failed to start"), and exit 0 while nominally running, which is not a crash, so the alert reads "mlx-serve exited" instead of claiming a failure.
    • Stop cancels a pending relaunch and resets the streak, and a manual Start during the wait wins over it, launching the model and flags the user just picked. The pill reads "Starting..." while waiting, so the wait is not an error the user has to dismiss.
  • SettingsView: one SettingsRow with a menu Picker. No dirty-tracking, because it is not a launch option and the form's Reset to Defaults does not own it.
  • ko and zh-Hans catalogs carry the new strings.

Verification

  • bash app/test.sh: 4,194 tests, 35 skipped, 0 failures, including ten new CrashRecoveryPolicyTests cases: the .ask no-op, both automatic modes, no-replay-args falling back to the alert, the full [1, 2, 4, 8, 16, 30] ladder and its exhaustion, the streak reset by a healthy run, a short-lived run continuing the streak, the 120 s stability boundary, explicit reset, the raw-value round-trip with an unknown value landing on .ask, and the default mode.
  • LocalizationTests and KoreanLocalizationTests pass with the new keys.

I did not touch CHANGELOG.md, since it is written per release. No Zig sources or engine behaviour change.

Opened as a draft: #978 asks for new PRs to wait for the big rewrite to land. The change is a single new file plus two surgical edits to existing call sites (start/startHeadless/stop/handleTermination), so it should re-base cleanly, and I can rebase it on request once #978 is merged.


Agent acting on behalf of @ianalitis

…cu#265)

An unexpected exit used to leave a modal alert and a dead server, so a model
that hit an out-of-memory kill or a driver hiccup stayed down until someone
noticed. Settings > Server gains "If the server exits on its own": Ask (the
old behaviour, default), Restart, and Restart + Notify.

`CrashRestartPolicy` is pure and unit-tested: delays of 1, 2, 4, 8, 16 and 30
seconds, six attempts, and the streak reset by a run that stayed up for two
minutes, so a crash loop still stops with the model's own error instead of
restarting forever. Stop cancels a pending relaunch, and a manual Start wins
over it with the model and flags the user just picked.

The relaunch REPLAYS the recorded launch arguments rather than re-deriving
them from the current form, and backoff counts from the moment the process
reported healthy, not from launch. An exit 0 while nominally running is not
a crash: the alert says "mlx-serve exited" instead of claiming a failure.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request] MLX Core: Auto-restart the server after an unexpected crash (with backoff)

1 participant