Problem
When the conductor process backing the web dashboard crashes or dies unexpectedly (e.g. a silent crash in --web-bg mode, an OOM kill, a SystemExit escaping the engine — see issue #116), the dashboard's WebSocket client (useWebSocket in src/conductor/web/frontend/src/hooks/use-websocket.ts) detects the dropped connection and enters reconnecting status. It then retries with exponential backoff (scheduleReconnect, capped at MAX_RECONNECT_DELAY = 30000ms) forever, with no upper bound on retry duration and no escalation.
The only visible UI feedback is a small spinner + "Reconnecting…" label in the status bar (StatusBar.tsx wsIndicator, case 'reconnecting'). Because the workflow status shown elsewhere in the UI (workflowStatus) is left at whatever it was before the connection dropped (typically 'running'), the dashboard looks like the workflow is still actively executing indefinitely. A user who isn't watching closely can be misled for a long time into believing the workflow is still in progress when the backing process actually died minutes or hours earlier.
Expected behavior
If the dashboard has been stuck in the reconnecting state for longer than some threshold (e.g. 60s), it should surface a prominent warning banner indicating that:
- The Conductor workflow process may have crashed/silently failed and is no longer running.
- Where to look for logs to investigate, i.e. the
bg_stderr_log / bg_stdout_log paths and the .events.jsonl file already stamped into the workflow_started event's system metadata for --web-bg runs (see WorkflowEngine.build_workflow_started_data() in src/conductor/engine/workflow.py, and the "Debugging --web-bg failures" section of AGENTS.md).
The warning should only disappear once the connection is actually restored (wsStatus returns to 'connected'), not on a timer — a refresh should not be the only way to dismiss the stale "running" impression.
Suggested implementation notes
- Track how long
wsStatus === 'reconnecting' has been continuously true (e.g. a timestamp set when transitioning into reconnecting, checked against a timer) in workflow-store.ts or a small hook, rather than counting retry attempts (backoff makes attempt-count an unreliable proxy for elapsed time).
- Render the warning banner near/above the existing status bar reconnecting indicator so it's hard to miss, similar treatment to the existing
workflowFailure banner.
- Surface
system.bg_stderr_log / system.bg_stdout_log (currently captured in the workflow_started event but not read anywhere in the frontend store/types) in the banner text when available; otherwise point at the default log locations documented in AGENTS.md.
- Add a Vitest test (mirroring the existing
workflow-store.test.ts / graph-layout.test.ts patterns) that verifies the warning appears after the timeout while reconnecting, and clears on reconnect.
Where this lives
src/conductor/web/frontend/src/hooks/use-websocket.ts — reconnect loop / backoff
src/conductor/web/frontend/src/stores/workflow-store.ts — WsStatus, setWsStatus
src/conductor/web/frontend/src/components/layout/StatusBar.tsx — current reconnecting indicator
src/conductor/engine/workflow.py — build_workflow_started_data() (log path metadata)
Problem
When the conductor process backing the web dashboard crashes or dies unexpectedly (e.g. a silent crash in
--web-bgmode, an OOM kill, aSystemExitescaping the engine — see issue #116), the dashboard's WebSocket client (useWebSocketinsrc/conductor/web/frontend/src/hooks/use-websocket.ts) detects the dropped connection and entersreconnectingstatus. It then retries with exponential backoff (scheduleReconnect, capped atMAX_RECONNECT_DELAY = 30000ms) forever, with no upper bound on retry duration and no escalation.The only visible UI feedback is a small spinner + "Reconnecting…" label in the status bar (
StatusBar.tsxwsIndicator, case'reconnecting'). Because the workflow status shown elsewhere in the UI (workflowStatus) is left at whatever it was before the connection dropped (typically'running'), the dashboard looks like the workflow is still actively executing indefinitely. A user who isn't watching closely can be misled for a long time into believing the workflow is still in progress when the backing process actually died minutes or hours earlier.Expected behavior
If the dashboard has been stuck in the
reconnectingstate for longer than some threshold (e.g. 60s), it should surface a prominent warning banner indicating that:bg_stderr_log/bg_stdout_logpaths and the.events.jsonlfile already stamped into theworkflow_startedevent'ssystemmetadata for--web-bgruns (seeWorkflowEngine.build_workflow_started_data()insrc/conductor/engine/workflow.py, and the "Debugging--web-bgfailures" section ofAGENTS.md).The warning should only disappear once the connection is actually restored (
wsStatusreturns to'connected'), not on a timer — a refresh should not be the only way to dismiss the stale "running" impression.Suggested implementation notes
wsStatus === 'reconnecting'has been continuously true (e.g. a timestamp set when transitioning intoreconnecting, checked against a timer) inworkflow-store.tsor a small hook, rather than counting retry attempts (backoff makes attempt-count an unreliable proxy for elapsed time).workflowFailurebanner.system.bg_stderr_log/system.bg_stdout_log(currently captured in theworkflow_startedevent but not read anywhere in the frontend store/types) in the banner text when available; otherwise point at the default log locations documented inAGENTS.md.workflow-store.test.ts/graph-layout.test.tspatterns) that verifies the warning appears after the timeout whilereconnecting, and clears on reconnect.Where this lives
src/conductor/web/frontend/src/hooks/use-websocket.ts— reconnect loop / backoffsrc/conductor/web/frontend/src/stores/workflow-store.ts—WsStatus,setWsStatussrc/conductor/web/frontend/src/components/layout/StatusBar.tsx— current reconnecting indicatorsrc/conductor/engine/workflow.py—build_workflow_started_data()(log path metadata)