Skip to content

Web dashboard shows no warning when stuck reconnecting after a silent conductor crash #330

Description

Problem

When the conductor process backing the web dashboard crashes or dies unexpectedly (e.g. a silent crash in --web-bg mode, an OOM kill, a SystemExit escaping the engine — see issue #116), the dashboard's WebSocket client (useWebSocket in src/conductor/web/frontend/src/hooks/use-websocket.ts) detects the dropped connection and enters reconnecting status. It then retries with exponential backoff (scheduleReconnect, capped at MAX_RECONNECT_DELAY = 30000ms) forever, with no upper bound on retry duration and no escalation.

The only visible UI feedback is a small spinner + "Reconnecting…" label in the status bar (StatusBar.tsx wsIndicator, case 'reconnecting'). Because the workflow status shown elsewhere in the UI (workflowStatus) is left at whatever it was before the connection dropped (typically 'running'), the dashboard looks like the workflow is still actively executing indefinitely. A user who isn't watching closely can be misled for a long time into believing the workflow is still in progress when the backing process actually died minutes or hours earlier.

Expected behavior

If the dashboard has been stuck in the reconnecting state for longer than some threshold (e.g. 60s), it should surface a prominent warning banner indicating that:

  1. The Conductor workflow process may have crashed/silently failed and is no longer running.
  2. Where to look for logs to investigate, i.e. the bg_stderr_log / bg_stdout_log paths and the .events.jsonl file already stamped into the workflow_started event's system metadata for --web-bg runs (see WorkflowEngine.build_workflow_started_data() in src/conductor/engine/workflow.py, and the "Debugging --web-bg failures" section of AGENTS.md).

The warning should only disappear once the connection is actually restored (wsStatus returns to 'connected'), not on a timer — a refresh should not be the only way to dismiss the stale "running" impression.

Suggested implementation notes

  • Track how long wsStatus === 'reconnecting' has been continuously true (e.g. a timestamp set when transitioning into reconnecting, checked against a timer) in workflow-store.ts or a small hook, rather than counting retry attempts (backoff makes attempt-count an unreliable proxy for elapsed time).
  • Render the warning banner near/above the existing status bar reconnecting indicator so it's hard to miss, similar treatment to the existing workflowFailure banner.
  • Surface system.bg_stderr_log / system.bg_stdout_log (currently captured in the workflow_started event but not read anywhere in the frontend store/types) in the banner text when available; otherwise point at the default log locations documented in AGENTS.md.
  • Add a Vitest test (mirroring the existing workflow-store.test.ts / graph-layout.test.ts patterns) that verifies the warning appears after the timeout while reconnecting, and clears on reconnect.

Where this lives

  • src/conductor/web/frontend/src/hooks/use-websocket.ts — reconnect loop / backoff
  • src/conductor/web/frontend/src/stores/workflow-store.tsWsStatus, setWsStatus
  • src/conductor/web/frontend/src/components/layout/StatusBar.tsx — current reconnecting indicator
  • src/conductor/engine/workflow.pybuild_workflow_started_data() (log path metadata)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

area:webWeb dashboard (server + frontend)bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions