Feature: Periodic / milestone checkpoints so stalled runs are recoverable

[jrob5756]

Summary

Checkpoints are only written when the engine catches an exception (_save_checkpoint_on_failure in engine/workflow.py). A workflow that runs for hours and stalls — provider hang, MCP server deadlock, network blip, sub-agent that never returns — produces zero recoverable state. conductor checkpoints shows nothing, and conductor resume has nothing to resume.

This is especially painful for long, expensive runs (e.g., multi-epic implement workflows with parallel coder agents). After 18+ hours of work and ~$1+ in tokens, the only recovery path is to either wait for an exception to bubble up, hard-kill and lose everything, or replay the events.jsonl for diagnosis only (cannot resume from it).

Repro

1. Start any long-running workflow with conductor run workflow.yaml --web-bg.
2. While it's mid-iteration, simulate a stall (e.g., disconnect the network, or pick a workflow where one agent reliably hangs).
3. Run conductor checkpoints workflow.yaml — empty for that run.
4. The dashboard still shows "Running" but no progress is being made.

Expected

A periodic / milestone-based checkpoint so a stalled run is recoverable without an exception.

Proposal

Add an opt-in checkpoint: block to the workflow runtime config, e.g.:

runtime:
  checkpoint:
    every_agent: true        # save after each agent completes successfully
    every_seconds: 300       # OR/AND time-based
    keep_last: 5             # rotation

Defaults could stay off to preserve current behavior. The existing _save_checkpoint_on_failure path already builds the full state object — a successful-path caller after each agent transition would be a small change.

Workaround today

Replay $TMPDIR/conductor/conductor-<workflow>-<ts>-<id>.events.jsonl for diagnostics. No way to resume execution from it.

Environment

• Conductor v0.1.18
• Windows