Cornelius seeder marks itself seeded on a volume-conflict 409 — Cornelius never created, never retried

[dolho]

Summary

cornelius_agent_service treats any HTTP 409 from the create path as "Cornelius already exists" and durably sets cornelius_seeded=true. But the create path also raises 409 for an entirely different condition — the #1671/#1664 unclaimed-data-volume refusal. When that fires, Cornelius is never created and never retried, on an install where the flag says it was seeded.

Observed on v0.8.5 (b8cf94ca), fresh install, empty DB, no cornelius agent and no agent_ownership row.

Evidence

Backend log, fresh install:

services.cornelius_agent_service  Brain Orb enabled by default (seeded brain_orb_enabled=true)
services.cornelius_agent_service  Cornelius seed: Cornelius already exists (409) — marking seeded

Resulting state — flag set, agent absent:

sqlite> select key from system_settings where key like '%seed%';
cornelius_seeded = true
sqlite> select count(*) from agent_ownership where agent_name='cornelius';
0

The only thing present was a leftover agent-cornelius-workspace volume with no owning row.

Root cause

services/agent_service/crud.py:303 raises HTTPException(status_code=409, …) for the volume-claim refusal:

Data volume 'agent-X-workspace' already exists and no agent claims it, so its contents would silently become this agent's home directory. Refusing to adopt another agent's leftover data.

services/cornelius_agent_service.py:148-154 branches on the status code alone:

# 409 = the agent already exists (another worker won, or a prior partial
# run). Treat as success: converge the flag so we stop retrying.
status_code = getattr(e, "status_code", None)
if status_code == 409:
    db.set_setting(_SEEDED_FLAG, "true")
    self._seed_brain_orb_flag()
    return self._skip(result, "already_exists", "Cornelius already exists (409) — marking seeded")

Two distinct conditions share one status code. "Another worker won" is genuinely converged; "a stale volume blocked creation" is a hard failure that is now unrecoverable, because the flag suppresses every later attempt (and cornelius_seeded=="true" additionally forces a NOT-fresh verdict in system_seed_service.ensure_first_run_seeded()).

Contrast — the sibling seeder gets this right

system_seed_service deliberately does not set its flag when 0 agents are created, so it retries on the next boot. Verified: after clearing the blocking volumes and restarting the backend, the starter fleet deployed correctly on the retry pass. Cornelius has no equivalent recovery.

Repro

1. Create a Trinity instance, let Cornelius seed, then docker compose down -v (destroys the DB; agent volumes are not compose-managed and survive).
2. Re-run ./scripts/deploy/start.sh and complete setup.
3. The leftover agent-cornelius-workspace triggers the 409 → cornelius_seeded=true, no Cornelius.

This is the ordinary "let me start over" flow, so it is reachable without any exotic setup.

Suggested direction

Distinguish the two 409s before converging the flag — e.g. a typed exception or an error code on the volume-claim refusal — and treat the volume conflict as retryable (or surface it as an operator alert) rather than as success.

[dolho]

Severity recalibrated priority-p1 → priority-p2 on review. Only reachable on a re-install over a host that still has an agent-cornelius-workspace volume (e.g. docker compose down -v then re-setup). A genuinely fresh machine is unaffected, so real-world exposure is small.