bug: a rebuilt agent base image is never picked up — no image-drift predicate, so the v0.8.5 "stop/start-with-recreate" upgrade step is a no-op

[AndriiPasternak31]

Summary

A rebuilt trinity-agent-base image is never picked up by an agent restart. needs_recreation in start_agent_internal compares eight config surfaces — none of them the image — so POST /api/agents/{name}/start on a running agent takes the no-op path and the container keeps running the old base image indefinitely.

This makes the v0.8.5 upgrade instruction incorrect. docs/releases/0.8.5.md tells operators:

Operator action: run ./scripts/deploy/build-base-image.sh, then recreate agents (a config-drift recreate or stop/start-with-recreate picks up the new image).

There is no such thing as "stop/start-with-recreate" — stop+start reuses the existing container. An operator who follows the release notes exactly ends up with a fleet still on the old image while believing the upgrade landed, and the release notes list what that silently costs: no PAT-free public-template clone (trinity-enterprise#123), no git-maintenance ownership (#1595/#1596), no preinstalled gh CLI (#1574), no pull-worker scaffolding.

Observed on a live 23-agent instance upgrading v0.8.0 → v0.8.5: after build-base-image.sh succeeded, every agent still resolved to the old image id, and a plain start changed nothing.

Root cause

services/agent_service/lifecycle.py:287-296:

needs_recreation = (
    not shared_folder_match or
    not check_public_folder_mount_matches(container, agent_name) or
    not check_api_key_env_matches(container, agent_name) or
    not check_github_pat_env_matches(container, agent_name) or
    not check_resource_limits_match(container, agent_name) or
    not check_full_capabilities_match(container, agent_name) or
    not check_guardrails_env_matches(container, agent_name) or
    not check_agent_auth_token_env_matches(container, agent_name)
)

All eight predicates live in agent_service/helpers.py (plus file_sharing.py); none reads container.attrs["Image"]. The only Image reference in the whole module is the write side of the recreate:

# lifecycle.py:413 — reads the TAG to build the replacement, never compares
image = old_config.get("Image", "trinity-agent-base:latest")

That line is also why the fix is cheap: Config.Image is the tag (trinity-agent-base:latest), so once a recreate is triggered for any reason it already resolves to the freshly built image. The only missing piece is the trigger.

Suggested fix

Add a ninth predicate — the container's resolved image id vs. what its tag resolves to now:

def check_base_image_matches(container, agent_name: str) -> bool:
    """False when the container runs an image id that its own tag no longer
    resolves to (i.e. the base image was rebuilt underneath it)."""
    running = container.attrs.get("Image")                       # sha256:… (resolved)
    tag = container.attrs.get("Config", {}).get("Image")         # trinity-agent-base:latest
    if not running or not tag:
        return True                                              # fail-safe: no churn
    try:
        return docker_client.images.get(tag).id == running
    except Exception:
        return True                                              # unreadable → never recreate

Fail-open on every error path matters here: a false negative recreates the whole fleet.

Worth deciding explicitly whether this should be opt-in, since making it unconditional means the next build-base-image.sh recreates every agent the moment anything calls start — desirable for an upgrade, surprising if it fires implicitly. An ?upgrade_image=true query param on the start endpoint, or a dedicated POST /api/agents/{name}/upgrade-image, would make the intent explicit and give the UI something to hang a "base image out of date" badge on.

Workaround (what I actually did)

Forced the drift branch and let the unmodified production path do the rest, so every agent still got the real recreate + credential/skill/read-only injection sequence:

lifecycle.check_resource_limits_match = lambda container, agent_name: False
await lifecycle.start_agent_internal(name)

Run via docker exec -i -w /app trinity-backend python3 - < script.py. 23/23 agents moved to the new image, 0 failures, workspace volumes preserved.

(Note for anyone reproducing: /tmp in the backend container is a noexec tmpfs, so docker cp into it silently vanishes — pipe the script via stdin and pass arguments through env vars.)

Acceptance criteria

• After build-base-image.sh, a running agent whose image tag now resolves elsewhere is detected as drifted
• The recreate preserves mounts, env, and the workspace volume (i.e. goes through recreate_container_with_updated_config, not the #1559 recovery path — see the sibling issue)
• An agent already on the current image is not recreated (no churn on ordinary starts)
• All predicate error paths fail open (unreadable image/tag ⇒ no recreate)
• docs/releases/*.md upgrade wording is corrected to name the actual mechanism

Environment

Trinity 0.8.5+ge9eb40da, prod compose, 23 agents, SQLite, single-host Docker.

[webmixgamer]

Claimed — starting work.