Fix full E2E coverage for AGN CLI, platform egress, and real-agent Ziti sidecars

[rowan-stein]

User request

E2E tests should be fully working. Latest bootstrap reportedly starts workloads that fail on the Ziti sidecar path, and full-cycle tests with messages and real agents must pass.

A fresh rerun of AGN CLI E2E failed: https://github.com/agynio/e2e/actions/runs/28334862714

Observed from that rerun:

• Bootstrap provisioning succeeded.
• Bootstrap health check reported the platform namespace and Argo CD applications healthy.
• Ziti controller/router logs showed initialization and terminator activity.
• go-agn-cli failed in TestAgynEgressRuleLifecycle because latest agn returned:

Error: unknown command "egress" for "agn"

• Diagnostics showed:

No resources found in agyn-workloads namespace.

So the rerun failed before exercising workload pod creation or Ziti sidecar initialization.

Research/specification

Source of truth is agynio/architecture.

Expected real-agent path from architecture:

1. A user/thread message creates an unacknowledged agent message.
2. Agents Orchestrator assembles a workload, including init image, env/secrets, egress CA, and OpenZiti enrollment.
3. Orchestrator calls Ziti Management to create an agent identity for (agent_id, workload_id) and receives an enrollment JWT.
4. Orchestrator injects that enrollment into the Ziti startup path and calls Runner StartWorkload.
5. k8s-runner creates a pod in agyn-workloads.
6. The pod’s Ziti init/restartable sidecar enrolls identity, configures TPROXY/DNS, and enables .ziti access to gateway/llm/tracing.
7. The agent reaches Gateway/Threads and posts a response.

Relevant architecture references:

• architecture/agents-orchestrator.md
• architecture/openziti.md
• architecture/k8s-runner.md
• architecture/operations/e2e-testing.md
• architecture/agyn-cli.md
• architecture/agn-cli.md
• changes/2026-05-14-egress-gateway.md

Key findings:

• agnio/agn-cli / agn is the agent loop CLI and does not implement platform resource commands like egress.
• Platform egress commands belong to agynio/agyn-cli / agyn.
• suites/go-agn-cli/tests/egress_test.go currently invokes agn egress rule ..., which is invalid against latest agn.
• Existing real-agent/message/workload tests include:
  • suites/go-core/tests/pipeline_test.go
  • TestFullPipelineMessageResponse
  • TestFullPipelineAgnMessageResponse
  • TestFullPipelineClaudeMessageResponse
  • suites/go-core/tests/agent_agyn_wait_test.go
  • TestAgentAgynCLIWaitToAnotherAgent
  • suites/go-core/tests/mcp_test.go
• The important real-agent + platform CLI test is TestAgentAgynCLIWaitToAnotherAgent, tagged svc_agents_orchestrator; it is not currently guaranteed by the AGN CLI workflow.
• Current default suite selection is inconsistent with the “full suite” architecture contract. In particular, some suites run smoke-only or nothing when TAGS is empty.

Implementation requirements

1. Fix AGN CLI suite ownership

The AGN CLI workflow/suite must only test agn agent-loop behavior.

• Update/remove suites/go-agn-cli/tests/egress_test.go from the go-agn-cli suite.
• Do not run agn egress ... anywhere.
• Ensure go-agn-cli still covers latest agn release behavior: summarization, tool pair, system prompt, exec/state/resume, etc.

2. Move/restore platform egress CLI coverage to agyn

If egress CLI coverage is retained, it must run against agynio/agyn-cli / agyn, not agn.

Possible implementation:

• Add a new suite such as suites/go-agyn-cli or equivalent.
• Add staging support for an AGYN_BINARY in .github/actions/run-tests/action.yml.
• Add/adjust workflow that downloads agyn-linux-amd64 from agynio/agyn-cli and runs the platform CLI egress lifecycle tests.

3. Ensure full/default E2E actually exercises real workloads

devspace run test-e2e with empty tags should match the architecture expectation of full E2E coverage, not silently run smoke-only subsets or skip chat/agent suites.

Review and fix suite selection for:

• suites/go-core/suite.yaml
• suites/playwright-chat-app/suite.yaml
• suites/playwright-tracing-app/suite.yaml
• suites/go-agn-cli/suite.yaml

At minimum, the full/default path must include real-agent message/workload coverage.

4. Add explicit Ziti sidecar assertions to real-agent tests

For at least one real-agent/message flow, preferably TestAgentAgynCLIWaitToAnotherAgent and/or TestFullPipelineAgnMessageResponse, assert that a real workload pod was created and that Ziti sidecar initialization succeeded.

Acceptance-level assertions should include:

• At least one workload-* pod exists in agyn-workloads for the test.
• The pod has the expected Ziti init/restartable sidecar containers.
• ziti-enroll completed successfully.
• ziti-gateway-wait completed successfully.
• ziti-sidecar is running/healthy under restartable init semantics.
• The agent still posts the expected response.

A test must fail if no workload pod is created; “No resources found in agyn-workloads namespace” cannot be considered acceptable for this path.

5. Improve diagnostics for workload sidecar failures

On failure, artifacts should include:

• kubectl get/describe pod for workload pods in agyn-workloads.
• Events from agyn-workloads.
• Logs for all workload containers and init containers, including previous logs where available.
• Platform logs for agents-orchestrator, k8s-runner, gateway, ziti-management.
• Ziti controller/router logs.

6. Document verification

Document the targeted and full verification commands in README or E2E docs.

Suggested commands:

# AGN agent-loop CLI suite only
E2E_SUITES=go-agn-cli \
AGN_BINARY=/path/to/agn \
devspace run test-e2e --tag svc_agn_cli

# Targeted real-agent/Ziti sidecar test
E2E_SUITES=go-core \
E2E_GO_TEST_RUN='TestAgentAgynCLIWaitToAnotherAgent' \
devspace run test-e2e --tag svc_agents_orchestrator

# Targeted single-agent full pipeline
E2E_SUITES=go-core \
E2E_GO_TEST_RUN='TestFullPipelineAgnMessageResponse' \
devspace run test-e2e --tag svc_agents_orchestrator

# Full suite
 devspace run test-e2e

Acceptance criteria

• TestAgynEgressRuleLifecycle no longer runs against the agn binary.
• go-agn-cli passes against latest agynio/agn-cli release.
• Platform CLI egress coverage, if retained, runs against latest agynio/agyn-cli using an agyn binary.
• Full/default E2E selection includes real-agent workload coverage and does not silently skip it.
• A dedicated real-agent/Ziti path runs and creates at least one pod in agyn-workloads.
• The real-agent/Ziti path explicitly validates Ziti sidecar initialization, not just final thread response.
• Sidecar failures produce useful workload and Ziti diagnostics.
• No skips are introduced to hide failing real-agent, workload, or Ziti paths.
• CI / required E2E verification passes on the resulting PR.

[rowan-stein]

Update after live workload reruns and architecture-first investigation:

• PR fix(e2e): restore real agent CLI coverage #214 is correctly exposing the real workload/Ziti failure; it should not be treated as green yet.
• Latest relevant runs:
  • https://github.com/agynio/e2e/actions/runs/28338256517
    • platform-usage: passed
    • k8s-runner-workloads: passed
    • orchestrator-workloads: failed in this PR run
    • real-agent-ziti: failed
  • https://github.com/agynio/e2e/actions/runs/28338265538
    • real-agent-ziti: failed
• Failure evidence:
  • ziti-sidecar logs show ziti edge enroll path/to/jwt [flags] and -j, --jwt string Enrollment token (JWT file). Required.
  • ziti-gateway-wait times out waiting for gateway.ziti.
  • agent containers remain PodInitializing.

Research conclusion:

• The remaining failure is upstream of agynio/e2e.
• Primary owning repo/file/function appears to be agynio/agents-orchestrator, internal/assembler/assembler.go, Assembler.Assemble, specifically the ziti-enroll, ziti-sidecar, and ziti-gateway-wait init container assembly.
• The sidecar currently uses openziti/ziti-tunnel:2.0.0-pre8 with args for tproxy but no explicit entrypoint override. The image entrypoint performs enrollment bootstrap before running the tunnel, conflicting with the orchestrator's separate ziti-enroll init-container model.
• Expected fix direction: make enrollment single-owner and ensure the sidecar only runs the tunnel from the generated identity, or intentionally let the image entrypoint own enrollment once. If keeping the separate ziti-enroll init container, the ziti-sidecar must bypass image enrollment bootstrap via an explicit command/entrypoint or compatible image contract.

Next coordination step: Casey is being routed to fix the upstream agents-orchestrator workload Ziti sidecar contract. PR #214 remains the live E2E coverage/detection PR and should only proceed to review after the real workload tests pass against the fixed platform.