Chat-app E2E failures after llm-proxy version bump (agent replies / notifications / tracing)

[rowan-stein]

Chat-app CI E2E is failing; suspected relation to llm-proxy service version bump.

Example failing run:

• https://github.com/agynio/chat-app/actions/runs/25715031628/job/76935131192

Observed symptoms in Playwright:

• Many request failures net::ERR_ABORTED to Chat/Gateway endpoints (notably NotificationsGateway/Subscribe).
• Agent reply never arrives (Agent did not reply within 180000ms).
• Tracing link expectations fail (ends on /message/... rather than /runs/...).
• Inline media render expectations fail (mermaid/vega not present), likely cascading from upstream failures.

Hypothesis:

• Recent llm-proxy version bump breaks agent execution or related platform integrations.

Requested investigation:

• Reproduce in a local cluster using the latest bootstrap.
• Identify root cause (llm-proxy vs other dependent service).
• Provide a concrete fix and validate E2E passes.

[casey-brooks]

Investigation update

I investigated the chat-app E2E failures from the failing run https://github.com/agynio/chat-app/actions/runs/25715031628/job/76935131192.

Local cluster attempt

• Tried to provision the latest agynio/bootstrap@main locally with ./apply.sh -y.
• The local container environment blocked k3d startup before Kubernetes came up:
  • failed to ensure tools node ... unable to start container process: error mounting "sysfs" to rootfs at "/sys" ... operation not permitted
• Because of that Docker/sysfs permission issue, I used the provided failing CI run artifacts/logs plus repository/version diffs as concrete evidence.

Evidence from failing chat-app run

• Bootstrap deployed llm-proxy chart/image 0.12.2:
  • argocd_application.llm_proxy with chart = "agynio/charts/llm-proxy"
  • target_revision = "0.12.2"
  • Helm image tag 0.12.2
• Platform pods were healthy at failure time, including:
  • llm-proxy-llm-proxy-c7fcd5fd-pkqqc 1/1 Running
  • agents-orchestrator-5f7f98648d-7dzc8 1/1 Running
  • gateway-gateway-7cd54c6587-tph2g 1/1 Running
  • notifications-d49846dd5-5wph2 1/1 Running
  • chat-app-7647876c6c-nn7rc 1/1 Running
• Playwright/JUnit symptoms:
  • chat-agent-response timed out after 180s waiting for an agent reply.
  • chat-with-agent timed out after 180s waiting for an agent reply.
  • chat trace link (codex) failed with Agent did not reply within 180000ms.
  • notification net::ERR_ABORTED appears in both passing and failing tests, so it is noisy/browser-navigation-related rather than the root cause by itself.
• This aligns with an upstream agent LLM call failure: tests that only need chat/gateway mostly pass; tests requiring agent LLM output fail or cascade.

Root cause

• llm-proxy v0.12.2 changed auth precedence to prefer Authorization: Bearer over Ziti identity, fixing one path, but llm-proxy still only accepted platform API tokens from Authorization.
• The documented contract says llm-proxy accepts API tokens from both Authorization: Bearer agyn_... and x-api-key: agyn_....
• Agyn agent CLI paths use API-key style headers/config (OPENAI_API_KEY for Codex and configured API key for agn). Over llm-proxy.ziti, those requests include a Ziti identity too.
• Since llm-proxy ignored x-api-key, it fell through to the Ziti agent identity, then checked model authorization as the agent instead of the injected user API-token identity. That prevents the agent LLM request from succeeding, so no reply is posted.

Fix PR

Opened one intent-scoped PR in llm-proxy:

• fix(auth): accept x-api-key tokens llm-proxy#59

The PR:

• accepts caller API tokens from x-api-key as well as Authorization: Bearer,
• preserves precedence: Authorization → x-api-key → Ziti,
• adds unit coverage and an in-cluster E2E auth test for x-api-key.

Local validation

• go test -count=1 ./... — package tests passed locally; 6 packages had no test files.
• go test -c -tags e2e ./test/e2e — E2E package compile check passed; not executed outside cluster.
• go vet ./... && go vet -tags e2e ./test/e2e — lint/vet passed with no errors.

Next steps

1. Merge fix(auth): accept x-api-key tokens llm-proxy#59.
2. Release llm-proxy 0.12.3.
3. Bump agynio/bootstrap llm_proxy_chart_version to 0.12.3 in a separate PR.
4. Rerun chat-app E2E (svc_chat_app) against the bumped bootstrap image to confirm agent replies, trace links, and inline media tests recover.

[casey-brooks]

Retry investigation update

I retried the requested hands-on local reproduction using latest agynio/bootstrap@main.

Local cluster retry

Commands tried:

nix profile install nixpkgs#kubectl nixpkgs#k3d
cd /workspace/bootstrap
./apply.sh -y

The bootstrap retry still cannot create the local k3d cluster in this environment. Exact error:

Error: failed to ensure tools node: failed to run k3d-tools node for cluster 'agyn-local': failed to start node 'k3d-agyn-local-tools': runtime failed to start node 'k3d-agyn-local-tools': docker failed to start container for node 'k3d-agyn-local-tools': Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "sysfs" to rootfs at "/sys": mount src=sysfs, dst=/sys, dstFd=/proc/thread-self/fd/8, flags=0xe: operation not permitted: unknown

Environment evidence:

Docker: rootless mode, cgroups disabled
kubectl before install: command not found
k3d before install: command not found

After installing kubectl/k3d, the failure moved to the Docker/k3d sysfs mount. No Kubernetes cluster or kubeconfig was created, so I could not run Playwright locally or collect fresh local pod logs.

CI cluster reproduction evidence

I used the available CI cluster artifacts as the closest hands-on reproduction evidence.

From the failing chat-app run:

• Bootstrap deployed llm-proxy 0.12.2:
  • target_revision = "0.12.2"
  • image tag 0.12.2
• Relevant pods were healthy at test-failure time:
  • llm-proxy-llm-proxy-c7fcd5fd-pkqqc 1/1 Running
  • agents-orchestrator-5f7f98648d-7dzc8 1/1 Running
  • gateway-gateway-7cd54c6587-tph2g 1/1 Running
  • notifications-d49846dd5-5wph2 1/1 Running
  • chat-app-7647876c6c-nn7rc 1/1 Running
• Playwright symptoms:
  • chat-agent-response timed out waiting for an agent reply.
  • chat-with-agent timed out waiting for an agent reply.
  • chat trace link (codex) failed with Agent did not reply within 180000ms.
  • NotificationsGateway/Subscribe — net::ERR_ABORTED appears in passing tests too, so I do not think it is the primary failure.

I also used the llm-proxy PR E2E run, which provisions bootstrap in CI and reproduces the agent-reply path:

• TestLLMProxyGatewayCreatedModel passed, showing direct user API-token access to llm-proxy works.
• TestFullPipelineAgnMessageResponse failed waiting for an agent reply.
• Agent workload log shows the concrete failure:

agn: llm request endpoint=http://llm-proxy-llm-proxy.platform.svc.cluster.local:8080/v1 model=b3db0992-94e5-4959-bcd5-7c7cbeb450d3 input_items=1 tools=1 stream=false
agn: llm error status=403 err=call llm: POST "http://llm-proxy-llm-proxy.platform.svc.cluster.local:8080/v1/responses": 403 Forbidden
initial sync messages failed: agn_turn failed: run agn turn ... call llm: POST "http://llm-proxy-llm-proxy.platform.svc.cluster.local:8080/v1/responses": 403 Forbidden ; retrying in 1s

Relevant llm-proxy pod log from that same CI cluster:

proxy: request body read method=POST path=/v1/responses size=536
proxy: parsed request model_id=b3db0992-94e5-4959-bcd5-7c7cbeb450d3 stream=false
proxy: resolved model remote_name=simple-hello endpoint=https://testllm.dev/v1/org/agynio/suite/agn/responses
proxy: error status=403 err=access denied

Chat-app and gateway logs from artifacts are mostly probe/static request logs; the actionable service logs are the llm-proxy 403 and agent workload retry/error logs above. The failing chat-app artifact did not include per-service logs beyond pod status/events.

Root cause

The original hypothesis is correct: the failures are related to the llm-proxy 0.12.2 auth behavior change, but the reproduction shows two linked authorization issues:

1. llm-proxy accepted API tokens only from Authorization: Bearer, not x-api-key, while agent CLI paths use API-key style config/header paths. I opened a PR for this part:

   • fix(auth): accept x-api-key tokens llm-proxy#59

2. The reproduced 403 shows a second authorization-contract problem: agent-originated llm-proxy requests are authorized as the resolved agent identity, but llm-proxy checks identity:<id> can_use model:<modelID>. The authorization model grants model.can_use via organization membership (member from org), and agents are not organization members. That leaves agent LLM calls denied even when the model and user token are valid.

Proposed fix / repos

• agynio/llm-proxy: PR fix(e2e): align chat agent replies #59 fixes API token intake from x-api-key and adds unit/E2E coverage.
• Follow-up intent likely needed in either:
  • agynio/llm-proxy, if the proxy should authorize agent-originated calls using the API-token user identity rather than the Ziti agent identity, or
  • agynio/authorization / service contract, if agents should be granted a first-class relation that permits model use.

I recommend deciding the intended contract, then opening one follow-up PR for that separate authorization intent. I did not combine it into PR #59 because the requested policy change is separate from accepting x-api-key tokens.

Validation for PR #59

• go test -count=1 ./... — passed locally; 6 packages had no test files.
• go test -c -tags e2e ./test/e2e — E2E package compile check passed.
• go vet ./... && go vet -tags e2e ./test/e2e — lint/vet passed with no errors.

[rowan-stein]

Update: re-ran chat-app main CI after bootstrap agents bump merge.

• Run: https://github.com/agynio/chat-app/actions/runs/26166948038
• Result: FAILED

Top-level symptoms remain:

• Agent reply never arrives: Agent did not reply within 180000ms / test timeouts.
• Many net::ERR_ABORTED requests, notably NotificationsGateway/Subscribe.

No explicit 403 Forbidden / access denied / llm-proxy lines appear in the workflow log output for this run.
Artifacts are available for deeper inspection:

• https://github.com/agynio/chat-app/actions/runs/26166948038/artifacts/7112754303

Next: need to inspect the uploaded diagnostics/logs to see whether llm-proxy is still returning 403 to agent workloads and whether the new agents chart actually rolled out in the ephemeral cluster.

[casey-brooks]

Privileged Docker retry update

I retried latest-bootstrap local cluster setup after the Docker privilege change.

Docker/k3d status

Docker is no longer rootless. Relevant docker info lines:

Client: Docker Engine - Community
 Version:    29.4.3
 Context:    default
 Debug Mode: false

Server:
 Server Version: 27.5.1
 Storage Driver: overlay2
  userxattr: false
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Docker Root Dir: /var/lib/docker

k3d cluster creation now works:

k3d_cluster.this: Creation complete after 39s [id=agyn-local]
kubeconfig_path = "/workspace/bootstrap/stacks/k8s/.kube/agyn-local-kubeconfig.yaml"

Latest bootstrap progress / blockers

I ran latest agynio/bootstrap@main (eb6a1ac, fix(platform): bump agents chart (#511)) with:

cd /workspace/bootstrap
./apply.sh -y

Bootstrap progressed through k8s/system/routing/deps and into data. Several registry pulls hit net/http: TLS handshake timeout; I manually pulled/imported the needed images into k3d as each one blocked startup:

• quay.io/argoproj/argocd:v3.3.1
• quay.io/jetstack/cert-manager-controller:v1.20.0
• quay.io/jetstack/cert-manager-webhook:v1.20.0
• quay.io/jetstack/cert-manager-cainjector:v1.20.0
• quay.io/jetstack/trust-pkg-debian-bookworm:20230311-deb12u1.5
• quay.io/jetstack/trust-manager:v0.22.0
• rancher/mirrored-library-busybox:1.37.0
• openziti/ziti-controller:2.0.0-pre10
• quay.io/minio/minio:RELEASE.2024-11-07T00-52-20Z
• postgres:16.6-alpine
• openfga/openfga:v1.12.0

Concrete bootstrap evidence:

cert-manager: Synced and Healthy
trust-manager: Synced and Healthy
ziti-controller: Synced and Healthy
minio: Synced and Healthy
openfga-db: Synced and Healthy
openfga: Synced and Healthy

Pods reached this point:

ziti-controller-6878d4dbd9-rtggc 1/1 Running
minio-75b5b4fc8-q2qvv            1/1 Running
openfga-db-0                     1/1 Running
openfga-6cdbd6458c-nv9km         1/1 Running

New bootstrap fix PR

While unblocking data, I found the OpenFGA chart defaulted the main container to imagePullPolicy: Always, so even after the init container successfully used the imported image, the app container tried to repull openfga/openfga:v1.12.0 and hit the same TLS timeout loop:

Failed to pull image "openfga/openfga:v1.12.0": ... net/http: TLS handshake timeout

I opened a small bootstrap PR for that separate setup issue:

• fix(data): avoid OpenFGA image repulls bootstrap#512

Validation for that PR:

terraform -chdir=stacks/data fmt -check
terraform -chdir=stacks/data validate
terraform -chdir=stacks/data apply -input=false -auto-approve

The data stack apply completed after this change, and OpenFGA became healthy.

Current blocker before chat-app Playwright

Full platform/chat-app E2E did not complete locally yet. The next blocker is Ziti provider authentication in the ziti stack after the ziti-controller PVC had to be recreated during image-pull recovery:

Error: Unexpected error: %s
  with provider["registry.terraform.io/netfoundry/ziti"],
  on providers.tf line 1, in provider "ziti":

{"error":{"code":"INVALID_AUTH","message":"The authentication request failed","requestId":"..."},"meta":{"apiEnrollmentVersion":"0.0.1","apiVersion":"0.0.1"}}

I verified the secret currently contains:

admin-user=admin
admin-password=3Iz8oj5fvypiSSSoZ5Tz6JQcMF9omu68

and direct API auth with that password fails:

POST https://ziti-mgmt.agyn.dev:2496/edge/management/v1/authenticate?method=password
{"error":{"code":"INVALID_AUTH","message":"The authentication request failed",...}}

Inside the controller pod, ziti edge list identities shows only the default admin identity:

ID        NAME           TYPE     ATTRIBUTES  AUTH-POLICY
XGOKVDBgi Default Admin  Default              Default

So the remaining local-cluster blocker is now Ziti admin credential drift/state after the PVC reset; platform stack cannot read ../ziti/state/terraform.tfstate until that stack applies.

Root cause status for chat-app failures

I still believe the chat-app failure root cause is in the llm-proxy/agent authorization path, based on the existing CI cluster repro evidence:

• agent workload calls llm-proxy /v1/responses
• llm-proxy resolves the model and upstream
• llm-proxy returns 403 access denied
• agent never posts a reply, causing Playwright Agent did not reply within 180000ms
• NotificationsGateway/Subscribe net::ERR_ABORTED appears noisy/cascading rather than primary

Existing fix PR remains:

• fix(auth): accept x-api-key tokens llm-proxy#59

It fixes the first confirmed contract bug: accepting API tokens via x-api-key as well as Authorization: Bearer.

Next actions:

1. Fix or reset the local Ziti admin credential state so the ziti stack can apply again.
2. Finish platform/apps bootstrap.
3. Run chat-app Playwright locally and collect fresh pod logs from llm-proxy, agents-orchestrator, gateway, notifications, and chat-app.
4. If local logs still show llm-proxy 403 access denied, implement the second policy/identity fix in the owning repo as a separate intent PR.