Skip to content

Leash fails on Docker Desktop Kubernetes: cgroup-based iptables isolation not supported #64

Closed

Description

@sunkadshreyas

Summary

Leash sidecar fails to start in quest pods on Docker Desktop Kubernetes (macOS, Apple Silicon). The cgroup-based control plane isolation cannot be applied because Docker Desktop's container runtime doesn't expose cgroup paths in the format leash expects for iptables -m cgroup --path matching.

Environment

  • Host: macOS (Darwin 25.3.0, Apple Silicon)
  • Kubernetes: Docker Desktop with Kubernetes enabled
  • Leash image: public.ecr.aws/s5i7k8t3/strongdm/leash:latest
  • Coder image: 989005111931.dkr.ecr.us-west-1.amazonaws.com/ng-coder:latest (linux/amd64 via Rosetta)

Error

no pod cgroup found (pod UID=a0eb5456-0384-431b-add8-ea3536ca7bfa)
2026/02/26 18:26:26 Warning: adjusted LEASH_PRIVATE_DIR permissions from 777 to 0700
2026/02/26 18:26:26 leashd_mounts public=/leash private=/leash-private
Loaded 1 file open policy rules
Default open policy result: ALLOW (root path '/' is allowed)
Loaded 1 exec policy rules
Default exec policy result: ALLOW (root path '/' is allowed)
2026/02/26 18:26:26 event=ca.generate public_dir=/leash private_dir=/leash-private
Loaded 3 connect IP rules (skipped 0 wildcard, 1 unresolved)
Default connect policy result: DENY (configured override)
2026/02/26 18:26:26 event=policy.restore source=file lsm_open=1 lsm_exec=1 lsm_connect=4 http_rewrites=0
2026/02/26 18:26:26 event=bootstrap.wait path=/leash/bootstrap.ready timeout=2m0s
2026/02/26 18:26:26 event=bootstrap.ready path=/leash/bootstrap.ready source=coder-container cgroup=/ mtime=2026-02-26T18:26:26.140889007Z
leash: applying network interception rules
2026/02/26 18:26:26 event=frontend.start addr=:18080
iptables v1.8.9 (nf_tables):  RULE_APPEND failed (Invalid argument): rule in chain OUTPUT
leash: FATAL: could not apply cgroup-based control plane isolation
leash: This security control is required to prevent target container from accessing leashd API
2026/02/26 18:26:26 exit status 1

Root Cause

  1. The discover-cgroup init container cannot find the pod's cgroup path under /sys/fs/cgroup on Docker Desktop — it writes empty strings to /leash/cgroup-path and /leash/cgroup-hier.

  2. The coder container reports its cgroup as / (from /proc/1/cgroup), which is not a valid cgroup2 hierarchy path for iptables matching.

  3. Leash then fails when trying to apply iptables -m cgroup --path / (or similar) — RULE_APPEND failed (Invalid argument).

Expected Behavior

Leash should either:

  • Gracefully degrade when cgroup-based isolation is unavailable (e.g., skip iptables isolation with a warning), or
  • Support an alternative isolation mechanism for environments where cgroup paths aren't discoverable (Docker Desktop, kind, minikube, etc.)

Reproduction

  1. Set up ai-agent-local on Docker Desktop Kubernetes
  2. Send a message that triggers a quest with SDM access
  3. Observe the quest pod's leash container crash

Workaround

For local development, removing the leash sidecar from the pod spec in ai-agent-brain/internal/k8s/pod.go allows quest pods to run (without network isolation).

Activity

  1. sunkadshreyas commented on Feb 26, 2026

    @sunkadshreyas
    Author

    Root Cause Analysis (Updated)

    After deeper investigation, there are three distinct bugs causing leash to fail on Docker Desktop Kubernetes:

    Bug 1: nftables comment quoting (affects ALL environments)

    The ensure_rule function in apply-nftables.sh passes comment strings like leash:return-mitm without nft-level quoting. The : character is a special token in nft's parser, causing all nftables rules to fail — not just the cgroup rule.

    Error: syntax error, unexpected colon
    add rule ip test1 out tcp dport 18000 return comment leash:return-mitm
                                                              ^
    

    Fix: comment "$comment" → comment "\"$comment\"" in ensure_rule().

    Bug 2: nftables cgroup path quoting

    The socket cgroupv2 level 1 $TARGET_CGROUP rule passes the cgroup path without nft-level quotes. Paths starting with / cause a syntax error.

    Fix: "$TARGET_CGROUP" → "\"$TARGET_CGROUP\"" in the cgroup rule.

    Bug 3: Docker Desktop kernel lacks cgroup matching support

    Docker Desktop's LinuxKit kernel (6.10.14-linuxkit) does not support:

    • iptables -m cgroup --path (xt_cgroup module)
    • nftables socket cgroupv2 level 1 (nft socket match)

    Both fail with "No such file or directory" / "Invalid argument". The kernel simply doesn't have these features compiled in.

    Fix: Add LEASH_CGROUP_ISOLATION env var (required by default, optional for Docker Desktop) so leash can start without cgroup-based control plane isolation where the kernel doesn't support it.

    Additional finding: nftables not installed in runtime image

    The Dockerfile.leash runtime-base stage installs iptables but not nftables. Since Debian bookworm's iptables uses the nf_tables backend, the nft binary is needed for the preferred nftables code path.

    Fix: Add nftables package to runtime-base in Dockerfile.leash.

    Fix

    PR incoming.

  2. added a commit that references this issue on Feb 26, 2026
    11e5d04
  3. navanchauhan commented on Mar 4, 2026

    @navanchauhan
    Collaborator

    Fixed by #66 — leash now falls back to a blanket port block when cgroup-based iptables matching is unavailable (same root cause as #60: Docker Desktop LinuxKit kernels lack xt_cgroup support).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions