Skip to content

feat(kubernetes): add proxy-pod topology (in-pod process supervisor, out-of-pod proxy) #2885

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
russellb wants to merge 3 commits into
base: main
Choose a base branch
from
Open

feat(kubernetes): add proxy-pod topology (in-pod process supervisor, out-of-pod proxy) #2885

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
85d5200
feat(kubernetes): add proxy-pod supervisor topology
TaylorMutch Sep 2, 2026
6a06bbd
feat(kubernetes): proxy-pod topology with in-pod process supervisor
russellb Sep 2, 2026
15c16b4
fix(kubernetes): preserve proxy URL for exec sessions
jhjaggars Sep 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 48 additions & 6 deletions .agents/skills/helm-dev-environment/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,18 +79,28 @@ mise run helm:skaffold:run:sidecar
mise run helm:skaffold:run:sidecar-mtls
```

Both commands build the `gateway` and `supervisor` images and deploy the OpenShell Helm
**Supervisor proxy-pod topology** (build once and leave running):
```bash
mise run helm:skaffold:run:proxy-pod
```

All Skaffold commands build the `gateway` and `supervisor` images and deploy the OpenShell Helm
chart. The sidecar profile renders an `openshell-network-init` init container for
nftables setup and an `openshell-supervisor-network` runtime sidecar for proxying.
Binary-aware policy mode runs that sidecar as UID 0 with `SYS_PTRACE` and
`DAC_READ_SEARCH`; relaxed mode can run it as the configured proxy UID, which
must be at least `1000` and distinct from the workload UID. The
sidecar-mTLS profile reuses `ci/values-sidecar.yaml` and restores
`server.disableTls=false` inline for Skaffold. The `pkiInitJob` hook (a pre-install
Job that runs `openshell-gateway generate-certs`) generates mTLS secrets on first
install. The default Skaffold values export gateway and Kubernetes-driver traces to
the collector service installed by `helm:k3s:create`. Envoy Gateway opt-in; see the
Optional Add-ons section below.
`server.disableTls=false` inline for Skaffold. The proxy-pod profile renders
network supervision in a separate supervisor Deployment with one pod and relies
on Kubernetes NetworkPolicy enforcement so the agent pod can reach only its
paired supervisor plus DNS. The default local k3s/k3d cluster keeps k3s's
embedded NetworkPolicy controller enabled; if you replace the CNI, install a
policy-enforcing CNI before using proxy-pod. The `pkiInitJob` hook (a
pre-install Job that runs `openshell-gateway generate-certs`) generates mTLS
secrets on first install. The default Skaffold values export gateway and
Kubernetes-driver traces to the collector service installed by
`helm:k3s:create`. Envoy Gateway opt-in; see the Optional Add-ons section below.

The gateway Service uses ClusterIP. Access is via Envoy Gateway (port `8080`) or
the unified local forwarding task:
Expand Down Expand Up @@ -131,6 +141,31 @@ export only while it is reachable.
create the Secret named `openshell-ha-pg` with a `uri` key, then run
`mise run helm:skaffold:run` or `mise run helm:skaffold:dev`.

### Kubernetes e2e profiles

Run the default Kubernetes e2e environment:

```bash
mise run e2e:kubernetes
```

Run the sidecar topology e2e environment:

```bash
mise run e2e:kubernetes:sidecar
```

Run the proxy-pod topology e2e environment:

```bash
mise run e2e:kubernetes:proxy-pod
```

The proxy-pod e2e task applies `ci/values-proxy-pod.yaml` through
`OPENSHELL_E2E_KUBE_EXTRA_VALUES`. Use an existing cluster with NetworkPolicy
enforcement, or let the wrapper create the default local k3d/k3s cluster with
k3s's embedded NetworkPolicy controller enabled.

### TLS behaviour

`ci/values-skaffold.yaml` sets `server.disableTls: true`, so Skaffold-based deploys run
Expand Down Expand Up @@ -194,6 +229,12 @@ For a sidecar-profile deployment:
mise run helm:skaffold:delete:sidecar
```

For a proxy-pod-profile deployment:

```bash
mise run helm:skaffold:delete:proxy-pod
```

### Delete the cluster entirely

```bash
Expand Down Expand Up @@ -321,6 +362,7 @@ for dependencies still declared in `Chart.yaml`.
| `deploy/helm/openshell/ci/values-high-availability.yaml` | HA test overlay (`replicaCount: 2` with external PostgreSQL Secret) |
| `deploy/helm/openshell/ci/values-keycloak.yaml` | Keycloak OIDC overlay |
| `deploy/helm/openshell/ci/values-sidecar.yaml` | Supervisor sidecar topology overlay for Kubernetes e2e/dev |
| `deploy/helm/openshell/ci/values-proxy-pod.yaml` | Supervisor proxy-pod topology overlay for Kubernetes e2e/dev; requires NetworkPolicy enforcement |
| `deploy/helm/openshell/ci/values-spire.yaml` | SPIFFE/SPIRE provider token grant overlay |
| `deploy/helm/openshell/ci/values-spire-stack.yaml` | SPIRE hardened chart values for local dev |
| `deploy/helm/openshell/ci/values-tls-disabled.yaml` | Lint-only: TLS + auth disabled (reverse-proxy edge termination) |
Expand Down
20 changes: 19 additions & 1 deletion .github/workflows/branch-e2e.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -436,6 +436,24 @@ jobs:
e2e-task: e2e:kubernetes:workspace-operator
conformance-artifact-prefix: openshell-conformance

kubernetes-proxy-pod-e2e:
needs: [pr_metadata, build-gateway, build-supervisor, build-cli]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
permissions:
actions: read
contents: read
packages: read
uses: ./.github/workflows/e2e-kubernetes-test.yml
with:
image-tag: ${{ github.sha }}
# kind's default CNI does not enforce NetworkPolicies, so this exercises
# the proxy-pod control-plane contract (companions, readiness, sessionless
# relay rejection). The CNI-enforced egress isolation test is tracked
# separately and needs a policy-enforcing CNI.
job-name: Kubernetes E2E (proxy-pod topology)
e2e-task: e2e:kubernetes:proxy-pod
cli-artifact-prefix: rust-binary-cli

kubernetes-ha-e2e:
needs: [pr_metadata, build-cli, build-conformance, build-gateway-image, build-supervisor-image]
if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_kubernetes_ha_e2e == 'true'
Expand Down Expand Up @@ -467,7 +485,7 @@ jobs:

core-e2e-result:
name: Core E2E result
needs: [pr_metadata, docker-e2e, podman-e2e, vm-e2e, docker-external-driver-e2e, podman-external-driver-e2e, vm-external-driver-e2e, kubernetes-e2e, kubernetes-external-driver-e2e, kubernetes-workspace-managed-e2e, kubernetes-workspace-operator-e2e]
needs: [pr_metadata, docker-e2e, podman-e2e, vm-e2e, docker-external-driver-e2e, podman-external-driver-e2e, vm-external-driver-e2e, kubernetes-e2e, kubernetes-external-driver-e2e, kubernetes-workspace-managed-e2e, kubernetes-workspace-operator-e2e, kubernetes-proxy-pod-e2e]
if: always() && needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true'
runs-on: ubuntu-latest
steps:
Expand Down
4 changes: 4 additions & 0 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ http-body-util = "0.1"
tokio-rustls = { version = "0.26", default-features = false, features = ["logging", "tls12", "ring"] }
rustls = { version = "0.23", default-features = false, features = ["std", "logging", "tls12", "ring"] }
rustls-pemfile = "2"
rcgen = { version = "0.13", features = ["crypto", "pem"] }
rcgen = { version = "0.13", features = ["crypto", "pem", "x509-parser"] }
webpki-roots = "1"
rustls-native-certs = "0.8"

Expand Down
22 changes: 20 additions & 2 deletions architecture/gateway.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -240,10 +240,28 @@ to the selected compute driver's `AuthenticateSandbox` RPC. A capable driver is
trusted to return the authenticated sandbox ID, while the gateway still requires
a matching durable sandbox record before minting a JWT. The Kubernetes driver
uses its own named configuration to run TokenReview and verify the live pod and
controlling Sandbox CR. The bootstrap path accepts
controlling Sandbox CR: agent pods must be directly controlled by the `Sandbox`
CR, while proxy-pod supervisor pods may be controlled through the Kubernetes
`Pod -> ReplicaSet -> Deployment -> Sandbox` chain. The bootstrap path accepts
both `agents.x-k8s.io/v1beta1` ownerReferences from newer Agent Sandbox
controllers and `agents.x-k8s.io/v1alpha1` ownerReferences from existing
deployments. Supervisors renew gateway JWTs in memory before expiry only while
deployments. The proxy-pod gateway Role follows least privilege: the supervisor
Deployment, Service, CA Secret, and supervisor-ingress NetworkPolicy are
owner-referenced to the Sandbox CR and garbage-collected with it, so the gateway
holds no `delete` on them (Deployment create/get/patch, Service
create/get, Secret create only, plus get on the ReplicaSet for the owner-chain
check). In shared (single-namespace) mode the namespaced Role also grants
Deployment `list`/`watch`, backing a supervisor Deployment watch that pushes a
refreshed sandbox status within seconds of a supervisor availability change;
managed and operator modes deliberately omit those verbs to avoid cluster-wide
Deployment enumeration, folding readiness in through get/list instead. A periodic
reconcile (alongside the one at watch establishment) corrects supervisor replica
drift and reaps orphaned fences without waiting for the watch to drop. The agent
egress NetworkPolicy — the workload's egress fence — carries
no owner reference so it can outlive the workload pod during deletion; the gateway
manages its lifecycle directly and holds create/get/delete/list on NetworkPolicies
for ordered teardown and orphan reaping.
Supervisors renew gateway JWTs in memory before expiry only while
the sandbox record still exists. Older tokens are not server-revoked; shared
deployments bound replay exposure with short `gateway_jwt.ttl_secs` lifetimes.
The config default is
Expand Down
Loading
Loading