feat(e2e): make e2e:kubernetes work transparently on OpenShift

[jgarciao]

User Story

As a contributor running the Kubernetes e2e test suite on an OpenShift cluster,
I want mise run e2e:kubernetes to detect OpenShift automatically and handle
Security Context Constraints (SCC) setup, gateway transport, and teardown,
so that I can run the full Rust e2e suite — including the connect-based
suites — on OpenShift without a separate task or manual steps.

Problem Statement

Running mise run e2e:kubernetes on OpenShift requires manual preparation, and
even with that preparation the connect-based suites do not pass:

1. SCC setup: the contributor must pre-create the namespace, grant the
   privileged SCC to the openshell-sandbox service account, and override the
   chart's hardcoded runAsUser: 1000 / fsGroup: 1000, which OpenShift's
   restricted-v2 SCC rejects because it requires UIDs from the
   namespace-assigned range.
2. Transport: the harness reaches the gateway via kubectl port-forward. On
   OpenShift this stalls the SSH-relay sandbox connect path — that command
   opens an SSH session needing many small round trips, each tunneled through
   port-forward, so the connect suites (live_policy_update, port_forward,
   sync, connect-based sandbox_lifecycle, settings_management) run to the
   ready timeout instead of passing.
3. Image freshness: on an existing cluster the harness pulls the published
   gateway/supervisor image (default latest), and nodes may serve a stale
   cached copy, so a run does not actually exercise the intended image.

A separate e2e:openshift task exists but only verifies the gateway pod
starts — it runs no Rust e2e coverage.

Impact / Why This Matters

Without this, contributors validating OpenShell on OpenShift must manually
create the namespace, grant/clean up SCCs, and pass Helm overrides — error-prone
and easy to forget, especially cleanup. Worse, even after all that, the
connect-based suites time out over port-forward, so the most important
sandbox-connectivity coverage silently fails and OpenShift appears broken when
it is not. The existing e2e:openshift task creates a false impression of
coverage. The harness should make the same command that works on Kind/k3d work
end-to-end on OpenShift.

Proposed Design

mise run e2e:kubernetes (and every variant like e2e:kubernetes:db) should
behave identically on vanilla Kubernetes and OpenShift:

1. Auto-detection: detect OpenShift by the presence of the
   route.openshift.io API group. No flags or env vars.
2. SCC handling (OpenShift only): apply an SCC-compatible Helm overlay that
   removes hardcoded runAsUser/fsGroup; grant privileged to
   openshell-sandbox; for DB scenarios grant anyuid to the PostgreSQL
   fixture service account.
3. Gateway transport (OpenShift only): reach the gateway through a
   passthrough OpenShift Route secured with mandatory mTLS instead of
   port-forward, so the connect suites pass. The harness derives the Route host
   from the cluster ingress domain (<release>-<namespace>.<apps-domain>), bakes
   it into the server cert SANs before the Route exists, extracts client mTLS
   material from the openshell-client-tls secret, waits for the Route to serve
   mTLS, asserts a certless caller is rejected at the TLS handshake, and
   registers an mTLS CLI gateway. The health endpoint continues to use
   port-forward (not on the SSH path).
4. Image pull policy (OpenShift overlay): force image.pullPolicy /
   supervisor.image.pullPolicy to Always so a run against latest uses the
   fresh image rather than a stale node-cached copy.
5. Existing/remote-cluster ergonomics (documentation): document that on an
   existing cluster the CLI is built from the branch while the image is the
   published one, so IMAGE_TAG should be pinned when versions diverge; and that
   e2e-host-gateway must be dropped on remote clusters because
   host.openshell.internal is unreachable from remote pods.
6. Cleanup: remove SCC bindings and extracted client-key material in the
   cleanup function, even on failure/interrupt.
7. Remove e2e:openshift: delete the task and e2e-openshift.sh;
   e2e:kubernetes now covers OpenShift.
8. No changes on vanilla Kubernetes: every OpenShift branch is gated on
   OPENSHIFT_DETECTED; the port-forward/plaintext path is unchanged.

Acceptance Criteria

• mise run e2e:kubernetes auto-detects OpenShift and prints "OpenShift detected" on an OpenShift cluster
• It does not falsely detect OpenShift on Kind/k3d/vanilla clusters
• The chart deploys without runAsUser/fsGroup on OpenShift (SCC overlay applied automatically)
• privileged SCC is granted to openshell-sandbox before Helm install and removed during cleanup
• anyuid SCC is granted to the PostgreSQL fixture service account in DB scenarios and removed during cleanup
• On OpenShift the gateway is reached over a passthrough Route with mandatory mTLS; a certless caller is rejected at the TLS handshake and the run fails loudly if it is not
• The connect-based suites (live_policy_update, port_forward, sync, connect-based sandbox_lifecycle, settings_management) pass on OpenShift
• The OpenShift overlay forces pullPolicy: Always for gateway and supervisor images
• Extracted client mTLS key material is removed during cleanup
• The e2e:openshift task and e2e-openshift.sh script are removed
• TESTING.md documents Kubernetes e2e tasks, OpenShift auto-detection, IMAGE_TAG pinning for CLI/image version skew, dropping e2e-host-gateway on remote clusters, and relevant environment variables
• All oc commands use --context "${KUBE_CONTEXT}" to target the correct cluster

Alternatives Considered

1. Keep e2e:openshift as a separate task: This is the status quo. It
   requires maintaining two scripts with diverging logic. The separate script
   only checks pod readiness and provides no actual test coverage. Rejected
   because it creates a false sense of OpenShift coverage.

2. Require explicit --openshift flag or environment variable: This would
   work but adds friction and is easy to forget. Auto-detection via the
   route.openshift.io API group is reliable and requires no user action.
   The API group check uses --no-headers output piped to grep -q . to
   avoid false positives from successful-but-empty kubectl api-resources
   responses.

3. Separate Helm values file passed manually: Contributors could pass
   OPENSHELL_E2E_KUBE_EXTRA_VALUES pointing to an SCC overlay. This works
   but pushes OpenShift knowledge onto each contributor and doesn't handle
   SCC grant/cleanup. Rejected in favor of full automation.

4. Keep port-forward on OpenShift: rejected — the SSH-relay connect suites
   stall to the ready timeout over port-forward's tunneled round trips, so the
   most valuable coverage never passes. A real network path (passthrough Route)
   is required.

5. Server-only TLS on the Route (no mTLS): rejected — that would expose the
   gateway on a routable hostname with no access gate. Mandatory mTLS (the client
   cert as the access gate, require_client_auth implied by clientCaSecretName

   • no OIDC) keeps the Route closed to certless callers.

Agent Investigation

The implementation was developed and tested on both Kind and OpenShift clusters.
Key findings:

• PR history: e2e-openshift.sh was created in PR feat(helm): add optional PostgreSQL backing store #1579 as a shortcut
  during PostgreSQL backing store work. The author deferred OpenShift integration
  into the main harness. PRs refactor(helm): require external postgres for ha #1844, fix(e2e): make postgres fixture compatible with OpenShift #2002, and chore(ci): disable telemetry in internal test runs #2648 iterated on the OpenShift
  e2e path without merging it into the main harness. No CI workflow references
  e2e:openshift.

• Helm template fix needed: The podSecurityContext block in
  _gateway-workload.tpl renders invalid YAML when values are null. Wrapping
  with {{- with }} makes it conditional. This is a production chart bug
  independent of the e2e harness and is being tracked separately in fix(helm): omit podSecurityContext block when value is null #3033.

• OpenShift detection pitfall: kubectl api-resources --api-group=route.openshift.io
  returns exit code 0 even on vanilla Kubernetes (empty result set). Detection
  must check for actual output, not just exit code.

• oc context targeting: oc adm policy commands default to oc's own
  current context, which may differ from KUBE_CONTEXT. All oc commands must
  use --context "${KUBE_CONTEXT}".

• Port-forward vs SSH relay: kubectl port-forward carries SSH's many small
  setup frames poorly, so the connect suites time out. The passthrough Route +
  mTLS path makes them pass.

• Deterministic Route host: OpenShift serves any name under the cluster
  ingress (apps) domain via the router wildcard, so
  <release>-<namespace>.<apps-domain> can be baked into the server cert SANs
  before the Route is created.

• Stale node image cache: latest can be served stale from a node's cache;
  pullPolicy: Always is required for a meaningful run.

• CLI/image version skew and remote clusters: on an existing cluster the CLI
  is built from the branch while the gateway/supervisor image is the published
  one, so IMAGE_TAG must be pinned when versions diverge; and e2e-host-gateway
  must be dropped on remote clusters because host.openshell.internal is
  unreachable from remote pods. Both are documented in TESTING.md.

• Test results: Both Kind and OpenShift test runs were completed. On Kind,
  all tests pass or fail with pre-existing issues (host.openshell.internal
  unreachable, Docker is Podman shim). On OpenShift, the same pre-existing
  failures occur. Tests that don't depend on sandbox SSH readiness (smoke,
  bypass_detection, community_image, landlock, no_proxy, etc.) pass normally.

Related issues:

• feat: Support restricted SecurityContextConstraints for managed Kubernetes platforms #899 (closed) — Delivered the sidecar topology for production OpenShift deploys. This feature request is about e2e test infrastructure, not the sandbox architecture — no overlap.
• feat(openshift): supported install path without the privileged SCC or disabled TLS #2812 (closed) — About eliminating privileged from the production install path; PR feat(kubernetes): add proxy-pod topology (in-pod process supervisor, out-of-pod proxy) #2885 is in progress. This feature request grants privileged during e2e test runs only — e2e tests need full capabilities to exercise the sandbox. No conflict.
• feat(helm): expose sandbox_uid / sandbox_gid as first-class Helm values #2697 (accepted) — Tangentially related to OpenShift UID assignment; this feature request doesn't touch it.
• fix(e2e): make postgres fixture compatible with OpenShift #2002 (merged) — Prior iteration on OpenShift e2e compatibility, specifically for the PostgreSQL fixture.
• fix(helm): omit podSecurityContext block when value is null #3033 — Chart bug spun off from this work: podSecurityContext: null renders invalid YAML in _gateway-workload.tpl (affects production installs, not just e2e).
• docs/kubernetes/openshift.mdx already documents the manual SCC commands and Helm overrides this feature request automates for e2e runs.

---

I have an implementation ready here and can open a PR once this is accepted.