Skip to content

test(kubernetes): add optional Envoy-backed e2e mode - #2259

Open
elezar wants to merge 2 commits into
mainfrom
codex/kubernetes-envoy-e2e
Open

test(kubernetes): add optional Envoy-backed e2e mode#2259
elezar wants to merge 2 commits into
mainfrom
codex/kubernetes-envoy-e2e

Conversation

@elezar

@elezar elezar commented Jul 14, 2026

Copy link
Copy Markdown
Member

Summary

Add an optional Envoy-backed execution mode for the standard single-replica Kubernetes E2E suite. The mode exercises the Gateway API/Envoy GRPCRoute path without enabling HA, changing replica counts, or requiring PostgreSQL.

Direct Service port-forwarding remains the default. Envoy mode installs or reuses an Envoy Gateway controller, creates test-owned Gateway API resources, disables Envoy request and stream duration limits for OpenShell's long-lived gRPC streams, discovers the generated proxy Service, and routes the client port-forward through that Service.

Related Issue

Extracted from the generally applicable Envoy/Gateway API setup in #1868. This PR intentionally leaves HA gateway rebalancing, multi-replica behavior, and external PostgreSQL coverage out of scope while the HA tests are not yet functional.

Changes

  • Add mise run e2e:kubernetes:envoy for local single-replica Envoy execution.
  • Add OPENSHELL_E2E_KUBE_USE_ENVOY=1 support to the Kubernetes e2e wrapper.
  • Add a use-envoy-gateway input to the reusable Kubernetes E2E workflow.
  • Add one core Kubernetes E2E matrix leg with topology: envoy and use-envoy-gateway: true.
  • Reuse an existing Envoy controller without upgrading or uninstalling its release.
  • Install Envoy with helm install only when no controller is present, using a unique test-owned namespace.
  • Create uniquely named test-owned GatewayClass and BackendTrafficPolicy resources, refusing to modify pre-existing resources with those names.
  • Delete only Envoy resources whose creation was recorded by the current invocation.
  • Route the client port-forward through the generated Envoy proxy Service while keeping the default direct-Service path unchanged.

Testing

  • bash -n e2e/with-kube-gateway.sh
  • git diff --check
  • Workflow YAML parse through the OpenShell devenv with pinned uv/PyYAML
  • Helm lint with ci/values-gateway.yaml
  • Helm render of Gateway and GRPCRoute with the per-run GatewayClass override
  • Focused shell simulations for existing-controller reuse, pre-existing namespace/Class refusal, and created-resource-only cleanup
  • mise run pre-commit completes (lint, format, workspace/E2E Clippy, Helm, Python, and other reported checks passed; the first run hit a fresh-virtualenv grpc_tools bootstrap race, and the rerun reached the five-minute command timeout during full Rust tests)
  • Kubernetes E2E (Docker was unavailable locally)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Runtime mode and reusable workflow exposure are separate logical commits
  • Envoy coverage is single-replica and does not add HA replicas, PostgreSQL setup, or HA tests

@copy-pr-bot

copy-pr-bot Bot commented Jul 14, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@elezar
elezar force-pushed the codex/kubernetes-envoy-e2e branch from 0dbb332 to 8e8d0cc Compare July 27, 2026 13:26
@elezar
elezar marked this pull request as ready for review July 27, 2026 14:52
Comment on lines +40 to +44
use-envoy-gateway:
description: "Run the e2e command through an Envoy Gateway GRPCRoute"
required: false
type: boolean
default: false

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the motivation for this? Is it to make sure we can test HA deployments?

@elezar elezar Jul 28, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The primary motivation for this PR (and some of the others) was to reduce the surface of #1868 (adding HA support) somewhat. This includes adding Envoy configuration / deployments to provide a stable endpoint for the HA case.

As reviewed, the PR added the option but didn't actually exercise any of the added functionality. Since using Envoy is a separate from an HA deployment, I have updated the PR to add an "envoy-enabled" path to the existing tests. This should mean that we can shake out any issues with the envoy-gateway interaction separately from the HA functionality.

Depending on the load on CI, we can consider removing it again or simplifying our test matrix for k8s once HA support lands.

elezar added 2 commits July 28, 2026 10:50
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
@elezar
elezar force-pushed the codex/kubernetes-envoy-e2e branch from 8e8d0cc to 9b5c53e Compare July 28, 2026 08:51
@elezar

elezar commented Jul 28, 2026

Copy link
Copy Markdown
Member Author

/ok-to-test 9b5c53e

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants