fix(e2e): Skill Discovery test fails due to insufficient webhook readiness check after controller restart

[pdettori]

Summary

The "Skill Discovery E2E > Feature gate enabled > should populate linkedSkills from annotation" test fails consistently in CI since PR #396 was merged (June 8, 2026). This blocks all PRs — the E2E Tests job is the only failing check across all open PRs.

Affected: Every CI run on every branch since June 8.
Impact: 1 test fails, 5 skip (cascading), 35 pass.

Root Cause

Insufficient webhook readiness verification after controller restart.

The test checks that the webhook Endpoints object has an IP address, but does NOT verify the webhook's TLS server is actually accepting connections. There's a race window (~5-30s) where the endpoint IP is populated but the webhook server hasn't completed TLS setup.

Failure Chain

1. EnableSkillDiscovery() patches the controller deployment with a volume mount → triggers pod restart
2. WaitForRollout succeeds (deployment reports Available)
3. Endpoint IP check passes immediately (pod IP is in Endpoints object)
4. Test creates AgentRuntime → controller reconciles → adds kagenti.io/type label + config-hash to PodTemplateSpec → triggers deployment rollout
5. ReplicaSet tries to create new pods → Kubernetes calls the inject.kagenti.io mutating webhook
6. Webhook TLS server is not yet ready → dial tcp 10.96.50.216:443: connect: connection refused
7. Pods never get created → deployment never rolls → linkedSkills never populates → test times out at 180s

Why It Started Failing with PR #396

PR #396 changed the "Combined" test (runs immediately before "Skill Discovery") to trigger a rolling update on AR deletion (verifying pods have no sidecars). This increases webhook activity right before the Skill Discovery block tears down and redeploys the controller, making the timing race more likely to manifest.

Evidence

From CI run 27247825117 (main branch, June 10):

Line 506: rollout status completes at 02:03:22
Line 510: endpoint IP check passes at 02:03:22 (same second!)
Line 512: AgentRuntime created at 02:03:22
Line 719: FailedCreate event: "failed calling webhook inject.kagenti.io: 
           dial tcp 10.96.50.216:443: connect: connection refused"
           (persists for 4+ minutes)

The endpoint IP exists but the TLS server isn't listening yet.

Proposed Fix

Location

• kagenti-operator/test/e2e/e2e_test.go — lines 2128-2136 (the "waiting for webhook endpoint to be ready after restart" block in the "Feature gate enabled" Context's BeforeAll)
• kagenti-operator/test/utils/utils.go — add a reusable WaitForWebhookReady() helper

Approach

Replace the endpoint-IP-only check with an active TLS connectivity probe. The webhook exposes a health endpoint at /healthz on port 9443 inside the pod. The check should:

1. Keep the existing endpoint IP check (fast pre-condition)
2. Add a follow-up that actually connects to the webhook service and verifies TLS handshake succeeds

Implementation

In test/utils/utils.go, add a WaitForWebhookReady function:

// WaitForWebhookReady waits until the webhook service is actually accepting
// TLS connections, not just until the Endpoints object has an IP.
func WaitForWebhookReady(namespace string, timeout time.Duration) error {
    // Phase 1: endpoint IP exists (fast gate)
    err := wait.PollUntilContextTimeout(context.Background(), 2*time.Second, timeout, true,
        func(ctx context.Context) (bool, error) {
            cmd := exec.Command("kubectl", "get", "endpoints",
                "kagenti-operator-webhook-service", "-n", namespace,
                "-o", "jsonpath={.subsets[0].addresses[0].ip}")
            output, err := Run(cmd)
            if err != nil || output == "" {
                return false, nil
            }
            return true, nil
        })
    if err != nil {
        return fmt.Errorf("webhook endpoint IP not ready: %w", err)
    }

    // Phase 2: TLS port is accepting connections
    return wait.PollUntilContextTimeout(context.Background(), 2*time.Second, timeout, true,
        func(ctx context.Context) (bool, error) {
            // Use kubectl to port-forward and check, or use a Job/exec approach.
            // Simplest: create a temporary pod that curls the webhook, or use
            // kubectl run with --rm to test connectivity.
            cmd := exec.Command("kubectl", "run", "webhook-probe", "--rm", "-i",
                "--restart=Never", "--image=curlimages/curl:latest",
                "-n", namespace, "--",
                "curl", "-sk", "-o", "/dev/null", "-w", "%{http_code}",
                "https://kagenti-operator-webhook-service."+namespace+".svc:443/healthz")
            output, err := Run(cmd)
            if err != nil {
                return false, nil
            }
            // Any HTTP response (even 404) means TLS is up
            return output != "" && output != "000", nil
        })
}

Alternative simpler approach (no extra pod needed): Use a raw TCP check via a shell command that attempts a TLS connection within the cluster network. Since the e2e tests already have the controller pod running, we can exec into it:

// WaitForWebhookReady verifies the webhook TLS server is accepting connections.
func WaitForWebhookReady(namespace, deployment string, timeout time.Duration) error {
    By("waiting for webhook endpoint IP")
    if err := waitForEndpointIP(namespace, timeout); err != nil {
        return err
    }

    By("waiting for webhook TLS server to accept connections")
    return Eventually(func(g Gomega) {
        // The manager process serves the webhook on :9443 inside the pod.
        // A successful wget --spider proves TLS is up.
        cmd := exec.Command("kubectl", "exec",
            "deploy/"+deployment, "-n", namespace, "--",
            "wget", "--spider", "--no-check-certificate", "-q", "-T", "2",
            "https://localhost:9443/healthz")
        _, err := Run(cmd)
        g.Expect(err).NotTo(HaveOccurred())
    }, timeout, 2*time.Second).Should(Succeed())
}

Note: The manager container image is distroless and may not have wget/curl. If so, use a Go-based TCP dial from a temporary pod, or simply add a short sleep (10s) after the endpoint check as a pragmatic fix. The most robust approach is to create a small utility binary or use the existing curl image approach.

Pragmatic minimal fix (if the above is too complex): Simply increase the wait and add a fixed delay:

By("waiting for webhook endpoint to be ready after restart")
Eventually(func(g Gomega) {
    cmd := exec.Command("kubectl", "get", "endpoints",
        "kagenti-operator-webhook-service", "-n", controllerNamespace,
        "-o", "jsonpath={.subsets[0].addresses[0].ip}")
    output, err := utils.Run(cmd)
    g.Expect(err).NotTo(HaveOccurred())
    g.Expect(output).NotTo(BeEmpty(), "webhook endpoint not yet populated")
}, 2*time.Minute, 2*time.Second).Should(Succeed())

// Allow webhook TLS server time to start accepting connections
// after the pod is Running and endpoint is populated.
time.Sleep(10 * time.Second)

Where to Apply

This fix needs to be applied at every location where "waiting for webhook endpoint to be ready" appears after a controller redeploy or restart. In the current e2e_test.go these are:

• Line 194 (initial BeforeSuite)
• Line 337 (AuthBridge)
• Line 689 (AgentCard)
• Line 1032 (AgentRuntime)
• Line 1506 (Combined)
• Line 1967 (Skill Discovery BeforeAll)
• Line 2128 (Skill Discovery Feature gate enabled — the failing one)
• Line 2431 (Istio mesh)

The most impactful fix is at line 2128 since that's where the test fails, but applying it to all locations prevents future regressions.

Verification

After the fix, re-run the CI workflow. Expected result: 36/36 tests pass (or more if new tests were added), with the linkedSkills test no longer timing out.

Additional Context

• The inject.kagenti.io webhook is a mutating admission webhook that intercepts pod CREATE operations in namespaces with kagenti.io/type-labeled workloads.
• cert-manager provides the TLS certificate for the webhook. After a pod restart, the new pod must read the certificate from a Secret mounted as a volume. This read can take a few seconds.
• The Istio mesh test (line 2469) already uses a retry pattern for AgentRuntime creation (3*time.Minute, 5*time.Second) — but that alone won't fix this issue because the AgentRuntime creation succeeds; it's the subsequent pod creation by the ReplicaSet that fails.
• The controller's webhook server starts asynchronously after the main reconciliation loop. On Kind clusters in CI (resource-constrained), this startup can be slower.

Labels

• bug
• e2e
• priority/high