Fix: reconciler conflict error prevents linkedSkills status update

[pdettori]

Summary

The Skill Discovery E2E test intermittently fails because the controller's status update for linkedSkills hits a Kubernetes optimistic concurrency conflict and does not retry. This causes the linkedSkills field to never be populated, and the test times out after 180s.

Affected: main branch — all recent CI runs fail on this test.
Impact: 1 test fails, 5 skip (cascading from same Describe block).

Root Cause

During reconciliation of an AgentRuntime with skill annotations, the controller attempts to update .status.linkedSkills. If another reconciliation loop modifies the same AgentRuntime object concurrently (e.g., adding labels, updating config-hash), the status update fails with:

Operation cannot be fulfilled on agentruntimes.agent.kagenti.dev "skill-discovery-agent":
the object has been modified; please apply your changes to the latest version and try again

The controller does not retry the status update after this conflict error. The reconciler returns the error, gets requeued, but the next reconciliation may take a different code path (object already has labels/hash), skipping the linkedSkills update.

Evidence

From CI run 27253451563 (branch fix/webhook-readiness-420, June 10 2026):

• Test polls for linkedSkills from 04:57:42 to 05:01:20 (4+ minutes)
• Controller logs show conflict error at 04:58:20Z
• The linkedSkills field is never populated after the conflict

Same failure on main branch: runs 27249800636, 27249348694, 27249047484 all fail.

Relationship to #420

Issue #420 identified two problems causing the Skill Discovery test failure:

1. Webhook readiness race — fixed by PR Fix: add TLS connectivity probe to webhook readiness checks #421 (pod readiness check after restart)
2. Reconciler conflict — this issue (status update not retried on conflict)

PR #421 resolves problem 1. This issue tracks problem 2.

Proposed Fix

In the Skill Discovery reconciliation path, when updating .status.linkedSkills:

1. Re-fetch the latest object before the status update (get the current resourceVersion)
2. Use client.Status().Patch() instead of client.Status().Update() to reduce conflict surface
3. Or: Add retry logic with retry.RetryOnConflict() from k8s.io/client-go/util/retry

Example pattern

import "k8s.io/client-go/util/retry"

err := retry.RetryOnConflict(retry.DefaultRetry, func() error {
    // Re-fetch the latest version
    if err := r.Get(ctx, req.NamespacedName, agentRuntime); err != nil {
        return err
    }
    // Update status
    agentRuntime.Status.LinkedSkills = discoveredSkills
    return r.Status().Update(ctx, agentRuntime)
})

Where to look

• The reconciler that handles AgentRuntime objects with skill annotations
• Search for where linkedSkills or LinkedSkills is set in the status
• The fix should wrap that status update with retry.RetryOnConflict

Labels

• bug
• e2e
• priority/high