feat(platform): Add operator SPIFFE authentication bootstrap by Alan-Cha · Pull Request #2135 · rossoctl/rossoctl

Alan-Cha · 2026-07-01T18:23:52Z

Summary

Adds operator SPIFFE authentication bootstrap: the operator registers itself
as a Keycloak client using its SPIFFE identity (JWT-SVID) instead of static
admin credentials.

What's Changed

Core feature

Python bootstrap script (kagenti/auth/operator-spiffe-bootstrap/) that configures
the SPIFFE Identity Provider in Keycloak and registers the operator client with
clientAuthenticatorType: federated-jwt
Helm post-install/upgrade Job template for the bootstrap (operator-client-bootstrap-job.yaml)
Feature gated behind kagenti-operator-chart.spiffe.operatorAuth.enabled (default: false)

Chart fixes found during E2E testing

authbridge-config in kagenti-system — operator's client registration controller
reads Keycloak config from this ConfigMap in its own namespace; it was only being
created in agent namespaces (team1, team2). Added conditional creation in
kagenti-system when components.agentOperator.enabled: true.
Duplicate resources → STATUS: failed — agent-namespaces.yaml and
agent-namespace-resources.yaml both rendered the same 12 resources per agent
namespace, causing 10 "already exists" errors on every install. Stripped
agent-namespaces.yaml down to Namespace objects only.
Helm template syntax — removed extra {{- end }} statements in
agent-namespace-resources.yaml

Script updates

scripts/kind/setup-kagenti.sh — adds --enable-operator-spiffe-auth flag; fixes helm key to kagenti-operator-chart.spiffe.operatorAuth.enabled
.github/scripts/local-setup/kind-full-test.sh — propagates ENABLE_OPERATOR_SPIFFE_AUTH

Feature flag defaults changed

featureFlags.authbridgeAPI: false → true (AuthBridge tab now shown by default)
featureFlags.admin: false → true (Platform Status card now shown by default)

E2E test results

Validated across multiple fresh-cluster runs:

✅ Operator pod: 2/2 containers (manager + spiffe-helper)
✅ Helm release: STATUS: deployed
✅ Test agent registered with SPIFFE ID spiffe://localtest.me/ns/team1/sa/test-agent
✅ Keycloak token endpoint: HTTP 200 with access token

Test materials (procedure + setup script + requirements):
https://gist.github.com/Alan-Cha/527be60c79b40aedd54fe7be586c3452

Note: the setup script in the gist requires a local checkout of
kagenti-operator on branch feat/spiffe-dcr-client-registration (PR #349).
It packages the operator helm chart from source at test time — the published
OCI chart does not yet include the spiffe-helper sidecar from that PR.

Checklist

All commits signed-off (DCO)
Feature gated (default: disabled)
Backward compatible
E2E tested (multiple fresh-cluster runs)

Assisted-By: Claude Code

Add Helm job to automatically bootstrap operator client in Keycloak with SPIFFE authentication. Update installation scripts to support ENABLE_OPERATOR_SPIFFE_AUTH flag. Changes: - charts/kagenti/templates/operator-client-bootstrap-job.yaml: New Helm hook job - Creates SPIFFE IdP in Keycloak if not exists - Creates operator client with federated-jwt authentication - Assigns manage-clients role to operator service account - charts/kagenti/values.yaml: Add kagentiOperator and spiffe configuration - kind-full-test.sh: Add ENABLE_OPERATOR_SPIFFE_AUTH env var support - setup-kagenti.sh: Add --enable-operator-spiffe-auth flag Bootstrap job runs as post-install/post-upgrade Helm hook and uses admin credentials for one-time setup. After bootstrap completes, operator authenticates with JWT-SVID (no admin credentials needed). Usage: ENABLE_OPERATOR_SPIFFE_AUTH=true ./.github/scripts/local-setup/kind-full-test.sh Assisted-By: Claude Code (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Alan Cha <Alan.cha1@ibm.com>

Add bootstrap job that registers the operator as a Keycloak client using federated-jwt (SPIFFE) authentication, eliminating the need for static admin credentials. Components: - Python bootstrap script (setup_operator_client.py) - Docker images for bootstrap job - Helm post-install/upgrade hook Job template - Configures SPIFFE Identity Provider in Keycloak - Registers operator client with federated-jwt auth type The operator can now authenticate to Keycloak using its SPIFFE ID (spiffe://$TRUST_DOMAIN/ns/$NS/sa/$SA) instead of admin credentials. Feature is controlled by kagentiOperator.spiffeAuth.enabled (default: false) for backward compatibility. Related: PR #1837, PR #349 Assisted-By: Claude Code Signed-off-by: Alan Cha <Alan.cha1@ibm.com>

1. Remove extra {{- end }} statements in agent-namespace-resources.yaml - Template had 5 closing statements but only 3 opening blocks - Caused Helm template rendering to fail 2. Fix agent namespace hook ordering - Ensure namespaces exist before RoleBinding creation - Prevents hook timeout errors 3. Consolidate operator SPIFFE auth config - Single value path: kagentiOperator.spiffeAuth.enabled - Simplifies configuration and reduces duplication Issues discovered during E2E testing and fixed in this commit. Related: PR #1837 Assisted-By: Claude Code Signed-off-by: Alan Cha <Alan.cha1@ibm.com>

Add complete end-to-end testing documentation and automation: 1. Manual test procedure (operator-spiffe-auth-manual-test-STABLE.md) - 9-step E2E test validating complete authentication flow - Troubleshooting guide with all issues encountered and fixed - Expected outputs and success criteria 2. Automated setup script (setup-operator-spiffe-test.sh) - Installs cert-manager, Istio, SPIRE, Keycloak - Runs SPIFFE IdP setup with local bootstrap image - Deploys Kagenti with operator SPIFFE auth enabled - Auto-applies local operator subchart (handles OCI registry issue) 3. Requirements document (spiffe-jwt-auth-requirements.md) - Complete checklist for SPIFFE JWT authentication with Keycloak - Covers Keycloak config, SPIRE config, client requirements - Common pitfalls and troubleshooting Testing validated successfully from clean Kind cluster (all 9 steps passed). Related: PR #1837, PR #349 Assisted-By: Claude Code Signed-off-by: Alan Cha <Alan.cha1@ibm.com>

Add setup script and manual test documentation for operator SPIFFE auth E2E testing. Assisted-By: Claude Code Signed-off-by: Alan Cha <Alan.cha1@ibm.com>

- Update branch name: feat/operator-spiffe-auth-bootstrap → feat/operator-spiffe-auth-clean - Update PR number: #1837 → #2135 The clean branch has 4 logical commits instead of 436 for easier review. Assisted-By: Claude Code Signed-off-by: Alan Cha <Alan.cha1@ibm.com>

…t registration The kagenti-operator clientregistration controller reads KEYCLOAK_URL and KEYCLOAK_REALM from an authbridge-config ConfigMap in its own namespace (kagenti-system). Previously this ConfigMap was only created in agent namespaces (team1, team2), causing the operator to log 'waiting for KEYCLOAK_URL/KEYCLOAK_REALM' indefinitely and never registering agents. Add authbridge-config to kagenti-system when components.agentOperator.enabled is true. Also fix the E2E test setup script to delete Chart.lock alongside the OCI tarball. Helm re-downloads the locked version when Chart.lock is present even after the tarball is removed, defeating the workaround that forces the local subchart directory (alpha.4, with spiffe-helper sidecar) to be used. Signed-off-by: Alan Cha <alan.cha@ibm.com> Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Alan Cha <Alan.cha1@ibm.com>

… subchart Move OCI tarball and Chart.lock removal to before the initial helm install so the correct local operator subchart (with spiffe-helper sidecar) is used on the first pass. Eliminates the two-step install+upgrade that was prone to silent failures when helm re-fetched the OCI tarball via Chart.lock. Also inline pullPolicy: Never into the values file so it's set from install time rather than as a post-install override. Signed-off-by: Alan Cha <alan.cha@ibm.com> Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Alan Cha <Alan.cha1@ibm.com>

Add charts/kagenti/charts/kagenti-operator-chart/ (v0.3.0-alpha.4) to the repo so the E2E test works on a fresh clone without depending on a previously cached helm pull. The published OCI chart versions do not include the spiffe-helper sidecar template or the spiffe.operatorAuth values needed for operator SPIFFE authentication. The setup script removes the OCI tarball and Chart.lock after running helm dependency update, causing helm to use this committed directory chart instead of the downloaded version. Signed-off-by: Alan Cha <alan.cha@ibm.com> Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Alan Cha <Alan.cha1@ibm.com>

agent-namespaces.yaml and agent-namespace-resources.yaml both rendered the same secrets (openai-secret, slack-secret, quay-registry-secret, ghcr-secret, github-token-secret, github-shipwright-secret) and configmaps (envoy-config, authbridge-runtime-config) plus OpenShift SCC RoleBindings for each agent namespace. This caused helm to report 10 'already exists' errors on every install, leaving the release in STATUS: failed even though all pods came up correctly. The intent documented in agent-namespace-resources.yaml was always that agent-namespaces.yaml should only create Namespace objects. Strip it down to that — all other per-namespace resources remain in agent-namespace-resources.yaml which is the canonical location. Signed-off-by: Alan Cha <alan.cha@ibm.com> Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Alan Cha <Alan.cha1@ibm.com>

Remove stale blocking-issue sections, outdated workarounds, and the 'PRs NOT READY TO MERGE' footer. Replace with accurate procedure reflecting the current clean state after three E2E runs. Key changes: - Updated status to COMPLETE / READY TO MERGE - Fixed bootstrap build command context (kagenti/auth/... not auth/...) - Replaced docker save/load tarball approach with simpler kind load docker-image - Updated Step 6 verify command to search full logs by pod name (--tail=50 can miss the SPIFFE auth message due to log ordering) - Documented helm pending-install transient state as a known non-issue - Added table of all six fixes included in PR #2135 - Trimmed from 1810 lines to a focused ~270-line procedure Signed-off-by: Alan Cha <alan.cha@ibm.com> Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Alan Cha <Alan.cha1@ibm.com>

The Dockerfile uses COPY auth/operator-spiffe-bootstrap/... paths relative to the build context, so the context must be kagenti/ (the inner subdirectory), not kagenti/auth/operator-spiffe-bootstrap/. The previous verify check (docker images | grep ...) gave a false positive when an old cached image existed even if the build failed. Replaced with echo of the exit code. Signed-off-by: Alan Cha <alan.cha@ibm.com> Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Alan Cha <Alan.cha1@ibm.com>

Same issue as bootstrap Step 3 — docker images grep can match an old cached image even if the build failed. Replace with exit code check. Signed-off-by: Alan Cha <alan.cha@ibm.com> Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Alan Cha <Alan.cha1@ibm.com>

…chart The kagenti-operator-chart directory contained a nested kagenti-operator/ subdirectory that is never rendered by helm — it would need to be under a charts/ subdirectory to be treated as a subchart, but it was at the chart root. It's a packaging artifact with no effect on the deployment. Its CRDs were also a different version from the outer chart's crds/ directory, making it a source of confusion. Removing 27 files / ~2218 lines. Signed-off-by: Alan Cha <alan.cha@ibm.com> Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Alan Cha <Alan.cha1@ibm.com>

The setup script deletes charts/kagenti/Chart.lock as part of the OCI subchart override workaround. Chart.lock is a tracked file, so after the script exits the working tree shows an unstaged deletion — confusing for users running git status and a risk of accidental commit. Restore it at the end of the script with git checkout --. Signed-off-by: Alan Cha <alan.cha@ibm.com> Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Alan Cha <Alan.cha1@ibm.com>

…ource instead The kagenti-operator-chart belongs in the kagenti-operator repo (PR #349, branch feat/spiffe-dcr-client-registration). Committing it to the kagenti repo was wrong — it duplicates chart ownership, adds helm dependency artifacts that don't belong in git, and used a different chart version than what's actually in PR #349. The correct approach: the setup script now packages the chart directly from the kagenti-operator source checkout using helm package, installs it temporarily for the helm install, then cleans up. This means the chart changes live where they belong (in kagenti-operator) and the kagenti repo stays clean. Setup script changes: - Require KAGENTI_OPERATOR_REPO env var (default: ../kagenti-operator) - Validate the repo has charts/kagenti-operator/ with spiffe-helper support - Package the chart at install time with helm package - Remove the temporary chart directory and restore Chart.lock at the end Test docs: update Step 2 to use correct branch name (feat/spiffe-dcr-client-registration, not pr-349) and Step 6 to show KAGENTI_OPERATOR_REPO usage. Signed-off-by: Alan Cha <alan.cha@ibm.com> Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Alan Cha <Alan.cha1@ibm.com>

Test procedure, setup script, and requirements doc moved to: https://gist.github.com/Alan-Cha/527be60c79b40aedd54fe7be586c3452 These are testing aids for validating PR #2135 + kagenti-operator#349 and don't belong as permanent fixtures in the repo. Signed-off-by: Alan Cha <alan.cha@ibm.com> Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Alan Cha <Alan.cha1@ibm.com>

Alan-Cha

Three findings — all suggestions/nits, no blocking issues. The core feature (SPIFFE bootstrap job, authbridge-config in kagenti-system, duplicate-resource fix in agent-namespaces.yaml) is correct and well-tested across multiple fresh-cluster E2E runs.

Areas reviewed: Python, Helm/K8s, Dockerfile, Shell
Commits: 17 commits, all signed-off ✅, all follow conventional commit format ✅
CI: DCO passes; E2E HyperShift pending maintainer trigger (expected)

Verdict: APPROVE (posted as comment — GitHub does not allow self-approval)

Alan-Cha · 2026-07-02T18:44:52Z

+            logger.info("✓ Admin credentials retrieved")
+            return username, password
+
+        except ApiException as e:


suggestion: self.session.verify = False disables TLS certificate verification for all requests made via this session. The current deployment uses HTTP for in-cluster Keycloak traffic so this has no runtime effect today, but it is a dangerous pattern — if Keycloak is ever configured with HTTPS, certificates won't be verified. Replace with the in-cluster CA bundle (/var/run/secrets/kubernetes.io/serviceaccount/ca.crt), or if HTTP-only is intentional add an explicit assertion and comment.

Alan-Cha · 2026-07-02T18:44:52Z

@@ -0,0 +1,12 @@
+FROM python:3.14-slim@sha256:5b3879b6f3cb77e712644d50262d05a7c146b7312d784a18eff7ff5462e77033


suggestion: Dockerfile.local is a dev-convenience file with looser version pins (requests==2.32.3 vs 2.34.2 in the main Dockerfile) and no digest pinning. Dev-only files like this tend to accumulate and create confusion about which image is canonical. Consider removing before merge, or add a .dockerignore entry and a comment to make the local-only intent explicit.

setup-kagenti.sh was passing --set kagentiOperator.spiffeAuth.enabled which no longer exists — values.yaml comments note it was consolidated into kagenti-operator-chart.spiffe.operatorAuth. The bootstrap job template checks kagenti-operator-chart.spiffe.operatorAuth.enabled so the old key was silently ignored, making --enable-operator-spiffe-auth a no-op. Signed-off-by: Alan Cha <alan.cha@ibm.com> Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Alan Cha <Alan.cha1@ibm.com>

Alan-Cha · 2026-07-02T19:32:31Z

Superseded by #2141 — clean 3-commit version with accurate descriptions.

Alan-Cha added 5 commits June 6, 2026 15:48

docs(testing): add test scripts and documentation

98641dd

Add setup script and manual test documentation for operator SPIFFE auth E2E testing. Assisted-By: Claude Code Signed-off-by: Alan Cha <Alan.cha1@ibm.com>

github-project-automation Bot added this to Rossoctl Issue Prioritization Jul 1, 2026

github-project-automation Bot moved this to New/ToDo in Rossoctl Issue Prioritization Jul 1, 2026

Alan-Cha mentioned this pull request Jul 1, 2026

feat(platform): Add operator SPIFFE authentication bootstrap #1837

Closed

Alan-Cha commented Jul 1, 2026

View reviewed changes

Comment thread charts/kagenti/values.yaml

Alan-Cha commented Jul 1, 2026

View reviewed changes

Comment thread charts/kagenti/values.yaml

Alan-Cha marked this pull request as draft July 1, 2026 18:31

Alan-Cha added 11 commits July 1, 2026 15:03

Alan-Cha commented Jul 2, 2026

View reviewed changes

Alan-Cha marked this pull request as ready for review July 2, 2026 18:48

Alan-Cha mentioned this pull request Jul 2, 2026

Feat: SPIFFE authentication for operator client registration (Admin API) rossoctl/operator#349

Closed

Alan-Cha mentioned this pull request Jul 2, 2026

feat(operator): add SPIFFE authentication bootstrap for operator client registration #2141

Merged

4 tasks

Alan-Cha closed this Jul 2, 2026

github-project-automation Bot moved this from New/ToDo to Done in Rossoctl Issue Prioritization Jul 2, 2026

pdettori deleted the feat/operator-spiffe-auth-clean branch July 15, 2026 22:31

Navigation Menu

Uh oh!

feat(platform): Add operator SPIFFE authentication bootstrap - #2135

feat(platform): Add operator SPIFFE authentication bootstrap#2135
Alan-Cha wants to merge 18 commits into
mainfrom
feat/operator-spiffe-auth-clean

Alan-Cha commented Jul 1, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Alan-Cha left a comment

Uh oh!

Alan-Cha Jul 2, 2026

Uh oh!

Alan-Cha Jul 2, 2026

Uh oh!

Alan-Cha commented Jul 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

		@@ -0,0 +1,12 @@
		FROM python:3.14-slim@sha256:5b3879b6f3cb77e712644d50262d05a7c146b7312d784a18eff7ff5462e77033

Uh oh!

Conversation

Alan-Cha commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What's Changed

Core feature

Chart fixes found during E2E testing

Script updates

Feature flag defaults changed

E2E test results

Related

Checklist

Uh oh!

Uh oh!

Uh oh!

Alan-Cha left a comment

Choose a reason for hiding this comment

Uh oh!

Alan-Cha Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

Alan-Cha Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

Alan-Cha commented Jul 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Alan-Cha commented Jul 1, 2026 •

edited

Loading