Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
186 changes: 186 additions & 0 deletions openspec/changes/20260508-operator-feature-parity-d-e/design.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,186 @@
# Design: Operator Feature Parity — Land D & E Epics on Kubernetes

---

## Approach

Four focused PRs land everything missing in the operator install path. The PR sequence is
sized so each is independently reviewable and the cluster keeps working after every merge.

## Opinionated Decisions

### 1. Role enum — drop the closed Enum, keep regex + webhook catalogue

Replace
```go
// +kubebuilder:validation:Enum=ingester;analyst;synthesizer;arbiter;observer
type AgentRole string
```
with
```go
// +kubebuilder:validation:Pattern=`^[a-z][a-z0-9_]{1,62}$`
// +kubebuilder:validation:MinLength=2
// +kubebuilder:validation:MaxLength=63
type AgentRole string
```

A new validating webhook (`agentcollective_webhook.go`, parallel to existing
`agentcorpus_webhook.go`) does the *semantic* check against a `KnownRoles` catalogue.
The catalogue is `go:embed`-baked from the live `roles/*/role.yaml` directory listing at
compile time, generated by `operator/hack/gen-catalogue.go`. Unknown roles return a clear
"did you mean…" error.

**Trade-off vs. a `RoleCatalogue` CRD**: a CRD would be reusable but adds a second
admission cycle and a fourth sample to maintain. For a 17-role catalogue that ships with
the operator binary anyway, embedding wins.

### 2. Manifest delivery — corpus-scoped ConfigMaps from operator embed.FS

Three ConfigMaps per `AgentCorpus`:

| Name | Source | Approx size |
|--------------|------------------|-------------|
| `acc-roles` | `roles/` tree | ~612 KiB |
| `acc-skills` | `skills/` tree | ~124 KiB |
| `acc-mcps` | `mcps/` tree | ~24 KiB |

All well under the 1 MiB ConfigMap limit per object. ConfigMap keys cannot contain `/`, so
keys flatten path separators to `__` (e.g. `coding_agent_implementer__role.yaml`) and the
volume mount projects them back to slash-paths via `items[]: [{key: ..., path: ...}]`.
This is a known operator pattern.

Agents read these via the existing env-var contract:

| Env var | Mounted at | Default in `acc/` |
|--------------------|-------------------|-------------------|
| `ACC_ROLES_ROOT` | `/etc/acc/roles` | `roles/` |
| `ACC_SKILLS_ROOT` | `/etc/acc/skills` | `skills/` |
| `ACC_MCPS_ROOT` | `/etc/acc/mcps` | `mcps/` |

A new corpus-level field `spec.manifestDelivery: {all|none}` (default `all`) lets users
who bake the trees into a custom agent image opt out.

**Trade-off vs. baking into `Containerfile.agent-core`**: ConfigMaps let users swap roles
without rebuilding/repushing the agent image, and let the operator OWN-reference them so
deletion is clean.

### 3. MCP servers — corpus-scoped sub-reconciler

A new `MCPServerSpec` slice on `AgentCorpusSpec` (corpus-scoped, mirroring compose's
single shared `acc-net`). The sub-reconciler emits Deployment + Service named
`acc-mcp-{name}` — matching the `url:` field in `mcps/*/mcp.yaml` so manifests need no
rewrite.

```go
type MCPServerSpec struct {
Name string // matches mcps/<name>/mcp.yaml
Image string
Replicas int32 // default 1
Port int32 // default 8080
Env []corev1.EnvVar
SecretEnv []corev1.EnvFromSource // BRAVE_API_KEY, etc.
ShmSizeMi int32 // browser-harness needs ≥256
Resources *corev1.ResourceRequirements
}
```

Status mirror: `Status.MCPServerStatuses map[string]MCPServerStatus` with `Ready bool`,
`Replicas int32`, `ServiceURL string`.

### 4. Reconciler chain — slot the two new sub-reconcilers

```go
func (r *AgentCorpusReconciler) buildSubReconcilers() []reconcilers.SubReconciler {
return []reconcilers.SubReconciler{
&reconcilers.PrerequisiteReconciler{...},
&manifests.ManifestDeliveryReconciler{...}, // NEW (PR-50) — slot 2
&reconcilers.UpgradeReconciler{...},
&infra.NATSReconciler{...},
&infra.RedisReconciler{...},
&infra.MilvusReconciler{},
&governance.OPABundleServerReconciler{...},
&governance.GatekeeperReconciler{...},
&bridge.KafkaBridgeReconciler{...},
&mcp.MCPServerReconciler{...}, // NEW (PR-51)
&observability.OTelCollectorReconciler{...},
&observability.PrometheusRulesReconciler{...},
&collectiverec.CollectiveReconciler{...},
}
}
```

`ManifestDelivery` slots #2 because ConfigMaps must exist before the collective
reconciler builds agent Deployments. `MCPServer` slots between Kafka and OTel: MCPs
should be Ready before agents try to call them, but the chain still progresses if MCPs
are degraded (agents handle MCP outages gracefully via `acc/mcp/registry.py` lazy-init).

### 5. Backwards compatibility — additive only

| Check | Status |
|---|---|
| Existing 5 roles still parse | Yes — regex `^[a-z][a-z0-9_]{1,62}$` matches all five |
| `acc_v1alpha1_agentcorpus_standalone.yaml` still applies clean | Yes |
| `acc_v1alpha1_agentcorpus_rhoai.yaml` still applies clean | Yes |
| Old CRs without `spec.mcpServers` reconcile fine | Yes — `omitempty`, reconciler iterates a (possibly nil) slice |
| Old CRs without `spec.manifestDelivery` mount the trees | Yes — webhook defaulter sets `"all"` |
| Older operator binary on new CRD | Tolerable — old binary ignores unknown spec fields |
| `make generate` no-op confirms `zz_generated.deepcopy.go` after `}` fix | Yes |

CSV graph: `acc-operator.v0.1.0` → `acc-operator.v0.2.0` with `replaces:
acc-operator.v0.1.0` for OLM seamless upgrade.

## Critical files

Modified:
- `operator/api/v1alpha1/agentcollective_types.go` (PR-49 — `}` fix + enum loosening)
- `operator/api/v1alpha1/agentcorpus_types.go` (PR-49 — MCP + delivery fields)
- `operator/api/v1alpha1/common_types.go` (PR-49 — enum loosening + role consts)
- `operator/internal/controller/agentcorpus_controller.go` (PR-50/51 — wire-in)
- `operator/internal/reconcilers/collective/agent_deployment.go` (PR-50 — volume + env)
- `operator/bundle/manifests/acc-operator.clusterserviceversion.yaml` (PR-52)
- `operator/config/samples/acc_tui_deployment.yaml` (PR-50 — TUI parity)

New:
- `operator/internal/rolecatalogue/catalogue.go` + `operator/hack/gen-catalogue.go` (PR-49)
- `operator/api/v1alpha1/agentcollective_webhook.go` (PR-49)
- `operator/internal/reconcilers/manifests/delivery.go` (PR-50)
- `operator/internal/reconcilers/mcp/server.go` (+ test) (PR-51)
- `operator/config/samples/acc_v1alpha1_agentcorpus_{autoresearcher,coding_split}.yaml` (PR-52)
- `operator/hack/test-kind.sh` (PR-52)

## Verification

- **PR-49**: `make generate manifests` is a no-op after the `}` fix; existing samples still
apply; new `operator/test/unit/role_catalogue_test.go` covers catalogue membership and
closest-match suggestions.
- **PR-50**: envtest creates the legacy `sol-corpus` sample, asserts `acc-roles` /
`acc-skills` / `acc-mcps` ConfigMaps exist with key counts equal to
`find roles -type f | wc -l` etc.; agent Deployment carries the three env vars + three
volumes; `manifest_delivery_test.go` exercises a small fixture FS.
- **PR-51**: envtest creates a corpus with `mcpServers: [{name: web-fetch, image: ...}]`,
asserts Deployment + Service `acc-mcp-web-fetch` exist; status populated.
- **PR-52**: end-to-end on kind via `hack/test-kind.sh`:
```
kind create cluster --name acc-operator-test
make install deploy IMG=...
kubectl apply -f config/samples/acc_v1alpha1_agentcorpus_autoresearcher.yaml
kubectl wait --for=condition=Ready agentcorpus/acc-autoresearcher-corpus --timeout=10m
kubectl exec deploy/acc-autoresearcher-01-research-planner -- ls /etc/acc/roles | grep research_planner
kubectl exec deploy/acc-autoresearcher-01-research-planner -- ls /etc/acc/mcps | grep web_search_brave
```
- **Bundle scorecard** (`make bundle && operator-sdk scorecard`) exercises the new
`alm-examples` automatically.

## Coordination with the second Claude instance (10.199.12.8)

Both instances rendezvous on this openspec change. `tasks.md` is the lock board: each task
has a `claimed by:` slot. Before starting work an instance:

1. `git pull` and check `tasks.md` for unclaimed PR slots and unclaimed tasks within them.
2. Edit `tasks.md` to set `claimed by: <hostname>`, commit + push immediately (single-file
commit). The push is the lock acquisition.
3. Branch naming: `feat/op-pr<NN>-<short-slug>` (e.g. `feat/op-pr48-api-foundations`).
4. Open PR as **draft** immediately on first push so `gh pr list --state all` reflects
the in-flight slot.
5. Mark each task line `- [x]` when its commit lands on the branch; convert PR from draft
to ready when all PR-NN tasks are checked.
87 changes: 87 additions & 0 deletions openspec/changes/20260508-operator-feature-parity-d-e/proposal.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
# Proposal: Operator Feature Parity — Land D & E Epics on Kubernetes

| Field | Value |
|------------|----------------------------------------------------------|
| Change ID | 20260508-operator-feature-parity-d-e |
| Date | 2026-05-08 |
| Status | Draft |
| PR slots | this proposal lands as #48; implementation in #49 → #52 |
| Depends on | D1–D6 (coding-split-skills, PRs #34–#40), E1–E6 (autoresearcher, PRs #41–#46) |

---

## Problem Statement

The repo has two deployment paths:

1. **Podman Compose** (`./acc-deploy.sh up` → `container/production/podman-compose.yml`) —
works end-to-end including everything from PRs #26–#46 (cluster_id propagation, role-md
authoring, sub-cluster estimator, slash commands, autoresearcher MCPs, split coding/
research personas).

2. **Kubernetes operator** (`make install && make deploy` from `operator/`) — frozen at the
original ACCv3 5-role design. None of the new features landed here.

The `examples/acc_autoresearcher/` and `examples/coding_split_skills/` showcases run only
via Podman Compose. They cannot be deployed on Kubernetes via the operator.

## Current Behavior (audited against source)

1. **`AgentRole` enum is locked to 5 legacy roles**.
`operator/api/v1alpha1/common_types.go:32` and `agentcollective_types.go:97` constrain
`agents[].role` to `[ingester, analyst, synthesizer, arbiter, observer]`. Same in the
generated CRD `operator/config/crd/bases/acc.redhat.io_agentcollectives.yaml:126-132,
278-284`. The API server will reject all 11 new personas at admission:
`coding_agent_{architect,dependency,implementer,reviewer,tester}` (D3) and
`research_{planner,strategist,economist,competitor,synthesizer,critic}` (E4). Even the
umbrella `coding_agent` used by `container/production/podman-compose.yml:212` is rejected.

2. **`roles/`, `skills/`, `mcps/` are not delivered to agent pods**.
`deploy/Containerfile.agent-core:57` copies only `acc/` + `acc-config.yaml`.
`operator/internal/reconcilers/collective/agent_deployment.go:147-189` mounts only
`acc-config`, `wasm-governance`, and a tiny inline `acc-role` ConfigMap rendered from
`spec.roleDefinition` (purpose / persona / taskTypes / seedContext / allowedActions /
categoryBOverrides / version — too narrow for full markdown roles with estimator /
default_skills / allowed_mcps / system_prompt). Compare to compose's
`../../roles:/app/roles:ro,z` bind mounts at lines 222, 251, 280.

3. **No MCP-server reconciler**. `grep -i 'skill\|mcp' operator/` returns empty. The three
E2 MCPs (`web_browser_harness`, `web_search_brave`, `web_fetch`) and the diagnostic
`echo_server` exist only as compose services under `--profile acc-autoresearcher` /
`--profile mcp-echo`.

4. **CSV `alm-examples` and sample manifests are stale**.
`operator/bundle/manifests/acc-operator.clusterserviceversion.yaml:42-46` and both
`operator/config/samples/*.yaml` only reference legacy 5 roles.

5. **TUI sample doesn't mount roles/**.
`operator/config/samples/acc_tui_deployment.yaml:36-78` doesn't set `ACC_ROLES_ROOT` or
mount `roles/`, while `container/production/podman-compose.yml:464` does. The cluster-
topology panel and markdown-role tooling will degrade.

6. **Compile bug**: `operator/api/v1alpha1/agentcollective_types.go:91-94` — the
`RoleDefinition` struct is missing its closing `}` before `AgentRoleSpec` begins.
`zz_generated.deepcopy.go` was generated from a previously correct version, so a fresh
`make generate` would fail.

## Desired Behavior

After this change, an operator can:

- `kubectl apply -f operator/config/samples/acc_v1alpha1_agentcorpus_autoresearcher.yaml`
and have the autoresearcher demo come up end-to-end (6 research personas + 3 MCP servers)
- `kubectl apply -f operator/config/samples/acc_v1alpha1_agentcorpus_coding_split.yaml`
and have the coding-split-skills demo come up end-to-end (5 coding personas + echo MCP)
- Use any role name found in `roles/` without the API server rejecting the manifest;
unknown names get a webhook error listing the closest matches.
- Run the legacy 5-role samples (`sol-corpus`, `rhoai-corpus`) **unchanged** — every change
is additive or a strict loosening.

## Non-Goals

- No edits to `container/production/podman-compose.yml` or `acc-deploy.sh`.
- No Python agent code changes — env-var contract (`ACC_ROLES_ROOT`,
`ACC_SKILLS_ROOT`, `ACC_MCPS_ROOT`) already in place
(`acc/skills/registry.py:39-45`, `acc/mcp/registry.py:42-46`).
- No new CRD kinds (e.g. `RoleCatalogue`, `MCPServer`) — kept inline in `AgentCorpusSpec`.
- No CSV graph rewrite — clean `replaces:` v0.1.0 → v0.2.0 for OLM upgrade.
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
# Spec: ACC Operator — Feature Parity for D & E Epics

| Field | Value |
|---------------|--------------------------------------------------------------------|
| Spec path | `openspec/changes/20260508-operator-feature-parity-d-e/specs/operator/spec.md` |
| Capability | operator |
| Base spec | `openspec/changes/20260414-acc-operator-v0.1.0/specs/operator/spec.md` |
| Change ID | 20260508-operator-feature-parity-d-e |

---

## MODIFIED

### Role validation

**REQ-OP-ROLE-001** (was: `agents[].role` enum constrained to 5 values) The
`AgentCollective.spec.agents[].role` and `spec.scaling.roleScaling[].role` fields SHALL
be validated by the regex `^[a-z][a-z0-9_]{1,62}$` with `MinLength=2` and
`MaxLength=63`. The closed `Enum=` marker SHALL be removed.

**REQ-OP-ROLE-002** (NEW behaviour layered on REQ-OP-ROLE-001) An admission webhook
registered for `AgentCollective` SHALL reject role names not present in the operator's
embedded `KnownRoles` catalogue. The rejection error SHALL include the closest matching
catalogue entries.

**REQ-OP-ROLE-003** The `KnownRoles` catalogue SHALL be generated at compile time from
the listing of `roles/*/role.yaml` directories in the source tree, via
`operator/hack/gen-catalogue.go` triggered by `//go:generate`.

## ADDED

### MCP server management

**REQ-OP-MCP-001** `AgentCorpusSpec` SHALL accept an optional `mcpServers` slice of
`MCPServerSpec` entries. Each entry SHALL carry: `Name` (DNS-label-safe pattern
`^[a-z][a-z0-9-]{1,62}$`), `Image`, `Replicas` (default 1), `Port` (default 8080), `Env`,
`SecretEnv`, `ShmSizeMi`, and `Resources`.

**REQ-OP-MCP-002** A new `MCPServerReconciler` SHALL emit one Deployment and one Service
per `MCPServerSpec`. The Service name SHALL be `acc-mcp-{Name}` so that it matches the
`url:` field present in `mcps/{Name}/mcp.yaml`, requiring no manifest rewrite.

**REQ-OP-MCP-003** When `MCPServerSpec.ShmSizeMi > 0`, the rendered Deployment SHALL
include a `Memory`-medium `emptyDir` mounted at `/dev/shm` with `sizeLimit` set to
`{ShmSizeMi} Mi`. This is required for the browser-harness MCP (Chromium needs ≥256 MiB
shared memory).

**REQ-OP-MCP-004** Per-MCP readiness SHALL be aggregated into
`AgentCorpusStatus.MCPServerStatuses[Name]` with fields `Ready`, `Replicas`, `ServiceURL`.

**REQ-OP-MCP-005** The `MCPServerReconciler` SHALL slot in the reconciler chain between
`KafkaBridgeReconciler` and `OTelCollectorReconciler`. MCP outages SHALL NOT block the
overall corpus phase from progressing — agents handle MCP unavailability via lazy-init in
`acc/mcp/registry.py`.

### Manifest delivery

**REQ-OP-MANIFEST-001** `AgentCorpusSpec` SHALL accept an optional `manifestDelivery`
enum field with values `all` (default) and `none`.

**REQ-OP-MANIFEST-002** When `manifestDelivery=all`, a new `ManifestDeliveryReconciler`
SHALL emit three corpus-namespace ConfigMaps: `acc-roles`, `acc-skills`, `acc-mcps`. The
content SHALL be sourced from `embed.FS` trees baked into the operator binary at compile
time from the live `roles/`, `skills/`, `mcps/` directories.

**REQ-OP-MANIFEST-003** ConfigMap keys SHALL flatten path separators by replacing `/`
with `__`. The corresponding agent-pod volume mount SHALL use an explicit `items[]`
projection list so that the in-pod filesystem layout preserves the original directory
structure (e.g. ConfigMap key `coding_agent_implementer__role.yaml` projects to
`/etc/acc/roles/coding_agent_implementer/role.yaml`).

**REQ-OP-MANIFEST-004** When `manifestDelivery=all`, agent pod containers SHALL receive
three `VolumeMount`s (`/etc/acc/roles`, `/etc/acc/skills`, `/etc/acc/mcps`, all
read-only) and three env vars (`ACC_ROLES_ROOT=/etc/acc/roles`,
`ACC_SKILLS_ROOT=/etc/acc/skills`, `ACC_MCPS_ROOT=/etc/acc/mcps`). When
`manifestDelivery=none`, the operator SHALL NOT inject these mounts or env vars (allowing
users to bake the trees into a custom agent image).

**REQ-OP-MANIFEST-005** The `ManifestDeliveryReconciler` SHALL slot first in the
reconciler chain after `PrerequisiteReconciler` and before `UpgradeReconciler`. The
ConfigMaps SHALL exist before any agent Deployment is built.

**REQ-OP-MANIFEST-006** The TUI sample at `operator/config/samples/acc_tui_deployment.yaml`
SHALL set `ACC_ROLES_ROOT=/etc/acc/roles` and `ACC_SKILLS_ROOT=/etc/acc/skills`, and SHALL
mount the corresponding ConfigMaps with `items[]` projection identical to the agent pods.
This achieves parity with `container/production/podman-compose.yml:464` where the TUI
container has `ACC_ROLES_ROOT=/app/roles`.

### Demo samples

**REQ-OP-SAMPLE-001** `operator/config/samples/` SHALL include a runnable autoresearcher
demo manifest (`acc_v1alpha1_agentcorpus_autoresearcher.yaml`) that deploys 6 research
personas plus the 3 autoresearcher MCP servers and matches the topology described in
`examples/acc_autoresearcher/expected_topology.md`.

**REQ-OP-SAMPLE-002** `operator/config/samples/` SHALL include a runnable coding-split
demo manifest (`acc_v1alpha1_agentcorpus_coding_split.yaml`) that deploys 5 coding
personas plus the echo MCP server and matches the topology described in
`examples/coding_split_skills/expected_topology.md`.

**REQ-OP-SAMPLE-003** The OLM bundle CSV `alm-examples` SHALL include both demo
manifests in addition to the legacy `sol-corpus` example, giving OperatorHub users a
one-click "Try it" path for either demo.

### Versioning

**REQ-OP-VERSION-001** The operator bundle CSV version SHALL bump to `v0.2.0` with
`replaces: acc-operator.v0.1.0` for OLM seamless upgrade. No CRD `apiVersion` bump is
required — all schema changes are additive or strict loosenings of validation.
Loading