Deploy ToolHive Operator into OpenShift

[ChrisJBurns]

We want to be able to deploy toolhive operator and associated MCP servers into OpenShift.

cc @jhrozek

[RoddieKieley]

@ChrisJBurns @jhrozek I recently started looking into this myself via my ootb-on-okd branch. In particular I generated a Dockerfile for thv-operator based on the Dockerfile for the Apache ActiveMQ Artemis broker operator we had been using. Also it follows the same pattern of using an entrypoint script. There's also an additional build task in the TaskFile.yml for the moment to build from the Dockerfile rather than modify the original.

Had you had ideas about how you thought it best to proceed in the context of the existing project setup? Not being that familiar with google ko myself this seemed like a quick and natural next step - i.e. to separate out the Dockerfile so that it can be customized directly depending on the need. Does that make sense or is there something simpler that can be done? This allows the operator to run in the test target env which is an OKD 4.19.z single node okd cluster.

[JAORMX]

@RoddieKieley are you struggling with the ko based container? Last I checked it ran fine on OpenShift. The main thing to tweak is the containers it spawns, for which we need to adjust the setup.

[JAORMX]

We had this thread before #342

[dmartinol]

I confirm that I could not deploy on Openshift with the documented Helm install.
And, once I workarounded it with a local controller (go run main.go), the MCPServer deployment failed with some file permission issues (I tried to solve by injecting the XDG_*** env vars with no success).

[JAORMX]

Ah, at some point we fixed it ao it wouldnt fail if it couldn't write to the filesystem. Seems like it regressed...

[RoddieKieley]

For me on my 4.19.z okd sno in an out of the box following of the deployment instructions that work no problem for a minikube deployment I saw the following in the event log:

and as well

That being said I haven't tried it this week directly from the existing configuration as those screenshots were from the 11th.

Will try and update further.

[RoddieKieley]

I got the latest source and deployed using the latest tags yesterday, 0.0.11 for the CRDs and 0.1.8 for the operator, and still encountered the operator failing to start on OKD out of the box. However in looking more closely I saw I could simply add the required values to the helm chart as per this commit. Which adds the required seccompProfile.type: RuntimeDefault to the podSecurityContext and removes the runAsUser: 1000 setting.

At this point the operator itself starts up successfully as per:

Shortly thereafter there is an issue related to the operand and the operator pod is restarted, however haven't looked into the details there yet.

Given that some kubernetes environments will have specific requirements such as the customized values here for OpenShift, is this something that should be mentioned in documentation? I'd imagine that for larger changes maybe separate reference values.yaml might be needed but this doesn't seem to be the case here.

[rdimitrov]

cc: @danbarr on the doc-related changes

[jhrozek]

@RoddieKieley

If I remember correctly, OCP can't run with an arbitrary user ID unless the SA is allowed to use the anyuid or privileged SCCs. And actually I don't see a reason for the operator and operand to run with an arbitrary ID, the randomly assigned IDs should work on OCP.

I think the approach about uncommenting the runAsUser in the operator deployment seems correct to me.

About the operand failing I suspect it's related to this code. What we could do is to check if the operator is running on OCP by checking for the presence of one of the proprietary OCP APIs, e.g. like this:

// Check for OpenShift-specific APIs
_, err := clientset.Discovery().ServerResourcesForGroupVersion("config.openshift.io/v1")
if err == nil {
    // Running on OpenShift
}

and then modify the code I linked above to not set UIDs.

[jhrozek]

@RoddieKieley btw I tried the OpenShift developer access last week but that didn't have enough privileges to install CRDs. If we could get access to a cluster, we'll be happy to help ourselves.

[jhrozek]

Actually one more side-node - there are MCP servers that require a particular UID. They are typically badly written (e.g. the original fetch MCP server requires to be running as root (!!!!!)). I'm not sure if "badly written MCPs" is something worth supporting but if yes, then the operator would have to be allowed to use the anyuid SCC. Maybe that could be a deployment option though.

[JAORMX]

That's why we ditched the fetch MCP in favor of gofetch which we wrote in such a way it doesn't require root.

[RoddieKieley]

I think the order of the day at the moment when it comes to MCP servers is flexibility. Ideally it would be great to enable everyone to run whatever MCP server they wanted, but in reality as you indicate tool use safety is a thing too given the seriously negative consequences that are possible; e.g. Replit AI went rogue, deleted a company's entire database

For the OCP detection, I'd be pretty sure that OKD should behave the same as OCP in this case. For the ActiveMQ Artemis Operator we detect OCP by looking for the availability of Routes. And OKD does indeed have Routes which are an OpenShift addition.

To be fair for a while installing OKD as per the single node instructions it wasn't completely smooth sailing, but I don't recall the specifics as I last clean installed some time ago.

Will have to take a look at the other options while also checking out the code pointed to above.

[jhrozek]

Ah, it seems like OKD came a long way in terms of usability of installation since the last time I looked. Let me check if we could deploy a small OKD cluster locally to team up on the work.