WIP: openshift-mcp-server: add mcpchecker eval job for Google Gemini - #78018
WIP: openshift-mcp-server: add mcpchecker eval job for Google Gemini#78018matzew wants to merge 1 commit into
Conversation
Add an optional, always-run presubmit job that runs mcpchecker evaluations using Google Gemini against the MCP server. The job will fail until Vault credentials are wired in a follow-up PR. Signed-off-by: Matthias Wessendorf <mwessend@redhat.com>
openshift-ci
Bot
added
the
do-not-merge/work-in-progress
WalkthroughTwo CI configuration files are updated to add a new optional test workflow step for evaluating the MCP server using mcpchecker with Google Gemini-based Kubernetes assessment. The workflow includes building, installing mcpchecker, running the server, executing the evaluation, and cleanup. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes 🚥 Pre-merge checks | ✅ 10✅ Passed checks (10 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: matzew The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
|
[REHEARSALNOTIFIER]
Prior to this PR being merged, you will need to either run and acknowledge or opt to skip these rehearsals. Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In
`@ci-operator/config/openshift/openshift-mcp-server/openshift-openshift-mcp-server-main.yaml`:
- Around line 50-62: The CI invokes non-existent Makefile targets make
run-server and make stop-server; replace them with the repository's actual
pattern by either calling the existing local-env-setup/local-env-teardown
targets or directly starting/stopping the built server binary
(kubernetes-mcp-server) with the correct flags (e.g., --port and --config) used
elsewhere; update the steps that reference make run-server TOOLSETS=core,config
and make stop-server to instead call local-env-setup/local-env-teardown or
run/kill the kubernetes-mcp-server process so the job no longer fails with "No
rule to make target 'run-server'".
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 47b53008-7a8a-449d-9d67-101387cd418a
📒 Files selected for processing (2)
ci-operator/config/openshift/openshift-mcp-server/openshift-openshift-mcp-server-main.yamlci-operator/jobs/openshift/openshift-mcp-server/openshift-openshift-mcp-server-main-presubmits.yaml
| # Start MCP server in background (no cluster needed for this step) | ||
| make run-server TOOLSETS=core,config | ||
|
|
||
| # Run mcpchecker evaluation with Google Gemini agent | ||
| # NOTE: This will fail until Vault credentials are wired (follow-up PR) | ||
| _output/tools/bin/mcpchecker check \ | ||
| evals/core-eval-testing/builtin-google/eval-kubernetes.yaml \ | ||
| --label-selector suite=kubernetes \ | ||
| --output json \ | ||
| || true | ||
|
|
||
| # Cleanup | ||
| make stop-server || true |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Verify whether run-server / stop-server targets in the openshift-mcp-server Makefile
# detach the server (e.g., via & or nohup) or run in foreground.
gh api repos/openshift/openshift-mcp-server/contents/Makefile --jq '.content' \
| base64 -d \
| awk '/^run-server:|^stop-server:|^mcpchecker:/{flag=1} flag{print} /^$/{flag=0}'Repository: openshift/release
Length of output: 43
🏁 Script executed:
# Search for Makefile in the repository that was configured
# First, check if there's an openshift-mcp-server reference or Makefile in the release repo
find . -name "Makefile" -type f | grep -E "(mcp|server)" | head -20Repository: openshift/release
Length of output: 43
🏁 Script executed:
# Also search the YAML file's directory for context about how the server is invoked
cd ci-operator/config/openshift/openshift-mcp-server/ 2>/dev/null && ls -la && head -100 openshift-openshift-mcp-server-main.yamlRepository: openshift/release
Length of output: 2781
🌐 Web query:
openshift-mcp-server Makefile run-server target backgrounding process
💡 Result:
The openshift-mcp-server repository's main Makefile does not contain a 'run-server' target. The Makefile provides targets like 'build', 'test', 'clean', 'local-env-setup', etc., for building the kubernetes-mcp-server binary and setting up local development (e.g., Kind cluster). Running the server is done directly with the built binary: ./kubernetes-mcp-server [--port 8008] [--config _output/config.toml]. No backgrounding (&, nohup) is used in the Makefile; servers are typically run in the foreground for development or as container ENTRYPOINT/CMD in Dockerfiles (e.g., CMD ["--port", "8080"]). For OpenShift builds, Dockerfile.ocp uses 'make -f Makefile-ocp.mk build-ocp', but no run-server target is evident. In container deployments on OpenShift, processes run in foreground as PID 1.
Citations:
- 1: https://github.com/openshift/openshift-mcp-server/blob/main/Makefile
- 2: https://github.com/openshift/openshift-mcp-server/blob/main/Dockerfile.ocp
- 3: https://github.com/openshift/openshift-mcp-server/blob/main/Dockerfile
- 4: https://github.com/openshift/openshift-mcp-server
🏁 Script executed:
# Search the openshift/release repo for references to run-server and stop-server
rg "run-server|stop-server" --type yaml -B 2 -A 2 | head -50Repository: openshift/release
Length of output: 1215
🏁 Script executed:
# Check if there are multiple Makefiles or build files
git ls-files | grep -iE "(makefile|make)" | head -20Repository: openshift/release
Length of output: 699
🌐 Web query:
openshift-mcp-server Makefile "run-server" OR "stop-server" target definition
💡 Result:
No "run-server" or "stop-server" targets are defined in the Makefile of the openshift-mcp-server repository (https://github.com/openshift/openshift-mcp-server). The Makefile includes standard targets like help, clean, build, test, format, tidy, local-env-setup, and local-env-teardown, but nothing matching "run-server" or "stop-server". The server is run directly via the built binary, e.g., ./kubernetes-mcp-server --port 8008 --config _output/config.toml as shown in local-env-setup output. A separate Makefile-ocp.mk exists for OCP builds, referenced in Dockerfile.ocp, but no evidence of those targets there either. "openshift-mcp-server" is a recent (2025) fork for Model Context Protocol (MCP), unrelated to legacy Machine Config Pool server.
Citations:
- 1: https://github.com/openshift/openshift-mcp-server/blob/main/Makefile
- 2: https://github.com/openshift/openshift-mcp-server/blob/main/Makefile-ocp.mk
- 3: https://linkedin.com/in/rahul-ruhela-4b4a6b52
- 4: https://github.com/openshift/origin/blob/main/Makefile
make run-server and make stop-server targets do not exist in the openshift-mcp-server Makefile.
The CI job invokes make run-server TOOLSETS=core,config and make stop-server, but these targets are not defined in the repository's Makefile or Makefile-ocp.mk. The Makefile contains standard targets like build, test, clean, local-env-setup, and local-env-teardown, but nothing matching "run-server" or "stop-server". The server is run directly via the built binary (e.g., ./kubernetes-mcp-server --port 8008 --config _output/config.toml). This job will fail immediately with "No rule to make target 'run-server'". Confirm the correct invocation pattern with the repository maintainers and update the commands to match the actual Makefile or direct binary invocation.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In
`@ci-operator/config/openshift/openshift-mcp-server/openshift-openshift-mcp-server-main.yaml`
around lines 50 - 62, The CI invokes non-existent Makefile targets make
run-server and make stop-server; replace them with the repository's actual
pattern by either calling the existing local-env-setup/local-env-teardown
targets or directly starting/stopping the built server binary
(kubernetes-mcp-server) with the correct flags (e.g., --port and --config) used
elsewhere; update the steps that reference make run-server TOOLSETS=core,config
and make stop-server to instead call local-env-setup/local-env-teardown or
run/kill the kubernetes-mcp-server process so the job no longer fails with "No
rule to make target 'run-server'".
|
@matzew: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
/close |
openshift-ci
Bot
added
the
needs-rebase
|
PR needs rebase. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
@matzew: Closed this PR. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
3 participants
Add an optional, always-run presubmit job that runs mcpchecker evaluations using Google Gemini against the MCP server. The job will fail until Vault credentials are wired in a follow-up PR.
Summary by CodeRabbit
/test mcpchecker-eval-googlecommand in pull requests