Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 13 additions & 19 deletions .github/workflows/github-remote-mcp-auth-test.lock.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

53 changes: 20 additions & 33 deletions .github/workflows/github-remote-mcp-auth-test.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ tools:
github:
mode: remote
toolsets: [repos, issues, discussions]
allowed: [get_repository, list_issues, issue_read]
allowed: [list_issues, issue_read]
timeout-minutes: 5
strict: true
imports:
Expand Down Expand Up @@ -187,37 +187,31 @@ jobs:
echo "- ❌ MCP tools/list: empty tool catalog" >> "$GITHUB_STEP_SUMMARY"
exit 1
fi
if ! jq -e '.result.tools[] | select(.name == "get_repository")' "$json_file" >/dev/null; then
echo 'MCP tools/list did not return the get_repository tool.'
if ! jq -e '.result.tools[] | select(.name == "list_issues")' "$json_file" >/dev/null; then
echo 'MCP tools/list did not return the list_issues tool.'
echo "available tools: $tool_names"
echo "- ❌ MCP tools/list: \`get_repository\` is unavailable (available tools: $tool_names)" >> "$GITHUB_STEP_SUMMARY"
echo "- ❌ MCP tools/list: \`list_issues\` is unavailable (available tools: $tool_names)" >> "$GITHUB_STEP_SUMMARY"
exit 1
fi
log_info "tools/list: \`get_repository\` is available"
echo "- ✅ MCP tools/list: $tool_count tools, including \`get_repository\`" >> "$GITHUB_STEP_SUMMARY"
log_info "tools/list: \`list_issues\` is available"
echo "- ✅ MCP tools/list: $tool_count tools, including \`list_issues\`" >> "$GITHUB_STEP_SUMMARY"

repository_owner="${GITHUB_REPOSITORY%%/*}"
repository_name="${GITHUB_REPOSITORY#*/}"
call_payload="$(jq -nc \
--arg owner "$repository_owner" \
--arg repo "$repository_name" \
'{jsonrpc:"2.0",id:4,method:"tools/call",params:{name:"get_repository",arguments:{owner:$owner,repo:$repo}}}')"
'{jsonrpc:"2.0",id:4,method:"tools/call",params:{name:"list_issues",arguments:{owner:$owner,repo:$repo,state:"OPEN",perPage:1}}}')"
call_code="$(mcp_post "$call_payload" "$response_headers_file" "${session_args[@]}")"
assert_success_response "MCP get_repository" "$call_code"
assert_success_response "MCP list_issues" "$call_code"
if jq -e '.result.isError == true' "$json_file" >/dev/null; then
tool_error="$(jq -r '([.result.content[]?.text] | join(" "))[0:200]' "$json_file")"
echo "MCP get_repository returned a tool error: $tool_error"
echo "- ❌ MCP get_repository: tool error \`$tool_error\`" >> "$GITHUB_STEP_SUMMARY"
echo "MCP list_issues returned a tool error: $tool_error"
echo "- ❌ MCP list_issues: tool error \`$tool_error\`" >> "$GITHUB_STEP_SUMMARY"
exit 1
fi
repository_content="$(jq -r '[.result.content[]? | select(.type == "text") | .text] | join(" ")' "$json_file")"
if [[ "$repository_content" != *"$GITHUB_REPOSITORY"* ]]; then
echo "MCP get_repository did not return the requested repository."
echo "- ❌ MCP get_repository: result did not identify \`$GITHUB_REPOSITORY\`" >> "$GITHUB_STEP_SUMMARY"
exit 1
fi
log_info "get_repository: retrieved $GITHUB_REPOSITORY"
echo "- ✅ MCP get_repository: retrieved \`$GITHUB_REPOSITORY\`" >> "$GITHUB_STEP_SUMMARY"
log_info "list_issues: retrieved open issues from $GITHUB_REPOSITORY"
echo "- ✅ MCP list_issues: retrieved open issues from \`$GITHUB_REPOSITORY\`" >> "$GITHUB_STEP_SUMMARY"
Comment on lines +213 to +214
echo "Raw GitHub remote MCP handshake succeeded with $tool_count tools available."
features:
gh-aw-detection: true
Expand All @@ -236,25 +230,18 @@ Test that the GitHub remote MCP server can authenticate and access GitHub API wi

### Test Procedure

1. **Verify Tool Availability**: FIRST, check that GitHub MCP tools are accessible
- Try to use the `get_repository` tool to get basic info about ${{ github.repository }}
1. **Verify Tool Availability**: FIRST, check that the `list_issues` GitHub MCP tool is accessible
- Try to use the `list_issues` tool to list open issues in ${{ github.repository }}
- This is a simple, read-only operation that should work if MCP tools are properly loaded
- **If this fails with errors like "tool not found", "unknown tool", or "capability not available":**
- The MCP toolsets are NOT loaded in the runner
- Report this using the `missing_tool` safe output with:
- Tool: "GitHub MCP tools (list_issues, get_repository)"
- Tool: "GitHub MCP tool (list_issues)"
- Reason: "MCP toolsets unavailable in runner - tools not loaded"
- Alternatives: "Check MCP configuration, verify remote mode is accessible, or use local mode fallback"
- **Do NOT proceed to step 2** - the test has failed due to missing tools

2. **List Open Issues**: If `get_repository` succeeded, now test with `list_issues`
- Use the GitHub MCP server to list 3 open issues in the repository ${{ github.repository }}
- Use the `list_issues` tool
- Filter for `state: OPEN`
- Limit to 3 results
- Extract issue numbers and titles
- The test has failed due to missing tools

3. **Verify Authentication**:
2. **Verify Authentication**:
- If the MCP tools successfully return data, authentication is working correctly
- If the MCP tools fail with authentication errors (401, 403, "unauthorized", or "invalid session"), authentication has failed
- **IMPORTANT**: Do NOT fall back to using `gh api` directly - this test must use the MCP server
Expand Down Expand Up @@ -292,7 +279,7 @@ If the test fails, create a discussion using safe-outputs based on the failure t
**MCP Tools Not Loaded**: The GitHub MCP toolsets (repos, issues, discussions) are not being loaded in the runner. This prevents the agent from accessing GitHub data through MCP.

### Impact
- Agent cannot use `list_issues`, `get_repository`, or other GitHub MCP tools
- Agent cannot use `list_issues` or other GitHub MCP tools
- Workflow cannot complete its authentication test
- This is a configuration/infrastructure issue, not an authentication issue

Expand All @@ -302,7 +289,7 @@ If the test fails, create a discussion using safe-outputs based on the failure t
github:
mode: remote
toolsets: [repos, issues, discussions]
allowed: [get_repository, list_issues, issue_read]
allowed: [list_issues, issue_read]
```

### Remediation Steps
Expand Down Expand Up @@ -370,7 +357,7 @@ If the test fails, create a discussion using safe-outputs based on the failure t
- Missing tools = Configuration/infrastructure issue
- Auth errors = Token/permissions issue
- **Use missing_tool safe output**: When tools aren't available, report it properly before creating a discussion
- **Check for MCP tools FIRST**: Start with a simple `get_repository` call to verify tools are loaded
- **Check for MCP tools FIRST**: Start with a simple `list_issues` call to verify tools are loaded
- **Include error details**: If authentication fails, include the exact error message from the MCP tool
- **Provide actionable remediation**: Include specific steps to resolve the detected issue type
- **Auto-cleanup**: Old test discussions will be automatically closed by the close-older-discussions setting
Expand Down
Loading