Refresh Azure.Sdk.Tools.Mock handler coverage to match live MCP tool list (#15852)#15854
Merged
Conversation
- New eng/scripts/Get-McpToolInventory.ps1 boots the live Azure.Sdk.Tools.Cli MCP server (via 'azsdk list -o json'), enumerates the IMockToolHandler implementations under Azure.Sdk.Tools.Mock, and reports the diff in three buckets: both / live-only / mock-only. - Cross-references mock-tier eval YAMLs under tools/azsdk-cli/Azure.Sdk.Tools.Vally/evals/ when present; gracefully no-ops when that folder hasn't landed yet (PR #15811). - '-CheckOnly' exits non-zero on (a) any stale handler that no longer maps to a live tool, or (b) any tool referenced by a mock-tier eval without a handler -- intended for the CI job tracked in #15829. - Documents the drift workflow in Azure.Sdk.Tools.Mock/README.md so a contributor flagged by the script knows how to add a handler. No stale handlers detected against the current live tool set.
github-actions
Bot
added
the
azsdk-cli
…om inventory script (#15852) - 13 new handler files covering 63 live tools that previously fell back to the default response (APIView, Codeowners, EngSys, GitHub, Package, Pipeline, ReleasePlan, TypeSpec, Verify, Core, Example). - Get-McpToolInventory.ps1: pure live-vs-mock parity (removes Vally eval cross-reference); -CheckOnly fails if either bucket is non-empty. - README: updated sync workflow to reflect parity-only check.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates Azure.Sdk.Tools.Mock to provide hand-written mock handlers for the Azure SDK Tools CLI MCP toolset and adds a PowerShell script to detect drift between the live tool inventory and the mock handler inventory, with accompanying documentation updates.
Changes:
- Added domain-grouped
IMockToolHandlerimplementations returning typedCommandResponsepayloads with realistic sample data. - Added
eng/scripts/Get-McpToolInventory.ps1to compare live MCP tools (azsdk list -o json) vs. mock handlerToolNamedeclarations and report drift. - Updated
Azure.Sdk.Tools.Mock/README.mdto document the sync workflow.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/azsdk-cli/Azure.Sdk.Tools.Mock/README.md | Documents the handler sync workflow and drift-check usage. |
| tools/azsdk-cli/Azure.Sdk.Tools.Mock/Handlers/Verify/VerifySetupHandler.cs | Adds a mock azsdk_verify_setup handler returning a successful typed response. |
| tools/azsdk-cli/Azure.Sdk.Tools.Mock/Handlers/TypeSpec/TypeSpecHandlers.cs | Adds mock handlers for TypeSpec-related tools with typed responses. |
| tools/azsdk-cli/Azure.Sdk.Tools.Mock/Handlers/ReleasePlan/ReleasePlanRemainingHandlers.cs | Adds mock handlers for remaining release-plan tools (some response types currently diverge from live). |
| tools/azsdk-cli/Azure.Sdk.Tools.Mock/Handlers/Pipeline/AnalyzeAndArtifactsHandlers.cs | Adds mock handlers for pipeline analysis and artifact listing tools. |
| tools/azsdk-cli/Azure.Sdk.Tools.Mock/Handlers/Package/PackageOperationHandlers.cs | Adds mock handlers for package operations (generate/build/test/check/pack/etc.). |
| tools/azsdk-cli/Azure.Sdk.Tools.Mock/Handlers/GitHub/GitHubHandlers.cs | Adds mock handlers for GitHub-related tools (PR/user details). |
| tools/azsdk-cli/Azure.Sdk.Tools.Mock/Handlers/Example/ExampleAndDebugHandlers.cs | Adds mock handlers for example/debug tools including deterministic failure paths. |
| tools/azsdk-cli/Azure.Sdk.Tools.Mock/Handlers/EngSys/EngSysHandlers.cs | Adds mock handlers for engsys log/test failure analysis and cleanup tools. |
| tools/azsdk-cli/Azure.Sdk.Tools.Mock/Handlers/Core/UpgradeHandler.cs | Adds a mock azsdk_upgrade handler returning an “already up to date” response. |
| tools/azsdk-cli/Azure.Sdk.Tools.Mock/Handlers/Config/ConfigHandlers.cs | Adds mock handlers for service label and CODEOWNERS tooling. |
| tools/azsdk-cli/Azure.Sdk.Tools.Mock/Handlers/APIView/ApiViewHandlers.cs | Adds mock handlers for APIView comment/review URL/copilot review tools. |
| eng/scripts/Get-McpToolInventory.ps1 | Adds a drift-detection script comparing live tool list vs. mock handler coverage. |
Addresses Copilot review on PR #15854: - azsdk_get_kpi_attestation_status: ReleaseWorkflowResponse -> ReleasePlanListResponse - azsdk_get_service_details_by_typespec_path: ReleaseWorkflowResponse -> ProductInfoResponse - azsdk_update_language_exclusion_justification: ReleaseWorkflowResponse -> DefaultCommandResponse
Per review discussion: the script only checked that an IMockToolHandler exists with the right ToolName; it could not detect handlers that exist but just return the placeholder DefaultCommandResponse. That blind spot makes the script of limited value. A unit test in Cli.Tests is a better fit for actual drift enforcement and is tracked as a follow-up. README updated to drop the script reference.
helen229
changed the title
samvaity
reviewed
Jun 3, 2026
4 tasks
praveenkuttappan
approved these changes
Jun 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Brings
Azure.Sdk.Tools.Mockback in sync with the liveAzure.Sdk.Tools.CliMCP tool list by addingIMockToolHandlerimplementations for the 63 tools that previously fell back to the generic{"message":"Success"}default response.Closes #15852.
Why
The mock auto-advertises every live tool via reflection over
SharedOptions.ToolsList, so the tool list never drifts. What does drift is handler coverage: tools without anIMockToolHandlerreturn the placeholder default. That's fine for routing tests but breaks chained-call eval scenarios (e.g.create_release_plan→ use returnedrelease_plan_idin the next call). Most of the live tool set had no handler — this PR brings coverage to 74/74.Changes
Added hand-written
IMockToolHandlerimplementations grouped by domain. Each handler returns the same typedCommandResponsesubtype as the live tool, populated with realistic sample data:Handlers/APIView/ApiViewHandlers.csHandlers/Config/ConfigHandlers.cs— service-label + codeownersHandlers/Core/UpgradeHandler.csHandlers/EngSys/EngSysHandlers.cs— log analysis, failed tests, cleanupHandlers/Example/ExampleAndDebugHandlers.cs— example services + hello-world-failHandlers/GitHub/GitHubHandlers.csHandlers/Package/PackageOperationHandlers.csHandlers/Pipeline/AnalyzeAndArtifactsHandlers.csHandlers/ReleasePlan/ReleasePlanRemainingHandlers.csHandlers/TypeSpec/TypeSpecHandlers.csHandlers/Verify/VerifySetupHandler.csUpdated
tools/azsdk-cli/Azure.Sdk.Tools.Mock/README.mdwith the workflow for adding a handler when a new MCP tool is introduced.Verification
dotnet build tools/azsdk-cli/Azure.Sdk.Tools.Mock— 0 errors (1 pre-existing CS9107 warning inMockToolRegistrations.cs, unrelated).Follow-ups (not in this PR)
Automated drift detection (live tool with no handler, or handler with only the placeholder default) — likely a unit test in
Azure.Sdk.Tools.Cli.Teststhat reflects over both sets and invokes each handler. Out of scope here; this PR just fixes the existing gap.