Add round-trip validation tests for MCP output schemas

[Copilot]

Current schema generation tests only verify structural correctness but don't validate that generated schemas can actually validate real output data.

Changes

Added TestGeneratedSchemasValidateRealOutput in pkg/cli/mcp_schema_test.go with three subtests:

• LogsData: Validates schema against log summary, workflow runs, token usage
• AuditData: Validates schema against overview, metrics, downloaded files
• WorkflowStatus: Validates schema against workflow state, engine info, run status

Implementation

Each test follows the pattern:

schema, _ := GenerateOutputSchema[LogsData]()
resolved, _ := schema.Resolve(&jsonschema.ResolveOptions{})

data := LogsData{
    Summary: LogsSummary{TotalRuns: 5, TotalTokens: 15000, ...},
    Runs: []RunData{{WorkflowName: "test-workflow", ...}},
}

jsonBytes, _ := json.Marshal(data)
var jsonValue map[string]any
json.Unmarshal(jsonBytes, &jsonValue)

err := resolved.Validate(jsonValue)  // Confirms schema works with real data

This ensures schemas are functionally correct for validating actual MCP tool outputs, not just structurally well-formed.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• go.googlesource.com
  • Triggering command: /update-job-proxy /update-job-proxy -s 172.18.0.0/16 -j DROP itest/dist/config.cjs ./node_modules/fflate/lib/workerGOMOD g/workflow/js/node_modules/.bin/GOMODCACHE pse_vars=true js.tmp ode node ache�� e.test.cjs -c dead_code=true,unuGOINSECURE (dns block)
  • Triggering command: /update-job-proxy /update-job-proxy -o br-e8b0804ce3ac -j DOCKER-ISOLATION-STAGE-2 /etc/ssl/certs/4bfab552.0 sed e/git $a\ /usr/share/ca-cecat-file /usr/bin/sed e/git -e 234a7ecf75459b3ee587660c6d179152b68ea0/shallow.lock sed e/git-remote-https C1.pem /usr/share/ca-ce--wait 9.0 e/git-remote-httfilter (dns block)
• go.uber.org
  • Triggering command: /update-job-proxy /update-job-proxy -s 172.18.0.0/16 -j DROP itest/dist/config.cjs ./node_modules/fflate/lib/workerGOMOD g/workflow/js/node_modules/.bin/GOMODCACHE pse_vars=true js.tmp ode node ache�� e.test.cjs -c dead_code=true,unuGOINSECURE (dns block)
  • Triggering command: /update-job-proxy /update-job-proxy -o br-e8b0804ce3ac -j DOCKER-ISOLATION-STAGE-2 /etc/ssl/certs/4bfab552.0 sed e/git $a\ /usr/share/ca-cecat-file /usr/bin/sed e/git -e 234a7ecf75459b3ee587660c6d179152b68ea0/shallow.lock sed e/git-remote-https C1.pem /usr/share/ca-ce--wait 9.0 e/git-remote-httfilter (dns block)
• go.yaml.in
  • Triggering command: /update-job-proxy /update-job-proxy -s 172.18.0.0/16 -j DROP itest/dist/config.cjs ./node_modules/fflate/lib/workerGOMOD g/workflow/js/node_modules/.bin/GOMODCACHE pse_vars=true js.tmp ode node ache�� e.test.cjs -c dead_code=true,unuGOINSECURE (dns block)
  • Triggering command: /update-job-proxy /update-job-proxy -o br-e8b0804ce3ac -j DOCKER-ISOLATION-STAGE-2 /etc/ssl/certs/4bfab552.0 sed e/git $a\ /usr/share/ca-cecat-file /usr/bin/sed e/git -e 234a7ecf75459b3ee587660c6d179152b68ea0/shallow.lock sed e/git-remote-https C1.pem /usr/share/ca-ce--wait 9.0 e/git-remote-httfilter (dns block)
• gopkg.in
  • Triggering command: /update-job-proxy /update-job-proxy -s 172.18.0.0/16 -j DROP itest/dist/config.cjs ./node_modules/fflate/lib/workerGOMOD g/workflow/js/node_modules/.bin/GOMODCACHE pse_vars=true js.tmp ode node ache�� e.test.cjs -c dead_code=true,unuGOINSECURE (dns block)
  • Triggering command: /update-job-proxy /update-job-proxy -o br-e8b0804ce3ac -j DOCKER-ISOLATION-STAGE-2 /etc/ssl/certs/4bfab552.0 sed e/git $a\ /usr/share/ca-cecat-file /usr/bin/sed e/git -e 234a7ecf75459b3ee587660c6d179152b68ea0/shallow.lock sed e/git-remote-https C1.pem /usr/share/ca-ce--wait 9.0 e/git-remote-httfilter (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Original prompt

---

This section details on the original issue you should resolve

<issue_title>[plan] Add schema validation test for round-trip testing</issue_title>
<issue_description>## Objective

Add comprehensive test to ensure generated schemas can validate actual output data (round-trip validation).

Context

Current tests verify schema generation but don't validate that schemas can actually validate real output data. This ensures schemas are not just structurally correct but functionally useful.

Implementation

Add new test function TestGeneratedSchemasValidateRealOutput in pkg/cli/mcp_schema_test.go:

func TestGeneratedSchemasValidateRealOutput(t *testing.T) {
    // Test LogsData
    schema, err := GenerateOutputSchema[LogsData]()
    require.NoError(t, err)

    resolved, err := schema.Resolve(&jsonschema.ResolveOptions{})
    require.NoError(t, err)

    data := LogsData{
        Summary: "Test summary",
        Runs: []RunData{{WorkflowName: "test-workflow"}},
    }

    jsonBytes, _ := json.Marshal(data)
    var jsonValue map[string]any
    json.Unmarshal(jsonBytes, &jsonValue)

    err = resolved.Validate(jsonValue)
    require.NoError(t, err, "Schema should validate real LogsData output")

    // Add similar tests for AuditData and WorkflowStatus
}

Files to Modify

• pkg/cli/mcp_schema_test.go - Add new test function

Acceptance Criteria

• Test validates LogsData schema against real data
• Test validates AuditData schema against real data
• Test validates WorkflowStatus schema against real data
• Tests use schema.Resolve() and Validate() correctly
• All tests pass
  Related to [plan] Upgrade and enhance google/jsonschema-go integration #6830

AI generated by Plan Command for discussion #6818

Comments on the Issue (you are @copilot in this section)

• Fixes [plan] Add schema validation test for round-trip testing #6833

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.