Repository navigation
test(compliance): MCP 2025-11-25 protocol compliance harness - #4301
Merged
Merged
Conversation
Build a FastMCP-based test harness that probes ContextForge against the MCP 2025-11-25 specification across three targets: reference stdio, gateway proxy (federated), and gateway virtual server. Includes a new reference server (mcp-servers/python/compliance_reference_server) with tools/resources/prompts covering lifecycle, transport, pagination, notifications, subscriptions, roots, sampling, elicitation, and logging. Key additions: - tests/protocol_compliance/: parametrized test suite with XPASS-capture, drift detection across targets, and gap-tracking workflow - scripts/compliance_matrix.py: orchestrator that runs the suite across engine modes (python/shadow/edge/full) and aggregates results - Runtime-mutable MCP mode tests covering shadow<->edge flip, boot-mode rails, publish/audit response contract, and data-plane runtime header - OAuth 2.1 / Authorization tests against Keycloak (tier-gated skip) - Security Best Practices probes (origin validation, bind address, confused-deputy) - Rewrite tests/e2e/test_mcp_cli_protocol.py as test_mcp_protocol_e2e.py using FastMCP Client directly (no external CLI deps) - Makefile: test-protocol-compliance* targets; testing-up folds in sso profile so Keycloak comes up with the gateway stack - pre-commit: scope name-tests-test hook to tests/unit/ (integration, e2e, and compliance suites legitimately host non-test Python modules) Closes #4273 (runtime-mutable MCP mode observability coverage). Signed-off-by: Jonathan Springer <jps@s390x.com>
Move loose non-test Python modules into conventional helpers/ subdirs so the test-naming hook can exclude a single pattern instead of enumerating each file. Moves: - tests/e2e/mcp_test_helpers.py -> tests/e2e/helpers/mcp_test_helpers.py - tests/protocol_compliance/_helpers.py -> tests/protocol_compliance/helpers/compliance.py - tests/protocol_compliance/_drift.py -> tests/protocol_compliance/helpers/drift.py Updates all import sites across tests/e2e, tests/e2e_rust, and tests/protocol_compliance. Config cleanup: - name-tests-test now excludes tests/*/(helpers|fixtures|targets|pages)/ as a single verbose-regex alternative instead of file-by-file. - Top-level pre-commit exclude and detect-secrets excludes converted to verbose-regex with inline comments. - Dropped non-existent excludes (scripts/sign_image.sh, scripts/zap, sonar-project.properties) and the redundant .secrets.baseline entry (detect-secrets skips its own baseline automatically). - detect-secrets hook in pre-commit now carries its own exclude list, kept in sync with the Makefile invocation. Signed-off-by: Jonathan Springer <jps@s390x.com>
…lient The local Keycloak dev realm disables the client_credentials grant, so compliance tests that exercise OAuth 2.1 token issuance against the preconfigured `mcp-gateway` client fail with `unauthorized_client: Client not enabled to retrieve service account`. Flip `serviceAccountsEnabled` to `true` so fresh `docker compose` volume imports come up with the grant available. Existing stacks can apply the same change via the Keycloak Admin API against the running realm; the stored volume overrides the import file once the realm exists, which is why the change has to land here to stick across rebuilds. Signed-off-by: Jonathan Springer <jps@s390x.com>
…oral coverage, type design, stream attribution
Bundles the fixes surfaced by running the pr-review-toolkit agents
(code-reviewer, pr-test-analyzer, comment-analyzer,
silent-failure-hunter, type-design-analyzer) plus two iterations of
Codex stop-review feedback.
Silent-failure fixes:
- _parse_junit / _run_slice / stale-junit unlink /
_wait_data_plane_converges return-value check close the
"0/0/0/0 row" silent path in the matrix orchestrator.
- Reference server stdout+stderr captured to a session tmp log; tail
dumped on readiness timeout instead of swallowed.
- flip_runtime_mode / _delete_if_exists / _wait_for_federation_sync
/ _probe_state / Keycloak token fetch surface last-status
diagnostics instead of collapsing to None.
- conftest fixture-resolution narrows catch-all so ImportError /
NameError / SyntaxError from a broken fixture definition
propagate; only runtime unreachability skips.
- pytest_runtest_logreport wraps sidecar write in try/except so an
unwriteable log doesn't poison the run.
Type design:
- ComplianceTarget: __init_subclass__ enforces name and
supported_transports on every concrete subclass; base client()
validates transport support and dispatches to abstract
_open_client(); per-subclass boilerplate deleted.
- SliceResult: stored `ran` field removed (derive from
skip_reason); __post_init__ rejects negative counts and the
inconsistent "skipped slice with non-zero counts" state.
- ReferenceUpstream: frozen, no raw Popen field — process handle
stays in fixture closure so tests can't race teardown.
- KeycloakConfig: token_endpoint is now a property; __repr__
redacts client_secret.
Stream attribution:
- MCP 2025-11-25 § Listening for Messages is explicit:
server→client *requests* (roots/list, sampling/createMessage,
elicitation/create) MUST NOT ride the standalone GET /mcp/
stream; they travel on the POST-correlated stream. GAP-002..005
and seven xfail reasons wrongly blamed #4205 (standalone-stream
closure). Rewrote the Why sections and reasons to point at
POST-correlated-stream relay instead.
- GAP-011 (resource-subscription updates) is the only gap that
legitimately points at #4205.
- Added a canonical "Stream-attribution note" to COMPLIANCE_GAPS.md.
- Filed GAP-011 (resource subscriptions) and GAP-012 (gateway
returns -32000 where spec requires -32601).
Behavioral coverage (6 new tests + 4 reference-server witnesses):
- notifications/cancelled end-to-end delivery.
- logging/setLevel filter-threshold round-trip.
- JSON-RPC reserved codes -32700 / -32601 / -32602.
- tools/call argument-type validation (narrowed to McpError).
- initialize-twice lifecycle rejection.
- MCP-Protocol-Version mismatch handling.
- Reference-server side-effect witnesses for mutate_resource_list /
mutate_prompt_list / bump_subscribable notifications and the
subscribe/unsubscribe handler round-trip.
Each raw-httpx test fails on 5xx and accepts 4xx as HTTP-level
rejection; no false-green paths where a server crash would read as
compliance.
Bug fixes:
- Reference server: mutator tools now use a dedicated
_mutation_counter to prevent ephemeral_0 name-collisions.
- test_drift._collect narrows to pytest.skip.Exception; other probe
errors are recorded so drift diffs show the root cause.
- Makefile PYTEST_IGNORE excludes tests/protocol_compliance.
Matrix rust_full slice: 58 passed / 0 failed / 40 xfailed / 0 xpassed
/ 11 skipped (was 53/0/34/0/11). Zero unexpected failures.
Signed-off-by: Jonathan Springer <jps@s390x.com>
Test defaulted its JWT secret to "my-test-key" (11 chars) when JWT_SECRET_KEY was unset; the gateway's validator requires ≥32 chars so every admin-API call the fixtures make would fail with 401 "Invalid authentication credentials" on default deployments. Import JWT_SECRET from helpers.mcp_test_helpers — the 40-char default the rest of the e2e suite already shares — so this test self-configures without requiring the operator to export JWT_SECRET_KEY. Overriding via the env var still works. Signed-off-by: Jonathan Springer <jps@s390x.com>
jonpspri
requested review from
brian-hussey,
crivetimihai,
dawid-nowak,
dima-zakharov,
kevalmahajan,
lucarlig and
madhav165
as code owners
April 19, 2026 08:54
Dependency ReviewThe following issues were found:
Vulnerabilitiesuv.lock
Only included vulnerabilities with severity moderate or higher. License Issuesmcp-servers/python/compliance_reference_server/pyproject.toml
uv.lock
OpenSSF ScorecardScorecard details
Scanned Files
|
… lite config The main pre-commit config already excludes these POM-style support directories, but CI runs the lite config (Makefile:4175 — `pre-commit run --config .pre-commit-lite.yaml`) which was a near-duplicate that hadn't been kept in sync. Add `targets` to its excluded-dir alternation so the four compliance-harness target modules stop tripping the hook in CI. Signed-off-by: Jonathan Springer <jps@s390x.com>
…ckage override authlib is a transitive dep of fastmcp, which the compliance harness uses as a test client — both live in the dev dependency group. mcpgateway itself imports nothing from authlib, so the constraint goes in `[dependency-groups] dev` rather than main project deps. Release 1.7.0 (2026-04-18) ships security fixes but was blocked by the repo-wide `exclude-newer = "10 days"` newness filter, leaving the lockfile on 1.6.9. Adds an `exclude-newer-package` entry for authlib (2026-04-19) so the filter allows the new release. Signed-off-by: Jonathan Springer <jps@s390x.com>
jonpspri
force-pushed
the
jps-mcp-compliance
branch
from
April 19, 2026 09:04
ca8f035 to
7de00e8
Compare
This was referenced Apr 19, 2026
dawid-nowak
reviewed
Apr 19, 2026
|
|
||
|
|
||
| def test_runtime_mode_state_is_readable(runtime_mode_state: dict) -> None: | ||
| """GET /admin/runtime/mcp-mode returns a well-shaped state payload.""" |
Collaborator
There was a problem hiding this comment.
Out of interest, how is this related to MCP compliance ?
gcgoncalves
pushed a commit
that referenced
this pull request
Apr 23, 2026
* test(compliance): add MCP 2025-11-25 protocol compliance harness Build a FastMCP-based test harness that probes ContextForge against the MCP 2025-11-25 specification across three targets: reference stdio, gateway proxy (federated), and gateway virtual server. Includes a new reference server (mcp-servers/python/compliance_reference_server) with tools/resources/prompts covering lifecycle, transport, pagination, notifications, subscriptions, roots, sampling, elicitation, and logging. Key additions: - tests/protocol_compliance/: parametrized test suite with XPASS-capture, drift detection across targets, and gap-tracking workflow - scripts/compliance_matrix.py: orchestrator that runs the suite across engine modes (python/shadow/edge/full) and aggregates results - Runtime-mutable MCP mode tests covering shadow<->edge flip, boot-mode rails, publish/audit response contract, and data-plane runtime header - OAuth 2.1 / Authorization tests against Keycloak (tier-gated skip) - Security Best Practices probes (origin validation, bind address, confused-deputy) - Rewrite tests/e2e/test_mcp_cli_protocol.py as test_mcp_protocol_e2e.py using FastMCP Client directly (no external CLI deps) - Makefile: test-protocol-compliance* targets; testing-up folds in sso profile so Keycloak comes up with the gateway stack - pre-commit: scope name-tests-test hook to tests/unit/ (integration, e2e, and compliance suites legitimately host non-test Python modules) Closes #4273 (runtime-mutable MCP mode observability coverage). Signed-off-by: Jonathan Springer <jps@s390x.com> * refactor(tests): group test helpers under tests/*/helpers/ subdirs Move loose non-test Python modules into conventional helpers/ subdirs so the test-naming hook can exclude a single pattern instead of enumerating each file. Moves: - tests/e2e/mcp_test_helpers.py -> tests/e2e/helpers/mcp_test_helpers.py - tests/protocol_compliance/_helpers.py -> tests/protocol_compliance/helpers/compliance.py - tests/protocol_compliance/_drift.py -> tests/protocol_compliance/helpers/drift.py Updates all import sites across tests/e2e, tests/e2e_rust, and tests/protocol_compliance. Config cleanup: - name-tests-test now excludes tests/*/(helpers|fixtures|targets|pages)/ as a single verbose-regex alternative instead of file-by-file. - Top-level pre-commit exclude and detect-secrets excludes converted to verbose-regex with inline comments. - Dropped non-existent excludes (scripts/sign_image.sh, scripts/zap, sonar-project.properties) and the redundant .secrets.baseline entry (detect-secrets skips its own baseline automatically). - detect-secrets hook in pre-commit now carries its own exclude list, kept in sync with the Makefile invocation. Signed-off-by: Jonathan Springer <jps@s390x.com> * fix(keycloak): enable service-accounts on the mcp-gateway dev realm client The local Keycloak dev realm disables the client_credentials grant, so compliance tests that exercise OAuth 2.1 token issuance against the preconfigured `mcp-gateway` client fail with `unauthorized_client: Client not enabled to retrieve service account`. Flip `serviceAccountsEnabled` to `true` so fresh `docker compose` volume imports come up with the grant available. Existing stacks can apply the same change via the Keycloak Admin API against the running realm; the stored volume overrides the import file once the realm exists, which is why the change has to land here to stick across rebuilds. Signed-off-by: Jonathan Springer <jps@s390x.com> * test(compliance): PR-review follow-ups — silent-failure fixes, behavioral coverage, type design, stream attribution Bundles the fixes surfaced by running the pr-review-toolkit agents (code-reviewer, pr-test-analyzer, comment-analyzer, silent-failure-hunter, type-design-analyzer) plus two iterations of Codex stop-review feedback. Silent-failure fixes: - _parse_junit / _run_slice / stale-junit unlink / _wait_data_plane_converges return-value check close the "0/0/0/0 row" silent path in the matrix orchestrator. - Reference server stdout+stderr captured to a session tmp log; tail dumped on readiness timeout instead of swallowed. - flip_runtime_mode / _delete_if_exists / _wait_for_federation_sync / _probe_state / Keycloak token fetch surface last-status diagnostics instead of collapsing to None. - conftest fixture-resolution narrows catch-all so ImportError / NameError / SyntaxError from a broken fixture definition propagate; only runtime unreachability skips. - pytest_runtest_logreport wraps sidecar write in try/except so an unwriteable log doesn't poison the run. Type design: - ComplianceTarget: __init_subclass__ enforces name and supported_transports on every concrete subclass; base client() validates transport support and dispatches to abstract _open_client(); per-subclass boilerplate deleted. - SliceResult: stored `ran` field removed (derive from skip_reason); __post_init__ rejects negative counts and the inconsistent "skipped slice with non-zero counts" state. - ReferenceUpstream: frozen, no raw Popen field — process handle stays in fixture closure so tests can't race teardown. - KeycloakConfig: token_endpoint is now a property; __repr__ redacts client_secret. Stream attribution: - MCP 2025-11-25 § Listening for Messages is explicit: server→client *requests* (roots/list, sampling/createMessage, elicitation/create) MUST NOT ride the standalone GET /mcp/ stream; they travel on the POST-correlated stream. GAP-002..005 and seven xfail reasons wrongly blamed #4205 (standalone-stream closure). Rewrote the Why sections and reasons to point at POST-correlated-stream relay instead. - GAP-011 (resource-subscription updates) is the only gap that legitimately points at #4205. - Added a canonical "Stream-attribution note" to COMPLIANCE_GAPS.md. - Filed GAP-011 (resource subscriptions) and GAP-012 (gateway returns -32000 where spec requires -32601). Behavioral coverage (6 new tests + 4 reference-server witnesses): - notifications/cancelled end-to-end delivery. - logging/setLevel filter-threshold round-trip. - JSON-RPC reserved codes -32700 / -32601 / -32602. - tools/call argument-type validation (narrowed to McpError). - initialize-twice lifecycle rejection. - MCP-Protocol-Version mismatch handling. - Reference-server side-effect witnesses for mutate_resource_list / mutate_prompt_list / bump_subscribable notifications and the subscribe/unsubscribe handler round-trip. Each raw-httpx test fails on 5xx and accepts 4xx as HTTP-level rejection; no false-green paths where a server crash would read as compliance. Bug fixes: - Reference server: mutator tools now use a dedicated _mutation_counter to prevent ephemeral_0 name-collisions. - test_drift._collect narrows to pytest.skip.Exception; other probe errors are recorded so drift diffs show the root cause. - Makefile PYTEST_IGNORE excludes tests/protocol_compliance. Matrix rust_full slice: 58 passed / 0 failed / 40 xfailed / 0 xpassed / 11 skipped (was 53/0/34/0/11). Zero unexpected failures. Signed-off-by: Jonathan Springer <jps@s390x.com> * fix(e2e): align test_oauth_jwks_e2e JWT secret with gateway minimum Test defaulted its JWT secret to "my-test-key" (11 chars) when JWT_SECRET_KEY was unset; the gateway's validator requires ≥32 chars so every admin-API call the fixtures make would fail with 401 "Invalid authentication credentials" on default deployments. Import JWT_SECRET from helpers.mcp_test_helpers — the 40-char default the rest of the e2e suite already shares — so this test self-configures without requiring the operator to export JWT_SECRET_KEY. Overriding via the env var still works. Signed-off-by: Jonathan Springer <jps@s390x.com> * ci(pre-commit): exclude tests/**/targets/ from name-tests-test in the lite config The main pre-commit config already excludes these POM-style support directories, but CI runs the lite config (Makefile:4175 — `pre-commit run --config .pre-commit-lite.yaml`) which was a near-duplicate that hadn't been kept in sync. Add `targets` to its excluded-dir alternation so the four compliance-harness target modules stop tripping the hook in CI. Signed-off-by: Jonathan Springer <jps@s390x.com> * chore(deps): pin authlib>=1.7.0 in the dev group via exclude-newer-package override authlib is a transitive dep of fastmcp, which the compliance harness uses as a test client — both live in the dev dependency group. mcpgateway itself imports nothing from authlib, so the constraint goes in `[dependency-groups] dev` rather than main project deps. Release 1.7.0 (2026-04-18) ships security fixes but was blocked by the repo-wide `exclude-newer = "10 days"` newness filter, leaving the lockfile on 1.6.9. Adds an `exclude-newer-package` entry for authlib (2026-04-19) so the filter allows the new release. Signed-off-by: Jonathan Springer <jps@s390x.com> --------- Signed-off-by: Jonathan Springer <jps@s390x.com>
brian-hussey
pushed a commit
that referenced
this pull request
May 5, 2026
* test(compliance): add MCP 2025-11-25 protocol compliance harness Build a FastMCP-based test harness that probes ContextForge against the MCP 2025-11-25 specification across three targets: reference stdio, gateway proxy (federated), and gateway virtual server. Includes a new reference server (mcp-servers/python/compliance_reference_server) with tools/resources/prompts covering lifecycle, transport, pagination, notifications, subscriptions, roots, sampling, elicitation, and logging. Key additions: - tests/protocol_compliance/: parametrized test suite with XPASS-capture, drift detection across targets, and gap-tracking workflow - scripts/compliance_matrix.py: orchestrator that runs the suite across engine modes (python/shadow/edge/full) and aggregates results - Runtime-mutable MCP mode tests covering shadow<->edge flip, boot-mode rails, publish/audit response contract, and data-plane runtime header - OAuth 2.1 / Authorization tests against Keycloak (tier-gated skip) - Security Best Practices probes (origin validation, bind address, confused-deputy) - Rewrite tests/e2e/test_mcp_cli_protocol.py as test_mcp_protocol_e2e.py using FastMCP Client directly (no external CLI deps) - Makefile: test-protocol-compliance* targets; testing-up folds in sso profile so Keycloak comes up with the gateway stack - pre-commit: scope name-tests-test hook to tests/unit/ (integration, e2e, and compliance suites legitimately host non-test Python modules) Closes #4273 (runtime-mutable MCP mode observability coverage). Signed-off-by: Jonathan Springer <jps@s390x.com> * refactor(tests): group test helpers under tests/*/helpers/ subdirs Move loose non-test Python modules into conventional helpers/ subdirs so the test-naming hook can exclude a single pattern instead of enumerating each file. Moves: - tests/e2e/mcp_test_helpers.py -> tests/e2e/helpers/mcp_test_helpers.py - tests/protocol_compliance/_helpers.py -> tests/protocol_compliance/helpers/compliance.py - tests/protocol_compliance/_drift.py -> tests/protocol_compliance/helpers/drift.py Updates all import sites across tests/e2e, tests/e2e_rust, and tests/protocol_compliance. Config cleanup: - name-tests-test now excludes tests/*/(helpers|fixtures|targets|pages)/ as a single verbose-regex alternative instead of file-by-file. - Top-level pre-commit exclude and detect-secrets excludes converted to verbose-regex with inline comments. - Dropped non-existent excludes (scripts/sign_image.sh, scripts/zap, sonar-project.properties) and the redundant .secrets.baseline entry (detect-secrets skips its own baseline automatically). - detect-secrets hook in pre-commit now carries its own exclude list, kept in sync with the Makefile invocation. Signed-off-by: Jonathan Springer <jps@s390x.com> * fix(keycloak): enable service-accounts on the mcp-gateway dev realm client The local Keycloak dev realm disables the client_credentials grant, so compliance tests that exercise OAuth 2.1 token issuance against the preconfigured `mcp-gateway` client fail with `unauthorized_client: Client not enabled to retrieve service account`. Flip `serviceAccountsEnabled` to `true` so fresh `docker compose` volume imports come up with the grant available. Existing stacks can apply the same change via the Keycloak Admin API against the running realm; the stored volume overrides the import file once the realm exists, which is why the change has to land here to stick across rebuilds. Signed-off-by: Jonathan Springer <jps@s390x.com> * test(compliance): PR-review follow-ups — silent-failure fixes, behavioral coverage, type design, stream attribution Bundles the fixes surfaced by running the pr-review-toolkit agents (code-reviewer, pr-test-analyzer, comment-analyzer, silent-failure-hunter, type-design-analyzer) plus two iterations of Codex stop-review feedback. Silent-failure fixes: - _parse_junit / _run_slice / stale-junit unlink / _wait_data_plane_converges return-value check close the "0/0/0/0 row" silent path in the matrix orchestrator. - Reference server stdout+stderr captured to a session tmp log; tail dumped on readiness timeout instead of swallowed. - flip_runtime_mode / _delete_if_exists / _wait_for_federation_sync / _probe_state / Keycloak token fetch surface last-status diagnostics instead of collapsing to None. - conftest fixture-resolution narrows catch-all so ImportError / NameError / SyntaxError from a broken fixture definition propagate; only runtime unreachability skips. - pytest_runtest_logreport wraps sidecar write in try/except so an unwriteable log doesn't poison the run. Type design: - ComplianceTarget: __init_subclass__ enforces name and supported_transports on every concrete subclass; base client() validates transport support and dispatches to abstract _open_client(); per-subclass boilerplate deleted. - SliceResult: stored `ran` field removed (derive from skip_reason); __post_init__ rejects negative counts and the inconsistent "skipped slice with non-zero counts" state. - ReferenceUpstream: frozen, no raw Popen field — process handle stays in fixture closure so tests can't race teardown. - KeycloakConfig: token_endpoint is now a property; __repr__ redacts client_secret. Stream attribution: - MCP 2025-11-25 § Listening for Messages is explicit: server→client *requests* (roots/list, sampling/createMessage, elicitation/create) MUST NOT ride the standalone GET /mcp/ stream; they travel on the POST-correlated stream. GAP-002..005 and seven xfail reasons wrongly blamed #4205 (standalone-stream closure). Rewrote the Why sections and reasons to point at POST-correlated-stream relay instead. - GAP-011 (resource-subscription updates) is the only gap that legitimately points at #4205. - Added a canonical "Stream-attribution note" to COMPLIANCE_GAPS.md. - Filed GAP-011 (resource subscriptions) and GAP-012 (gateway returns -32000 where spec requires -32601). Behavioral coverage (6 new tests + 4 reference-server witnesses): - notifications/cancelled end-to-end delivery. - logging/setLevel filter-threshold round-trip. - JSON-RPC reserved codes -32700 / -32601 / -32602. - tools/call argument-type validation (narrowed to McpError). - initialize-twice lifecycle rejection. - MCP-Protocol-Version mismatch handling. - Reference-server side-effect witnesses for mutate_resource_list / mutate_prompt_list / bump_subscribable notifications and the subscribe/unsubscribe handler round-trip. Each raw-httpx test fails on 5xx and accepts 4xx as HTTP-level rejection; no false-green paths where a server crash would read as compliance. Bug fixes: - Reference server: mutator tools now use a dedicated _mutation_counter to prevent ephemeral_0 name-collisions. - test_drift._collect narrows to pytest.skip.Exception; other probe errors are recorded so drift diffs show the root cause. - Makefile PYTEST_IGNORE excludes tests/protocol_compliance. Matrix rust_full slice: 58 passed / 0 failed / 40 xfailed / 0 xpassed / 11 skipped (was 53/0/34/0/11). Zero unexpected failures. Signed-off-by: Jonathan Springer <jps@s390x.com> * fix(e2e): align test_oauth_jwks_e2e JWT secret with gateway minimum Test defaulted its JWT secret to "my-test-key" (11 chars) when JWT_SECRET_KEY was unset; the gateway's validator requires ≥32 chars so every admin-API call the fixtures make would fail with 401 "Invalid authentication credentials" on default deployments. Import JWT_SECRET from helpers.mcp_test_helpers — the 40-char default the rest of the e2e suite already shares — so this test self-configures without requiring the operator to export JWT_SECRET_KEY. Overriding via the env var still works. Signed-off-by: Jonathan Springer <jps@s390x.com> * ci(pre-commit): exclude tests/**/targets/ from name-tests-test in the lite config The main pre-commit config already excludes these POM-style support directories, but CI runs the lite config (Makefile:4175 — `pre-commit run --config .pre-commit-lite.yaml`) which was a near-duplicate that hadn't been kept in sync. Add `targets` to its excluded-dir alternation so the four compliance-harness target modules stop tripping the hook in CI. Signed-off-by: Jonathan Springer <jps@s390x.com> * chore(deps): pin authlib>=1.7.0 in the dev group via exclude-newer-package override authlib is a transitive dep of fastmcp, which the compliance harness uses as a test client — both live in the dev dependency group. mcpgateway itself imports nothing from authlib, so the constraint goes in `[dependency-groups] dev` rather than main project deps. Release 1.7.0 (2026-04-18) ships security fixes but was blocked by the repo-wide `exclude-newer = "10 days"` newness filter, leaving the lockfile on 1.6.9. Adds an `exclude-newer-package` entry for authlib (2026-04-19) so the filter allows the new release. Signed-off-by: Jonathan Springer <jps@s390x.com> --------- Signed-off-by: Jonathan Springer <jps@s390x.com> Signed-off-by: Brian Hussey <brian.hussey@ie.ibm.com>
msureshkumar88
pushed a commit
that referenced
this pull request
May 13, 2026
* test(compliance): add MCP 2025-11-25 protocol compliance harness Build a FastMCP-based test harness that probes ContextForge against the MCP 2025-11-25 specification across three targets: reference stdio, gateway proxy (federated), and gateway virtual server. Includes a new reference server (mcp-servers/python/compliance_reference_server) with tools/resources/prompts covering lifecycle, transport, pagination, notifications, subscriptions, roots, sampling, elicitation, and logging. Key additions: - tests/protocol_compliance/: parametrized test suite with XPASS-capture, drift detection across targets, and gap-tracking workflow - scripts/compliance_matrix.py: orchestrator that runs the suite across engine modes (python/shadow/edge/full) and aggregates results - Runtime-mutable MCP mode tests covering shadow<->edge flip, boot-mode rails, publish/audit response contract, and data-plane runtime header - OAuth 2.1 / Authorization tests against Keycloak (tier-gated skip) - Security Best Practices probes (origin validation, bind address, confused-deputy) - Rewrite tests/e2e/test_mcp_cli_protocol.py as test_mcp_protocol_e2e.py using FastMCP Client directly (no external CLI deps) - Makefile: test-protocol-compliance* targets; testing-up folds in sso profile so Keycloak comes up with the gateway stack - pre-commit: scope name-tests-test hook to tests/unit/ (integration, e2e, and compliance suites legitimately host non-test Python modules) Closes #4273 (runtime-mutable MCP mode observability coverage). Signed-off-by: Jonathan Springer <jps@s390x.com> * refactor(tests): group test helpers under tests/*/helpers/ subdirs Move loose non-test Python modules into conventional helpers/ subdirs so the test-naming hook can exclude a single pattern instead of enumerating each file. Moves: - tests/e2e/mcp_test_helpers.py -> tests/e2e/helpers/mcp_test_helpers.py - tests/protocol_compliance/_helpers.py -> tests/protocol_compliance/helpers/compliance.py - tests/protocol_compliance/_drift.py -> tests/protocol_compliance/helpers/drift.py Updates all import sites across tests/e2e, tests/e2e_rust, and tests/protocol_compliance. Config cleanup: - name-tests-test now excludes tests/*/(helpers|fixtures|targets|pages)/ as a single verbose-regex alternative instead of file-by-file. - Top-level pre-commit exclude and detect-secrets excludes converted to verbose-regex with inline comments. - Dropped non-existent excludes (scripts/sign_image.sh, scripts/zap, sonar-project.properties) and the redundant .secrets.baseline entry (detect-secrets skips its own baseline automatically). - detect-secrets hook in pre-commit now carries its own exclude list, kept in sync with the Makefile invocation. Signed-off-by: Jonathan Springer <jps@s390x.com> * fix(keycloak): enable service-accounts on the mcp-gateway dev realm client The local Keycloak dev realm disables the client_credentials grant, so compliance tests that exercise OAuth 2.1 token issuance against the preconfigured `mcp-gateway` client fail with `unauthorized_client: Client not enabled to retrieve service account`. Flip `serviceAccountsEnabled` to `true` so fresh `docker compose` volume imports come up with the grant available. Existing stacks can apply the same change via the Keycloak Admin API against the running realm; the stored volume overrides the import file once the realm exists, which is why the change has to land here to stick across rebuilds. Signed-off-by: Jonathan Springer <jps@s390x.com> * test(compliance): PR-review follow-ups — silent-failure fixes, behavioral coverage, type design, stream attribution Bundles the fixes surfaced by running the pr-review-toolkit agents (code-reviewer, pr-test-analyzer, comment-analyzer, silent-failure-hunter, type-design-analyzer) plus two iterations of Codex stop-review feedback. Silent-failure fixes: - _parse_junit / _run_slice / stale-junit unlink / _wait_data_plane_converges return-value check close the "0/0/0/0 row" silent path in the matrix orchestrator. - Reference server stdout+stderr captured to a session tmp log; tail dumped on readiness timeout instead of swallowed. - flip_runtime_mode / _delete_if_exists / _wait_for_federation_sync / _probe_state / Keycloak token fetch surface last-status diagnostics instead of collapsing to None. - conftest fixture-resolution narrows catch-all so ImportError / NameError / SyntaxError from a broken fixture definition propagate; only runtime unreachability skips. - pytest_runtest_logreport wraps sidecar write in try/except so an unwriteable log doesn't poison the run. Type design: - ComplianceTarget: __init_subclass__ enforces name and supported_transports on every concrete subclass; base client() validates transport support and dispatches to abstract _open_client(); per-subclass boilerplate deleted. - SliceResult: stored `ran` field removed (derive from skip_reason); __post_init__ rejects negative counts and the inconsistent "skipped slice with non-zero counts" state. - ReferenceUpstream: frozen, no raw Popen field — process handle stays in fixture closure so tests can't race teardown. - KeycloakConfig: token_endpoint is now a property; __repr__ redacts client_secret. Stream attribution: - MCP 2025-11-25 § Listening for Messages is explicit: server→client *requests* (roots/list, sampling/createMessage, elicitation/create) MUST NOT ride the standalone GET /mcp/ stream; they travel on the POST-correlated stream. GAP-002..005 and seven xfail reasons wrongly blamed #4205 (standalone-stream closure). Rewrote the Why sections and reasons to point at POST-correlated-stream relay instead. - GAP-011 (resource-subscription updates) is the only gap that legitimately points at #4205. - Added a canonical "Stream-attribution note" to COMPLIANCE_GAPS.md. - Filed GAP-011 (resource subscriptions) and GAP-012 (gateway returns -32000 where spec requires -32601). Behavioral coverage (6 new tests + 4 reference-server witnesses): - notifications/cancelled end-to-end delivery. - logging/setLevel filter-threshold round-trip. - JSON-RPC reserved codes -32700 / -32601 / -32602. - tools/call argument-type validation (narrowed to McpError). - initialize-twice lifecycle rejection. - MCP-Protocol-Version mismatch handling. - Reference-server side-effect witnesses for mutate_resource_list / mutate_prompt_list / bump_subscribable notifications and the subscribe/unsubscribe handler round-trip. Each raw-httpx test fails on 5xx and accepts 4xx as HTTP-level rejection; no false-green paths where a server crash would read as compliance. Bug fixes: - Reference server: mutator tools now use a dedicated _mutation_counter to prevent ephemeral_0 name-collisions. - test_drift._collect narrows to pytest.skip.Exception; other probe errors are recorded so drift diffs show the root cause. - Makefile PYTEST_IGNORE excludes tests/protocol_compliance. Matrix rust_full slice: 58 passed / 0 failed / 40 xfailed / 0 xpassed / 11 skipped (was 53/0/34/0/11). Zero unexpected failures. Signed-off-by: Jonathan Springer <jps@s390x.com> * fix(e2e): align test_oauth_jwks_e2e JWT secret with gateway minimum Test defaulted its JWT secret to "my-test-key" (11 chars) when JWT_SECRET_KEY was unset; the gateway's validator requires ≥32 chars so every admin-API call the fixtures make would fail with 401 "Invalid authentication credentials" on default deployments. Import JWT_SECRET from helpers.mcp_test_helpers — the 40-char default the rest of the e2e suite already shares — so this test self-configures without requiring the operator to export JWT_SECRET_KEY. Overriding via the env var still works. Signed-off-by: Jonathan Springer <jps@s390x.com> * ci(pre-commit): exclude tests/**/targets/ from name-tests-test in the lite config The main pre-commit config already excludes these POM-style support directories, but CI runs the lite config (Makefile:4175 — `pre-commit run --config .pre-commit-lite.yaml`) which was a near-duplicate that hadn't been kept in sync. Add `targets` to its excluded-dir alternation so the four compliance-harness target modules stop tripping the hook in CI. Signed-off-by: Jonathan Springer <jps@s390x.com> * chore(deps): pin authlib>=1.7.0 in the dev group via exclude-newer-package override authlib is a transitive dep of fastmcp, which the compliance harness uses as a test client — both live in the dev dependency group. mcpgateway itself imports nothing from authlib, so the constraint goes in `[dependency-groups] dev` rather than main project deps. Release 1.7.0 (2026-04-18) ships security fixes but was blocked by the repo-wide `exclude-newer = "10 days"` newness filter, leaving the lockfile on 1.6.9. Adds an `exclude-newer-package` entry for authlib (2026-04-19) so the filter allows the new release. Signed-off-by: Jonathan Springer <jps@s390x.com> --------- Signed-off-by: Jonathan Springer <jps@s390x.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a pytest-based MCP 2025-11-25 protocol-compliance harness, a FastMCP reference server to act as the known-good witness, and a matrix orchestrator that runs the harness across every runnable engine mode of the runtime-mutable MCP gateway (#4273, #4296).
tests/protocol_compliance/— parametrized probe suite covering lifecycle, transport, tools, resources, prompts, logging, utilities (ping/progress/cancellation), roots, sampling, elicitation, notifications (list_changed, subscription updates), pagination, OAuth 2.1 via Keycloak, security best practices, runtime-mutable MCP mode, and JSON-RPC envelope invariants.mcp-servers/python/compliance_reference_server/— FastMCP-based reference server with explicit witnesses for every capability the harness probes, including newget_cancellation_countand_mutation_counterplumbing so end-to-end notification delivery is observable from tests.scripts/compliance_matrix.py— orchestrator that flips runtime-mode, waits for data-plane convergence, invokes the harness per engine (reference / python / rust_edge / rust_full), and renders a console / markdown / JSON summary with XPASS-sidecar capture for detecting silently-closed gaps.tests/protocol_compliance/COMPLIANCE_GAPS.md— tracked backlog of documented divergences (GAP-001 … GAP-012) with workflow for filing, narrowing on partial closure, and the canonical stream-attribution note for MCP 2025-11-25 § Listening for Messages.test-protocol-compliance*targets;testing-upnow folds in thessocompose profile so Keycloak comes up with the stack.infra/keycloak/realm-export.json— service-accounts enabled on the dev client so the client_credentials OAuth tests work out of the box.Scope of changes
5 commits, all linear on top of the latest
origin/main(post-#4296, post-#4299):test(compliance)— the harness, reference server, matrix script, rewrittentest-mcp-protocol-e2etarget.refactor(tests)— group test helpers undertests/*/helpers/+ verbose-regex precommit excludes.fix(keycloak)— enable service-accounts on the dev realm client.test(compliance)follow-ups — silent-failure fixes, type-design tightening, behavioral coverage, corrected stream attribution, 6 new tests.fix(e2e)— aligntest_oauth_jwks_e2eJWT secret with the gateway's 32-char minimum.Current matrix status
After
make testing-up-rust-full:Zero unexpected failures, zero XPASS. Each xfail points at a documented GAP; 11 gaps currently open. Python and
rust_edgeingress paths behave identically under the probe set (same pass/xfail/skip counts), which is the drift parity we want to pin.Test plan
make test-protocol-compliance-reference— reference-only, green.make test-protocol-compliance-gateway— gateway-only, green with expected xfails.make test-protocol-compliance-matrix— full matrix under boot=edge and boot=full.Follow-ups (filed / tracked)
tasks/*capability coverage (blocked on FastMCPtasksextra).COMPLIANCE_GAPS.md, each with an xfail anchor.