feat(security): add container image scanner plugin with Trivy/Grype integration (#2216) - #3713
feat(security): add container image scanner plugin with Trivy/Grype integration (#2216)#3713kamath-a wants to merge 141 commits into
Conversation
…y evaluation, caching, storage, plugin, and some unit testing Signed-off-by: Agnetha <agnethakamath@gmail.com>
Signed-off-by: Agnetha <agnethakamath@gmail.com>
Signed-off-by: Agnetha <agnethakamath@gmail.com>
Signed-off-by: Agnetha <agnethakamath@gmail.com>
Signed-off-by: Agnetha <agnethakamath@gmail.com>
Signed-off-by: Agnetha <agnethakamath@gmail.com>
Signed-off-by: Agnetha <agnethakamath@gmail.com>
Signed-off-by: Agnetha <agnethakamath@gmail.com>
|
Thanks @kamath-a. Strong implementation with excellent test coverage (109 unit + 18 integration). However, this PR modifies 10 core gateway files (main.py, admin.py, framework hooks, new router, templates) — a plugin should be self-contained within |
|
Hi @crivetimihai, apologies for the late response, and thank you so much for the feedback! This is indeed the active PR, #3658 is now outdated. I have looked into some of the other plugins and their structures, and from my understanding, the following changes are required:
Please let me know if I have misunderstood or omitted anything! Thanks again! |
|
DO NOT MERGE before #3754 is merged. |
| from pydantic import Field | ||
|
|
||
| # First-Party | ||
| from mcpgateway.plugins.framework.models import PluginPayload, PluginResult |
There was a problem hiding this comment.
Hi @kamath-a Could you rebase this PR. The plugin framework was transformed into a python package. These imports now would import from cpex.
from cpex.framework import PluginPayload, PluginResult
…E-formatted strings (IBM#3595) * fix: SSE resource subscribe endpoint yielding raw dicts instead of SSE-formatted strings Signed-off-by: Marek Dano <Marek.Dano@ibm.com> * fix: lint issue in main.py Signed-off-by: Marek Dano <Marek.Dano@ibm.com> * fix: move orjson import to module level and align docstring style Move the orjson import from inside test function to module-level imports (PEP 8 compliance). Align sse_generator() docstring with the existing generate_events() pattern used by the roots subscribe endpoint. Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> --------- Signed-off-by: Marek Dano <Marek.Dano@ibm.com> Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> Co-authored-by: Marek Dano <Marek.Dano@ibm.com> Co-authored-by: Mihai Criveti <crivetimihai@gmail.com>
* Restore cargo lock file Signed-off-by: Dmitry Zakharov <zakharov@ibm.com> * pin maturin version Signed-off-by: Dmitry Zakharov <zakharov@ibm.com> * stub file addtion Signed-off-by: Dmitry Zakharov <zakharov@ibm.com> * restore Cargo.lock files Signed-off-by: Dmitry Zakharov <zakharov@ibm.com> * update stub files Signed-off-by: Dmitry Zakharov <zakharov@ibm.com> * Follow suggestion of CI manifest check Signed-off-by: Dmitry Zakharov <zakharov@ibm.com> * Ugrade aws-lc-rs as found by CI CVE checks Signed-off-by: Dmitry Zakharov <zakharov@ibm.com> * Ugrade rustls-webpki as found by CI CVE checks Signed-off-by: Dmitry Zakharov <zakharov@ibm.com> * Remove before refactor layout Cargo.lock file Signed-off-by: Dmitry Zakharov <zakharov@ibm.com> * pin libs Signed-off-by: Dima Zakharov <zakharov@ibm.com> --------- Signed-off-by: Dmitry Zakharov <zakharov@ibm.com> Signed-off-by: Dima Zakharov <zakharov@ibm.com>
Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>
… sessions (IBM#3731) (IBM#3813) * fix(db): restore transaction control to get_db() for middleware sessions PR IBM#3600 introduced a transaction management violation where ObservabilityMiddleware commits the shared database session instead of get_db(), breaking the established contract where get_db() controls transaction boundaries. This creates data integrity risks where failed validations can be committed to the database. This fix restores the correct behavior: - Middleware manages session lifecycle (create/close) - get_db() manages transactions (commit/rollback) Changes: - Remove commit logic from ObservabilityMiddleware (observability_middleware.py:210-216) - Add commit/rollback handling to get_db() for middleware sessions (main.py:3137-3164) - Update get_db() docstring to document transaction control responsibility - Update 2 existing tests to reflect new behavior - Add 7 comprehensive tests for transaction semantics Security implications: - Fixes data integrity bug where invalid data could be committed - Maintains proper transaction isolation per request - Preserves connection invalidation on broken connections - No impact on auth/RBAC (middleware runs before route handlers) Trade-offs: - Observability data (traces/spans) is rolled back on errors (acceptable - best-effort tracing) Closes IBM#3731 Signed-off-by: Mohan Lakshmaiah <mohan.economist@gmail.com> * test(db): add coverage for double-failure edge case in get_db() Signed-off-by: Mohan Lakshmaiah <mohan.economist@gmail.com> * fix(tests): clean up lint violations in transaction control tests Remove unused AsyncMock import and unused variable assignments flagged by ruff (F401, F841). Apply isort/black formatting. Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> --------- Signed-off-by: Mohan Lakshmaiah <mohan.economist@gmail.com> Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> Co-authored-by: Mihai Criveti <crivetimihai@gmail.com>
* virtual-servers-select-all-count Signed-off-by: NAYANA.R <nayana.r7813@gmail.com> * remove unused originalText variable in select all handler Signed-off-by: NAYANA.R <nayana.r7813@gmail.com> * fix(ui): apply Select All count display consistently across all init*Select functions Extend the Select All button count fix from initToolSelect to initResourceSelect, initPromptSelect, and initGatewaySelect for consistency. Remove stale originalText + setTimeout pattern from all three sibling functions. Update Playwright assertions to match new button text format and add JS unit tests for count display behavior. Closes IBM#3833 Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> --------- Signed-off-by: NAYANA.R <nayana.r7813@gmail.com> Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> Co-authored-by: Mihai Criveti <crivetimihai@gmail.com>
…update roadmap (IBM#3716) - Remove importchecker and unimport from Makefile, pyproject.toml, and CI workflow (redundant with ruff/flake8 F401 rule) - Change `make coverage` pytest flag from -rA to -rfE to suppress log capture noise from passing tests - Mark 33 closed issues in roadmap.md (⏳ → ✅) - Fix pr-review skill to select exactly one recommendation - Align e2e test JWT secret default with docker-compose.yml - Add --admin flag to register_fast_test JWT token generation - Update lock files Closes IBM#1290 Signed-off-by: Jonathan Springer <jps@s390x.com>
…endpoints (IBM#3676) * feat(api): Add gateway_id filtering to prompts and resources listing endpoints Add server-side gateway_id filtering to GET /prompts and GET /resources, matching the existing capability on GET /tools. This eliminates the need for client-side filtering when retrieving prompts or resources associated with a specific physical MCP server (gateway). Closes IBM#3638 Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> * fix: populate gateway_id in ResourceRead, add limit to prompt cache hash, strengthen tests - Add gateway_id to convert_resource_to_read() so ResourceRead responses actually include the field added to the schema in the prior commit - Include limit in prompt service cache hash to prevent cache poisoning across different page sizes (resource service already had this) - Add endpoint-level tests verifying gateway_id forwarding from main.py - Add integration tests for partition completeness and null filtering - Strengthen unit tests to verify actual SQL WHERE clauses via query compilation - Add gateway_id to resource mock fixtures to fix MagicMock/Pydantic validation conflicts - Fix pre-existing ruff lint errors in touched test files Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Jonathan Springer <jps@s390x.com> --------- Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> Signed-off-by: Jonathan Springer <jps@s390x.com> Co-authored-by: Jonathan Springer <jps@s390x.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…BM#3677) * fix(mcp): Use compute_passthrough_headers_cached in Streamable HTTP direct proxy paths Replace manual header loops in _proxy_list_tools_to_gateway, _proxy_list_resources_to_gateway, and _proxy_read_resource_to_gateway with calls to compute_passthrough_headers_cached so that X-Upstream-Authorization → Authorization rename, global passthrough config, and header sanitization are applied consistently. Closes IBM#3643 Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> * fix(test): Widen `app` fixture scope from function to module The function-scoped `app` fixture was causing ~25s setup per test due to repeated mcpgateway.main imports. Module scope matches the existing `app_with_temp_db` fixture and pays the import cost once per module. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Jonathan Springer <jps@s390x.com> * fix(mcp): Skip DB lookup for passthrough headers when gateway config is explicit When a gateway has explicit passthrough_headers (including empty []), use them directly instead of querying the global config cache. This avoids an unnecessary DB round-trip and ensures an explicit empty list does not fall through to the global allowlist. Closes IBM#3643 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Jonathan Springer <jps@s390x.com> --------- Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> Signed-off-by: Jonathan Springer <jps@s390x.com> Co-authored-by: Jonathan Springer <jps@s390x.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* updating dependancies for dependabot Signed-off-by: Yosief Eyob <yosiefogbazion@gmail.com> * fix: move undici to overrides instead of dependencies Signed-off-by: Yosief Eyob <yosiefogbazion@gmail.com> * Broaden required checks so they always run Signed-off-by: Jonathan Springer <jps@s390x.com> --------- Signed-off-by: Yosief Eyob <yosiefogbazion@gmail.com> Signed-off-by: Jonathan Springer <jps@s390x.com> Co-authored-by: Jonathan Springer <jps@s390x.com>
* fix(api): deactive a2a agents tools when a2a agents are inactive Signed-off-by: Marek Dano <mk.dano@gmail.com> * fix: add tool_lookup_cache and try catch with handling errors when updating a2a agent tools Signed-off-by: Marek Dano <mk.dano@gmail.com> * fix: add gateway_id to 'tool_lookup_cache.invalidate' function in 'set_agent_state' function Signed-off-by: Marek Dano <mk.dano@gmail.com> * fix: apply black formatting and correct stale docstring path - Run black on test_a2a_service.py to fix line-length violations in new tests - Update docstring Location path in test_a2a_agent.py after rename from test_issue_840_a2a_agent.py Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> * fix: tighten docs and stale docstring per code review feedback - Update test_a2a_agent.py module docstring to reflect both IBM#840 and IBM#2997 coverage instead of stale issue-840-only wording - Clarify a2a.md cascade docs: invocation was already rejected for disabled agents; the fix ensures the tool's enabled flag stays in sync so it no longer appears in listings Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> * fix: let cascade failures propagate instead of swallowing them Remove try/except around the tool cascade in set_agent_state() so that DB failures surface to the caller. This aligns with gateway_service.py's set_gateway_state() which also commits the parent first (line 2773) then cascades to child tools/prompts/resources without catching exceptions. The previous best-effort pattern silently returned success when the tool UPDATE failed, leaving agent disabled but tool still enabled — the exact inconsistency this PR is meant to fix. Update test_cascade_tool_update_failure to assert the exception propagates instead of being swallowed. Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> --------- Signed-off-by: Marek Dano <mk.dano@gmail.com> Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> Co-authored-by: Marek Dano <mk.dano@gmail.com>
Consistent (non-triggering) EXAMPLE AWS API key in unit tests. Signed-off-by: Jonathan Springer <jps@s390x.com>
* Final true up of detect-secrets work. Disabling gitleaks as it conflicts with detect-secrets Signed-off-by: Brian Hussey <brian.hussey@ie.ibm.com> * True up final pieces to pass other pre-commit stages that were previously missed Signed-off-by: Brian Hussey <brian.hussey@ie.ibm.com> --------- Signed-off-by: Brian Hussey <brian.hussey@ie.ibm.com>
* fix: remove unmaintained rustls-pemfile from mcp runtime Signed-off-by: lucarlig <luca.carlig@ibm.com> * test: avoid panic when native root store is empty Signed-off-by: lucarlig <luca.carlig@ibm.com> * fix: refresh detect-secrets baseline Signed-off-by: lucarlig <luca.carlig@ibm.com> --------- Signed-off-by: lucarlig <luca.carlig@ibm.com>
…regression tests (IBM#3840) * fix(pii_filter): add mask strategy preservation and nested key support in Rust implementation Signed-off-by: lucarlig <luca.carlig@ibm.com> * fix(pii_filter): add comprehensive detection patterns and ReDoS protection to Rust implementation Signed-off-by: lucarlig <luca.carlig@ibm.com> * fix(pii_filter): add input validation and error handling to Rust masking logic Signed-off-by: lucarlig <luca.carlig@ibm.com> * fix(pii_filter): add resource limits and config validation to Rust detector Signed-off-by: lucarlig <luca.carlig@ibm.com> * fix(pii_filter): add comprehensive test coverage for Rust detection edge cases Signed-off-by: lucarlig <luca.carlig@ibm.com> * docs(pii_filter): document ReDoS protection limits and add resource limit upper bounds Signed-off-by: lucarlig <luca.carlig@ibm.com> Signed-off-by: lucarlig <luca.carlig@ibm.com> * fix(pii_filter): resolve clippy warnings in Rust implementation Signed-off-by: lucarlig <luca.carlig@ibm.com> * fix(pii_filter): tighten rust ssn and limit validation Signed-off-by: lucarlig <luca.carlig@ibm.com> * fix(pii_filter): restore benchmark config build Signed-off-by: lucarlig <luca.carlig@ibm.com> * fix: raise default rust pii text limit Signed-off-by: lucarlig <luca.carlig@ibm.com> * docs: clarify rust pii filter coverage Signed-off-by: lucarlig <luca.carlig@ibm.com> * fix: restore rust pii coverage edge cases Signed-off-by: lucarlig <luca.carlig@ibm.com> * fix: harden loopback passthrough and ssn validation Signed-off-by: lucarlig <luca.carlig@ibm.com> * fix(pii_filter): address rust review follow-ups Signed-off-by: lucarlig <luca.carlig@ibm.com> * fix: address pii filter review follow-ups Signed-off-by: lucarlig <luca.carlig@ibm.com> * fix: audit detect-secrets baseline for pii filter tests Signed-off-by: lucarlig <luca.carlig@ibm.com> --------- Signed-off-by: lucarlig <luca.carlig@ibm.com>
* fix(validation): remove single pipe from forbidden description patterns
Remove single pipe character ("|") from the default
TOOL_DESCRIPTION_FORBIDDEN_PATTERNS list. The pipe is a valid character in
LogQL (|=, |~), PromQL, regex patterns, and Markdown tables. The dangerous
shell OR operator "||" remains blocked.
Also aligns ToolUpdate.validate_description with ToolCreate by using the
configurable settings.tool_description_forbidden_patterns instead of a
hardcoded list, ensuring consistent behavior when the pattern list is
customized via environment variables.
Closes IBM#3811
Signed-off-by: NAYANA.R <nayana.r7813@gmail.com>
Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>
* fix(env): align .env.example JWT secret with docker-compose default
.env.example still used the old short `my-test-key` while
docker-compose.yml and the E2E test helpers were updated to
`my-test-key-but-now-longer-than-32-bytes` in IBM#3716. Users who
copied .env.example to .env got a secret mismatch that caused
`make test-mcp-cli` to hang.
See IBM#3889 for the remaining files that need the same update.
Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>
* chore: apply linter and formatter fixes
Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>
* fix(env): align all JWT secret defaults with docker-compose (IBM#3889)
PR IBM#3716 updated docker-compose.yml and mcp_test_helpers.py to use
`my-test-key-but-now-longer-than-32-bytes` (meeting the 32-byte minimum
for HS256 per RFC 7518 §3.2) but missed ~70 other files that still
hardcoded the old `my-test-key`.
This caused test failures (make test-mcp-cli hangs, load tests fail auth)
when .env is derived from .env.example.
Updated:
- All docker-compose variant files (debug, embedded, performance, verbose)
- All E2E and load test defaults
- All scripts and smoketests
- Helm chart values, schema, and docs
- Makefile targets
- All documentation examples
- Added long key to validate_env.py weak_jwt list
Not changed (intentionally):
- mcpgateway/config.py:286 — Python config default (source of truth for
standalone `make dev`)
- mcpgateway/config.py:888 — already lists both keys in weak_secrets
- mcpgateway/config.py:1088, main.py:1955,2040 — guards checking for
the default value
- tests/unit/test_main_helpers_extra.py:55 — test that mocks the default
Closes IBM#3889
Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>
* fix(security): detect both JWT default secrets in guard paths
The secure_secrets flag and critical-issues check only matched the
short `my-test-key` default. Users running with the docker-compose
default `my-test-key-but-now-longer-than-32-bytes` would bypass the
security warning and the `secure_secrets: false` status flag.
- config.py get_security_status(): `!=` → `not in (short, long)`
- main.py validate_security_configuration(): same
- main.py security recommendations log: same
- validate_env.py already had both (updated in prior commit)
- config.py:888 weak_secrets already had both
Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>
* chore: update .secrets.baseline line numbers after reformatting
Pre-commit detect-secrets hook requires line numbers to match. The
linter/formatter commit shifted lines in db_util.py. All entries
remain is_secret=false (confirmed false positives).
Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>
* chore: allowlist test JWT secrets for detect-secrets hook
The longer JWT default triggers IBM detect-secrets "Secret Keyword"
detection on 7 lines that are all test/dev defaults. Added
`pragma: allowlist secret` inline comments.
Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>
* chore: update .secrets.baseline after allowlist pragmas
Pre-commit detect-secrets hook removed entries that are now
covered by inline `pragma: allowlist secret` comments, and
adjusted line numbers.
Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>
* baseline
Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>
---------
Signed-off-by: NAYANA.R <nayana.r7813@gmail.com>
Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>
Co-authored-by: Mihai Criveti <crivetimihai@gmail.com>
Signed-off-by: lucarlig <luca.carlig@ibm.com>
…#3682) * fix: resolve MAX_MEMBERS_PER_TEAM not applying to existing teams Stop baking the global MAX_MEMBERS_PER_TEAM config default into team DB rows at creation time. Instead, store NULL and resolve the effective limit at check time via get_effective_max_members(), which falls back to the current settings value. This allows changing the env var to take effect for all teams that don't have an explicit per-team override. Also adds missing member limit check in approve_join_request() and updates tests accordingly. Closes IBM#3588 Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> * fix: harden MAX_MEMBERS_PER_TEAM resolution with UNSET sentinel, admin UI, and centralized capacity checks - Stop baking MAX_MEMBERS_PER_TEAM config default into team DB rows at creation time; store NULL so changing the env var takes effect for all teams without explicit per-team overrides - Add get_effective_max_members() helper that resolves NULL → current settings.max_members_per_team at check time - Add check_team_member_capacity() to centralize member-limit enforcement across add_member, create_invitation, accept_invitation, and approve_join_request - Introduce typed _Unset sentinel so update_team can distinguish "caller omitted max_members" from "caller passed None to clear override" - Add ge=1 validation to max_members on schemas and admin form handlers - Add admin UI checkbox to toggle between global default and per-team override - Catch TeamMemberLimitExceededError in invitation router endpoints - Move max_members cap enforcement from router into service layer Signed-off-by: Jonathan Springer <jps@s390x.com> --------- Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> Signed-off-by: Jonathan Springer <jps@s390x.com> Co-authored-by: Jonathan Springer <jps@s390x.com>
…horized Access (IBM#3892) * fix(security): validate server ID in Streamable HTTP to prevent unauthorized access The hex-only regex _SERVER_ID_RE only captured server IDs matching [a-fA-F0-9\-]+. Non-hex IDs (e.g. "xyz", "hello world", empty segments) caused the regex to silently fail, setting server_id to None and falling through to unscoped global access — exposing all tools, prompts, resources, and agents to any authenticated caller. Three-layer defense: - Widen _SERVER_ID_RE from [a-fA-F0-9\-]+ to [^/]+ so ALL server- scoped paths are captured and validated against the database - Add defense-in-depth guard via _SERVER_SCOPED_PATH_RE to reject edge cases like /servers//mcp where the primary regex still doesn't match - Add server ID format validation in MCPPathRewriteMiddleware before path rewriting occurs Additional improvements: - Extract _validate_server_id() to SessionManagerWrapper for clarity - Extract entity_exists() to BaseService for lightweight existence checks - Add require_valid_server FastAPI dependency for the message endpoint - Use ORJSONResponse pattern for consistency with rest of codebase - Remove server_id from error messages to prevent information leakage Closes IBM#3891 Co-authored-by: Jonathan Springer <jps@s390x.com> Signed-off-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com> Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> * fix(security): anchor regex and middleware to canonical /servers/ paths Address Codex review findings: 1. Move _validate_server_id() before affinity routing branches so nonexistent server IDs get a clean 404 even with stateful session affinity enabled (USE_STATEFUL_SESSIONS + SESSION_AFFINITY). 2. Anchor _SERVER_ID_RE and _SERVER_SCOPED_PATH_RE with ^ so paths like /foo/servers/xyz/mcp cannot embed a server segment and bypass route matching. 3. Restrict MCPPathRewriteMiddleware to only rewrite paths starting with /servers/ — arbitrary prefixes (e.g. /foo/mcp) now pass through without rewriting to /mcp/. 4. Add entity_exists mock to 8 existing affinity tests that now hit the earlier validation gate. Closes IBM#3891 Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> --------- Signed-off-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com> Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> Co-authored-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com> Co-authored-by: Jonathan Springer <jps@s390x.com> Co-authored-by: Mihai Criveti <crivetimihai@gmail.com>
* feat: add retry-with-exponential-backoff plugin - Add retry_delay_ms field to PluginResult in models.py - Add recursive retry loop in tool_service.py invoke_tool (retry_attempt param) - Fix manager.py to propagate retry_delay_ms signal across plugin chain - Add RetryWithBackoffPlugin with full-jitter exponential backoff - Add plugin-manifest.yaml and package __init__.py - Add 35 unit tests covering all components Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com> * feat: IBM#3746 implemented retry with backoff (RUST/Python) Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com> * IBM#3746 add README.md for retry_with_backoff Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com> * IBM#3746 resolve lint failures and add Rust plugin Makefile/deny.toml Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com> * IBM#3746 add pyo3-stub-gen support and Python stub for retry_with_backoff Rust plugin Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com> * IBM#3746 cover tool_service retry path to satisfy diff-cover 95% threshold Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com> * benchmark Signed-off-by: Dima Zakharov <zakharov@ibm.com> * IBM#3746 retry on exception path — honour plugin retry_delay_ms when tool raises Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com> * benchmark script Signed-off-by: Dmitry Zakharov <zakharov@ibm.com> * IBM#3746 add Rust unit tests and pyo3-log integration for retry_with_backoff Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com> * IBM#3746 cargo fmt: apply rustfmt to lib.rs Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com> * Coverage targets Signed-off-by: Dmitry Zakharov <zakharov@ibm.com> * summary coverage fix Signed-off-by: Dmitry Zakharov <zakharov@ibm.com> * IBM#3746 address PR review comments: clarify retry count and document resource retry limitation Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com> * IBM#3746 address review: harden retry plugin, add timeout retry path, fix Rust monotonic clock - Refactor retry logic into _run_timeout_post_invoke and _retry_tool_invocation helpers, eliminating 4 copies of timeout post-invoke and 3 copies of retry invocation code - Add retry support for timeout path via ToolTimeoutError.retry_delay_ms - Add status-code-aware isError handling: non-transient HTTP errors (400, 401, 404) are no longer blindly retried when the gateway can extract the status code from httpx.HTTPStatusError - Switch Rust state TTL from SystemTime to Instant (monotonic clock) to match Python's time.monotonic() and avoid wall-clock jump issues - Add defensive guard in _run_timeout_post_invoke for None plugin_manager - Add state TTL eviction, docstrings, and check_text_content to plugin manifest - Add tests: timeout retry path, timeout no-retry re-raise, _run_timeout_post_invoke hook invocation, HTTP status forwarding to plugin, non-/servers/ MCP path passthrough - Remove unused ToolHookType imports in test_tool_service_coverage.py Signed-off-by: Jonathan Springer <jps@s390x.com> --------- Signed-off-by: Madhu Mohan Jaishankar <madhu.mohan.jaishankar@ibm.com> Signed-off-by: Dima Zakharov <zakharov@ibm.com> Signed-off-by: Dmitry Zakharov <zakharov@ibm.com> Signed-off-by: Jonathan Springer <jps@s390x.com> Co-authored-by: Dima Zakharov <zakharov@ibm.com> Co-authored-by: Jonathan Springer <jps@s390x.com>
* fix: infinite /partial request loop triggered by search input Signed-off-by: Marek Dano <Marek.Dano@ibm.com> * fix: merging conflicts Signed-off-by: Marek Dano <Marek.Dano@ibm.com> * fix: npm deps for high vulnerabilities Signed-off-by: Marek Dano <Marek.Dano@ibm.com> * fix: package-lock.json file Signed-off-by: Marek Dano <Marek.Dano@ibm.com> * fix: E401 authentication error on npm install and pointing to the public registry Signed-off-by: Marek Dano <Marek.Dano@ibm.com> * fix: package-lock.json file Signed-off-by: Marek Dano <Marek.Dano@ibm.com> * fix: improve search init comments, lint, and test robustness Update stale doc comment on initializeSearchInputs() to reflect that HTMX handlers no longer re-invoke it. Prefix unused entityType parameter with underscore to satisfy linter. Add URL cleanup in beforeEach to prevent test contamination on mid-test failures. Signed-off-by: Jonathan Springer <jps@s390x.com> --------- Signed-off-by: Marek Dano <Marek.Dano@ibm.com> Signed-off-by: Jonathan Springer <jps@s390x.com> Co-authored-by: Marek Dano <Marek.Dano@ibm.com> Co-authored-by: Jonathan Springer <jps@s390x.com>
…BM#3764) * fix: tighten secrets detection coverage Signed-off-by: lucarlig <luca.carlig@ibm.com> * fix: avoid codeql parser edge case in pre-commit lite config Signed-off-by: lucarlig <luca.carlig@ibm.com> * fix: preserve safe defaults in secrets detection config Signed-off-by: lucarlig <luca.carlig@ibm.com> * docs: clarify broader secrets heuristic coverage Signed-off-by: lucarlig <luca.carlig@ibm.com> * chore: restore pre-commit lite comments after rebase Signed-off-by: lucarlig <luca.carlig@ibm.com> * fix: restore detect-secrets flow and align secret coverage Signed-off-by: lucarlig <luca.carlig@ibm.com> * fix: stabilize secrets detection pre-commit coverage Signed-off-by: lucarlig <luca.carlig@ibm.com> * fix: unify is_enabled default and align config with code changes The detection and redaction paths in the Python secrets plugin used different defaults for unknown patterns (False vs True), which could cause redaction of patterns that were never detected. Extract an is_enabled() method on both the Python and Rust config classes so the safe default (disabled) is defined in exactly one place. Also commits config.yaml changes that align with the code: - remove stale detect_aws_keys / detect_api_keys from PII filter config - set SecretsDetection mode to disabled (opt-in, not enforced by default) - default unknown patterns to disabled in Rust scanner Signed-off-by: Jonathan Springer <jps@s390x.com> * docs: add make rust-check to pre-commit quality checklists CLAUDE.md and DEVELOPING.md only listed Python linters in their "before committing" sections. Add make rust-check so developers and AI agents run clippy -D warnings before pushing Rust changes. Signed-off-by: Jonathan Springer <jps@s390x.com> --------- Signed-off-by: lucarlig <luca.carlig@ibm.com> Signed-off-by: Jonathan Springer <jps@s390x.com> Co-authored-by: Jonathan Springer <jps@s390x.com>
* fix(sso): reconcile team memberships on SSO login to remove stale grants Signed-off-by: Jonathan Springer <jps@s390x.com> * fix: address CI test failures and linting issues Signed-off-by: Jonathan Springer <jps@s390x.com> * fix(sso): update grant_source on reactivation, strengthen tests, fix formatting - Update reactivation path in add_member_to_team to set grant_source when provided, preventing SSO-reactivated memberships from escaping reconciliation on subsequent logins - Strengthen test_apply_team_mapping_queries_sso_memberships_correctly to compile the SQLAlchemy statement and verify filter clauses - Add test coverage for whitespace-only team_mapping keys (sso_service) and grant_source update on membership reactivation (team_management) - Fix black formatting in Alembic migration and test files - Add missing docstrings to _Unset.__repr__ and _Unset.__bool__ Signed-off-by: Jonathan Springer <jps@s390x.com> * refactor: extract helpers from sso_service and team_management_service to reduce duplication sso_service.py: - Extract _extract_groups_and_roles and _build_normalized_user_info from _normalize_user_info, eliminating 4x duplicated group extraction and 7x duplicated dict construction - Extract _enrich_user_data_from_claims from _get_user_info, separating provider-specific claim enrichment from the HTTP fetch/normalize flow - Extract _check_pending_approval and _reset_pending_approval, flattening 5-level nested approval dispatch to early-return style - Extract _should_sync_roles, eliminating duplicated sync logic in both the existing-user and new-user paths - Remove redundant guard in Entra id_token_claims check - Use set lookups instead of list-in-list O(n^2) checks in _should_user_be_admin - Use typed _Unset sentinel enum for _build_normalized_user_info overrides so callers can explicitly pass None team_management_service.py: - Extract _assign_team_rbac_role from add_member_to_team and approve_join_request - Extract _invalidate_membership_caches from 3 membership-change methods - Extract _apply_team_list_filters from list_teams, get_all_team_ids, and get_teams_count, fixing filter divergence - Add personal_owner_email parameter to get_teams_count for consistency - Extract _check_user_team_limit for the max-teams-per-user guard - Fix print() to logger.error in verify_team_for_user test_sso_service.py: - Add 44 new tests for extracted helpers - Reorganize test classes by method under test Signed-off-by: Jonathan Springer <jps@s390x.com> --------- Signed-off-by: Jonathan Springer <jps@s390x.com> Co-authored-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com> Co-authored-by: Jonathan Springer <jps@s390x.com>
…BM#3918) Conflict markers were reintroduced by rebase. Also re-fix the bob command inline code line break. Signed-off-by: Jonathan Springer <jps@s390x.com>
…n checks (IBM#3919) * fix(rbac): enforce session-token team narrowing in Layer 2 permission checks Signed-off-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com> * Added unit tests for token narrowing Signed-off-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com> * Fixed pre-commit issues Signed-off-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com> * fix(rbac): correct misleading test and clean up token narrowing tests - Fix test_check_permission_public_only_token that passed due to wildcard string mismatch rather than actual role filtering; renamed to test_check_permission_public_only_token_retains_non_admin_team_perms with realistic permissions and accurate assertion (Layer 1 handles public-only visibility, not Layer 2 role filtering) - Remove unused imports (datetime, timezone) and unused variable (role_c) - Remove trailing "Made with Bob" comment - Apply black/isort formatting to match project standards - Clean up duplicate inline comments on or_() clause Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> * fix: add missing Returns docstring sections in _Unset sentinel (DAR201) Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> * fix(rbac): deduplicate token_teams in cache key and strengthen regression test Address two findings from code review: 1. Cache key deduplication: token_teams with duplicates (e.g. ["team-a", "team-a"]) now produces the same cache key as the deduplicated form (["team-a"]), preventing avoidable cache fragmentation. Added test_cache_key_deduplicates_token_teams. 2. Strengthen test_check_permission_narrowed_session_restricts_to_token_teams: replaced unrealistic "teams.*" wildcard permissions with explicit permissions matching bootstrap_db.py built-in roles (teams.create, teams.read, etc.). The previous test passed even when team-B roles leaked because "teams.create" != "teams.*" in exact string matching. The new test would correctly fail if role filtering broke. Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> --------- Signed-off-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com> Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> Co-authored-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com> Co-authored-by: Mihai Criveti <crivetimihai@gmail.com>
…paths (IBM#3932) * fix(security): enforce token_teams narrowing across all Layer 2 RBAC paths Signed-off-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com> * fix(security): close remaining token_teams gaps in RBAC paths - PermissionChecker.has_admin_permission: forward token_teams to check_admin_permission (middleware/rbac.py) - has_admin_permission on PermissionService: add token_teams param, suppress admin bypass for public-only tokens (permission_service.py) - AdminAuthMiddleware: forward token_teams to has_admin_permission (main.py) - get_my_permissions endpoint: forward token_teams to get_user_permissions (routers/rbac.py) - _get_user_roles: reject explicit team_id when token_teams=[] (public-only tokens must not access team-specific roles) - Fix stale test asserting token_teams=[] retains team perms (contradicts Option A strict isolation) - Fix stale test asserting team_id ignores token_teams (team_id is now validated against token_teams) - Add regression tests for all new paths Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> * fix(tests): update has_admin_permission assertions for token_teams param Update test assertions in test_permission_service.py and test_main_extended.py to include the new token_teams parameter added to has_admin_permission(). Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> * fix(security): close remaining token_teams gaps from codex review Address all 4 findings from the Codex code review: Finding 1 (HIGH) — _get_caller_permissions admin bypass: - Only treat admin as unrestricted when token_teams is None. Narrowed/public-only admin sessions now derive permissions through the token-aware path (routers/tokens.py). - create_token auto-inheritance now applies to narrowed admins too. Finding 2 (HIGH) — scope_id IS NULL team roles on public-only: - _get_user_roles with team_id=None, include_all_teams=False now excludes scope='team', scope_id=NULL roles when token_teams=[] (permission_service.py). Finding 3 (HIGH) — list_all_tokens/admin_revoke_token admin guard: - Both endpoints now require un-narrowed admin (token_teams is None). Narrowed and public-only admin sessions are rejected with 403 (routers/tokens.py). Finding 4 (LOW) — cache key collision None vs []: - Cache keys now use __public__ sentinel for token_teams=[] and include token_teams in team_id cache keys (permission_service.py). Add 12 regression tests covering all fixed paths. Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> --------- Signed-off-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com> Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> Co-authored-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com> Co-authored-by: Mihai Criveti <crivetimihai@gmail.com>
…detection plugin (IBM#3906) * feat(encoded-exfil): production-ready encoded exfiltration detection with full Rust parity - Add allowlist_patterns config with regex validation to suppress false positives - Add extra_sensitive_keywords and extra_egress_hints for configurable detection tuning - Add nested encoding detection with max_decode_depth (peels base64-of-hex etc.) - Add per_encoding_score for per-encoding suspicion thresholds - Add parse_json_strings to detect encoded payloads inside JSON string values - Add resource_post_fetch hook to scan fetched resources for exfiltration - Add container recursion depth limiting (max_recursion_depth) - Add detection logging (log_detections flag, no sensitive content in logs) - Port all features to Rust with full parity via persistent ExfilDetectorEngine class - Add compare_performance.py: Python vs Rust benchmarks (4.3x-12.1x speedup) - Add 112 TDD tests: config validation, bypass resistance, parity, integration, nested encoding, JSON parsing, and xfail-documented limitations - Full README rewrite with config reference, tuning guide, and worked examples Closes IBM#3807 Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com> * chore: regenerate detect-secrets baseline after rebase onto main Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com> * fix: mark compare_performance.py as executable to match shebang Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com> * fix: resolve CI lint and Rust fmt failures - Replace object.__setattr__() with setattr() to fix ruff PLC2801 - Rename unused __context to _context to fix vulture - Run cargo fmt on Rust test code Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com> * chore: audit new detect-secrets baseline entries as false positives Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com> * fix: resolve CI clippy and pylint failures - Collapse nested if statements in Rust to satisfy clippy collapsible_if - Add pylint disable for model_post_init arguments-differ Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com> * fix(encoded-exfil): prevent double-counting when JSON-within-strings is enabled When a string is valid JSON, scan only the parsed structure — do not also scan the raw text. This prevents the same encoded value from being counted twice (once in the raw string, once in the parsed JSON value), which could incorrectly trip min_findings_to_block. Add regression test verifying a single secret in a JSON string produces exactly 1 finding, not 2. Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com> * fix(encoded-exfil): address reviewer feedback — JSON guard, Rust tests, README fixes - Add JSON parse heuristic: only attempt json.loads if string starts with { or [ and is within max_scan_string_length (Python + Rust) - Add Rust-path test coverage for per-encoding thresholds, JSON parsing, heuristic skip, and malformed JSON (8 new parametrized tests) - Fix README: correct wheel name to mcpgateway-encoded-exfil-detection, remove implemented features from Known Limitations, add Performance section - Include pattern index in allowlist validation error messages - Regenerate secrets baseline Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com> * fix(encoded-exfil): deduplicate boundary_chars allocation, add allowlist partial match test - Hoist core_chars.replace('=', "") to a single let above both boundary checks in has_valid_boundaries() to avoid redundant string allocation - Add test verifying allowlist partial match suppresses detection Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com> * fix(encoded-exfil): scan dict keys for encoded secrets, prevent JSON type mutation - Scan dict keys as strings when len >= min_encoded_length to detect secrets used as JSON object keys (both Python and Rust) - Prevent JSON-within-strings from mutating return type: scan raw text first (preserves original string), then parse JSON for additional findings with deduplication by match preview - Add tests for key scanning and type preservation - Update JSON test assertions to verify string return type Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com> * fix(encoded-exfil): lowercase extra keywords/hints for case-insensitive matching - Fix parity bug: extra_sensitive_keywords and extra_egress_hints were not lowercased in Python, causing case-insensitive matching to fail when users configured mixed-case keywords. Rust already lowercases both (kw.to_lowercase() / h.to_lowercase()). - Update module docstring to list all 3 hooks (was missing resource_post_fetch). - Add pragma: no cover to Rust-only code paths (import, scan, engine init). - Add regression tests for mixed-case extra keywords and egress hints. - Add test for max_recursion_depth container depth guard. - Add test for JSON dedup path (Unicode-escaped base64 only found via JSON parse). - Achieve 100% differential test coverage on Python plugin. - Update .secrets.baseline for new test file entries (false positives). Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> * baseline Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> --------- Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com> Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> Co-authored-by: Mihai Criveti <crivetimihai@gmail.com>
Signed-off-by: Agnetha <agnethakamath@gmail.com>
|
Closing because the linked issue #2216 ([FEATURE][SECURITY]: Container vulnerability scanner — Trivy/Grype integration) was closed as NOT_PLANNED. The feature has been explicitly declined; this PR no longer has a merge target. |
🧱 New Plugin
🔗 Closes
Closes #2216
🚀 Summary
Scans container images for CVEs using Trivy or Grype before they are registered or deployed through the gateway, blocking or auditing based on configurable severity policy.
Included:
(mcpgateway/routers/container_scanner_router.py): health endpoint, scan list with most-recent-first ordering, per-image lookup by ref or digest, full vulnerabilities[] array with fixed_version and description in response(container_scanner.py): full pipeline: cache → auth → scanner → policy → persist, graceful fail_open/fail_closed error handling, PluginViolation construction on block(mcpgateway/plugins/framework/hooks/gateway.py): server_pre_register and runtime_pre_deploy hook types with ServerPreRegisterPayload / RuntimePreDeployPayload, registered into global hook registry on module importTesting
Unit tests (109 passed)
Cover each component in isolation with all external calls mocked:
pytest tests/unit/plugins/test_container_scanner/ -vIntegration tests (18 passed)
Covers 2 end-to-end scenarios with real policy/repository/evaluator, and scanner CLI mocked:
test_api.py: health endpoint empty/populated state, scan list ordering (most-recent-first), per-image lookup by ref and digest, 404 on unknown image, unauthenticated requests rejectedtest_hook_pipeline.py: hook fires and result stored in shared singleton, blocked/unblocked results reflected correctly in API response, disabled mode skips runner and stores no result, CRITICAL vuln triggers block with PluginViolation, unblocked result has continue_processing=Truepytest tests/integration/test_container_scanner/ -v🧪 Checks
make lint pluginspassesmake testpasses