[EXPERIMENTAL] : Add Rust CPEX secrets detection experiment

[gandhipratik203]

Summary

• adds an isolated experimental Rust-native CPEX 0.2.2 secrets detection crate under experiments/secrets_detection_rust
• uses a self-contained Cargo workspace/toolchain so the spike does not change the production plugin or root workspace
• ports the existing scanner/config/pattern behavior into a native CPEX PluginFactory + shared core + stage-bound CMF handlers
• registers separate stage-bound handlers for cmf.prompt_pre_fetch, cmf.tool_pre_invoke, cmf.tool_post_invoke, and cmf.resource_post_fetch
• adds an example host at experiments/secrets_detection_rust/examples/manual_probe.rs for manual PluginManager probing
• adds a committed external crate-consumer smoke fixture under experiments/secrets_detection_rust/smoke/crate_consumer

Scope

This is intentionally a draft spike and is not intended to merge as-is. The goal is to understand how a Rust-native secrets detection plugin integrates with the new CPEX framework and what parity looks like against the current plugin behavior.

Related to #117. This spike is not intended to close the issue as-is.

No production plugin code under plugins/rust/python-package/secrets_detection is changed by this PR.

Coverage Added

• PluginManager registration/loading for validator/secrets-detection
• allow/redact/block behavior across the supported CMF hook stages
• field allowlist/denylist behavior for prompt/tool arguments and tool result JSON
• resource direct text scanning, including the current behavior where field filters are ignored for direct text
• clean payload no-op behavior
• direct-handler metadata gating on trace context
• direct-handler redacted payload behavior for blocked results
• manual probe scenarios for tool redaction, tool blocking, prompt filters, tool-result filters, and resource blocking
• crate-smoke-equivalent PluginManager tests for invalid config rejection and nested allowlist/denylist behavior
• external crate-consumer smoke fixture that depends on the spike crate by relative path

Manual Testing

From experiments/secrets_detection_rust:

cargo run --example manual_probe -- tool-redact
cargo run --example manual_probe -- tool-block
cargo run --example manual_probe -- prompt-filter
cargo run --example manual_probe -- tool-allow-deny
cargo run --example manual_probe -- tool-result-filter
cargo run --example manual_probe -- resource-block

The probe registers the plugin factory, loads CPEX YAML config, invokes the selected CMF hook through PluginManager, and prints the resulting PipelineResult fields plus the returned CMF content.

Manual Crate-Level Smoke Test

This is the Rust equivalent of the wheel-level smoke test from #134. Instead of installing a Python wheel into a clean venv, it uses a standalone Cargo consumer committed under the spike directory and depends on the spike crate by relative path.

Tested from:

cd experiments/secrets_detection_rust/smoke/crate_consumer
cargo run

The consumer uses only the public crate exports:

use secrets_detection_rust::{SecretsDetectionFactory, KIND};

What it verifies:

• the crate can be consumed externally as a Cargo dependency
• invalid field config is rejected at plugin load
• cmf.tool_pre_invoke redacts tool arguments
• cmf.tool_pre_invoke blocks tool arguments
• cmf.prompt_pre_fetch applies field filters to prompt arguments
• field_allowlist=["accounts"] scans/redacts accounts.keep
• field_denylist=["accounts.skip"] excludes accounts.skip
• cmf.tool_post_invoke applies field filters to tool result JSON
• cmf.resource_post_fetch blocks direct resource text
• fields outside the allowlist, such as ignored, remain unchanged
• the original CMF payload remains unchanged after redaction

Result:

invalid config rejected: load failed: configuration error: field_allowlist path "bad." must not start or end with '.'
tool-redact passed
tool-block passed
prompt-filter passed
original args:
{
  "accounts": {
    "keep": "<test-input-a>",
    "skip": "<test-input-b>"
  },
  "ignored": "<test-input-c>"
}
modified args:
{
  "accounts": {
    "keep": "<redacted>",
    "skip": "<test-input-b>"
  },
  "ignored": "<test-input-c>"
}
tool-allow-deny passed
tool-result-filter passed
resource-block passed
crate-level smoke tests passed

Known CPEX 0.2.2 Gaps

The spike pins two framework-boundary gaps with explicit tests:

• PluginResult.metadata is produced by the plugin at the handler layer, but is dropped by the manager/executor path when returned as a PipelineResult.
• blocked/denied handler results can carry a redacted modified_payload, but PluginManager does not currently surface that payload on denied results.

Validation

• cargo test in experiments/secrets_detection_rust: 47 passed (34 inline/unit tests + 13 PluginManager integration tests)
• manual probe scenarios passed: tool-redact, tool-block, prompt-filter, tool-allow-deny, tool-result-filter, resource-block
• external crate-consumer smoke fixture in experiments/secrets_detection_rust/smoke/crate_consumer: passed, including the manual probe hook scenarios
• existing plugin baseline checks run separately:
  • cargo test -p secrets_detection: 96 passed
  • make install: passed
  • make test-integration: 114 passed
  • make test-unit: not available because cargo-nextest is not installed

[lucarlig]

Experimental scope looks good. Leaving only non-blocking nitpicks for the dataplane trial.

[lucarlig]

nit (non-blocking for the spike): this deep-clones the complete CMF payload before we know whether redaction is enabled or any finding exists. Even clean/default (redact: false) invocations therefore allocate proportional to payload size, which could skew the dataplane experiment for large tool/resource results. Could we scan the borrowed payload first and clone lazily only when returning a modification?

[lucarlig]

nit: these are four fixed hook names, so leaking a fresh String on every factory creation seems unnecessary. CPEX may invoke the factory again for route-level config overrides, so this can accumulate beyond startup. Could the match return the corresponding static literal or CMF hook constant alongside Stage?

[lucarlig]

nit/versioning: if 0.2.2 is the release carrying the breaking plugin API/boundary changes this spike is adapting to, this should really be CPEX 0.3.0. A 0.2.x patch communicates compatibility, while this is breaking for plugin implementers/consumers.

[Lang-Akshay]

Thanks for the PR @gandhipratik203

Please address following blocking Changes

#	Area	File	Line	Blocking reason	Required change
1	Security	experiments/secrets_detection_rust/src/patterns.rs; src/scanner.rs	48; 129-137	High / CWE-200: private_key_block matches only a PEM BEGIN marker. With redact: true and non-blocking operation, the key body and END marker remain in the forwarded payload.	Detect and replace the complete bounded PEM block (header, body, footer), or fail closed whenever a private-key marker is found. Add coverage for multiline PEM redaction and blocking.
2	Security	experiments/secrets_detection_rust/src/lib.rs	90-94; 333-349	High / CWE-693: the factory accepts non-blocking CPEX modes even when block_on_detection is enabled. CPEX suppresses continue_processing=false for Transform/Audit/FireAndForget; the deny fallback can attach the original payload, allowing detected secrets to continue.	Reject enforcement configurations paired with non-enforcing modes; require a mode that honors denial (or fail closed), and never attach the original unredacted payload as a deny fallback.

[gandhipratik203]

Thanks Akshay. This PR is scoped to validating the Rust CPEX integration path, with production-level secrets detection behavior deferred to follow-up hardening work.

Agreed on both points. The private key matcher should be tightened before production use, and block_on_detection with non-enforcing modes is a valid safety concern. Current Python CPEX suppresses blocking in TRANSFORM / AUDIT / FIRE_AND_FORGET rather than rejecting config at load time.

I’ve created #162 to track these production hardening items.

[Lang-Akshay]

Blocking Changes

#	Area	File	Line	Blocking reason	Required change
1	Security	src/patterns.rs	48	CWE-522: private_key_block regex matches only the -----BEGIN … PRIVATE KEY----- header line. detect_and_redact redacts only that single matched span; the entire key body and -----END … PRIVATE KEY----- remain in output when redact=true. A payload containing a PEM private key is silently under-redacted.	Extend the pattern to cover the full PEM block (header through footer, including the base64 body), or post-process to strip everything between the matched header and the corresponding footer line.
2	Security	src/lib.rs	93	CWE-200: On deny (should_block = true), result.modified_payload is set to `scan.modified_payload.or_else(

[gandhipratik203]

closing as the realated work is got moved to the pr here: contextforge-org/contextforge-data-plane#74