feat: Add SpanAttributeCustomizer plugin for customizable OpenTelemetry span attributes

[vishu-bh]

Summary

Implements GitHub issue #4274 - Add customizable OpenTelemetry span attributes to observability system.

Changes

Plugin Implementation

• Created SpanAttributeCustomizerPlugin with comprehensive feature set
• Global attributes for all spans
• Per-tool attribute overrides
• Attribute transformations (hash, uppercase, lowercase, truncate)
• Conditional attributes based on tool name matching
• Attribute removal for privacy/compliance

Integration

• Modified ObservabilityService to accept plugin context
• Added context parameter to start_span(), trace_span(), and trace_tool_invocation()
• Producer-consumer pattern via GlobalContext.state

Testing

• Unit Tests: 28 tests with 92% code coverage ✅
• Integration Tests: 7 end-to-end tests ✅
• All unit tests passing

Configuration

• Added example configuration to plugins/config.yaml
• Comprehensive README with usage examples

Files Changed

• plugins/span_attribute_customizer/ (new plugin)
• mcpgateway/services/observability_service.py (integration)
• plugins/config.yaml (configuration example)
• tests/unit/mcpgateway/plugins/span_attribute_customizer/ (unit tests)
• tests/integration/plugins/ (integration tests)

Testing Plan

• ✅ Unit tests: 28/28 passing, 92% coverage
• ✅ Ruff linting: Passing
• ✅ Full linting suite: bandit, interrogate, pylint

Closes

Closes #4274

[Lang-Akshay]

Thanks allot for the PR @vishu-bh . Really appreciate it. Please can you implement following changes.

Security

The plugin operates below the auth/RBAC layer (observability data only). No auth invariants are violated. Security concerns are scoped to PII masking correctness and state handling.

#	File	Line	Severity	CWE	Description
1	span_attribute_customizer.py:130	130	High	CWE-916/327	SHA-256 truncated to 16 hex chars, no salt — insufficient for PII masking; rainbow-table reversible for common emails
2	span_attribute_customizer.py:211	211–212	High	CWE-668	resource_pre_fetch only sets custom_span_attributes but does not reset remove_span_attributes / span_attribute_mapping; stale tool-invocation state bleeds into resource spans
3	span_attribute_customizer.py:105	105	Medium	CWE-20	condition.split("==") raises ValueError silently caught for any condition value containing ==; intended security attributes (audit_required, compliance_level) silently dropped
4	config_schema.py:40	40–58	Medium	CWE-400	global_attributes: Dict[str, Any] and conditions[*].add accept unbounded nested objects; OTEL SDKs only accept str/int/float/bool — oversized or wrong-typed values cause silent data loss
5	observability_service.py:505 + manager.py:447	multiple	Medium	CWE-668	Attribute rename loop duplicated verbatim 3 times across two files; future security behavior added to one copy will silently diverge
6	span_attribute_customizer.py:101	101	Low	CWE-95	Comment "can be enhanced with safe eval" invites eval() injection by future developers
7	span_attribute_customizer.py:173	173–180	Low	CWE-362	GlobalContext.state is an unguarded plain dict; non-atomic read-modify-write in multi-thread Gunicorn/Granian modes
8	span-attribute-customization.md:208	208, 258	Low	CWE-94	Docs show "{{ tenant_id }}" / "{{ POD_NAME }}" implying Jinja2 support that does not exist; invites SSTI if ever implemented without review

Remediation suggestions

• Finding 1: Use HMAC-SHA-256 with settings.auth_encryption_secret, keep ≥32 hex chars. Document it as pseudonymization, not anonymization.
• Finding 2: In resource_pre_fetch, set all three state keys (custom_span_attributes, remove_span_attributes, span_attribute_mapping) explicitly.
• Finding 3: Change condition.split("==") → condition.split("==", 1). Add startup validation against ^[a-zA-Z_.]+\s*==\s*"[^"]*"$.
• Finding 4: Add Pydantic field_validator on attribute dicts limiting values to str|int|float|bool and capping key/count sizes.
• Finding 5: Extract to a single apply_attribute_mapping(attrs, mapping) helper.

---

Redundant Code

#	File	Line(s)	Type	Description	Suggestion
1	manager.py:447 + observability_service.py:510	multiple	Duplicated logic	Attribute rename dict comprehension copy-pasted verbatim across 3 sites	Extract to apply_attribute_mapping() helper
2	test_span_attribute_mapping_integration.py:60	60–90	Duplicated test logic	Several integration tests manually replicate the manager.py/observability_service.py mapping loop rather than testing the actual implementation	Replace with tests that call the real code and assert observable outcomes
3	test_upstream_session_registry_attribute_mapping	full function	Dead test	Only asserts that a key exists in a manually-constructed dict — no actual behavior under test	Remove or replace with a meaningful assertion
4	span_attribute_customizer.py:205	205–209	Empty hook	tool_post_invoke returns immediately with no action; declared comment # Can add attributes based on tool results is stale intent	Either implement or remove the method and the hook from the hooks list

[vishu-bh]

Thanks @Lang-Akshay, addressed the findings.

[Lang-Akshay]

Thanks for the addressing previously mentioned issue @vishu-bh

• ✅ SHA-256 → HMAC-SHA-256
• ✅ split("==", 1)
• ✅ global_attributes field_validator
• ✅ apply_attribute_mapping helper extracted
• ✅ plugin disabled by default
• ✅ resource_pre_fetch resets all 3 state keys

---

Please address this issue

Finding 1: Race Condition in State Access

File: span_attribute_customizer.py:195
Lines: 195–228
Severity: High
CWE: CWE-362 (Concurrent Execution using Shared Resource with Improper Synchronization)

Description

_state_lock guards writes only. observability_service.start_span reads all 3 state keys locklessly. Concurrent resource_pre_fetch (which resets span_attribute_mapping={}) can execute between consecutive reads in start_span, producing a torn read: tool-phase custom_attrs + empty mapping/removal from resource phase.

Remediation

Snapshot all 3 state keys under _state_lock before passing to start_span, or use contextvars.ContextVar per async task.

Example Implementation:

with self._state_lock:
    snapshot = {
        'custom_attrs': self._custom_attrs.copy(),
        'mapping': self._get_attribute_mapping().copy(),
        'removal': self._attributes_to_remove.copy()
    }
# Pass snapshot to start_span

---

Finding 2: Insufficient Input Validation

File: config_schema.py:22
Lines: 22, 31
Severity: Medium
CWE: CWE-20 (Improper Input Validation)

Description

ToolOverride.attributes and ConditionalAttribute.add are typed Dict[str, Any] with no validators — allows nested objects, lists, None. OTEL SDK silently rejects non-primitive types. No entry count or value size limits.

Remediation

Add @field_validator to ToolOverride.attributes and ConditionalAttribute.add mirroring validate_global_attributes.

Example Implementation:

@field_validator("attributes")
@classmethod
def validate_attributes(cls, v: Dict[str, Any]) -> Dict[str, Any]:
    """Validate that all attribute values are primitives."""
    if not v:
        return v
    for key, value in v.items():
        if not isinstance(value, (str, int, float, bool)):
            raise ValueError(
                f"Attribute '{key}' has invalid type {type(value).__name__}. "
                "Only str, int, float, bool are allowed."
            )
    return v

---

Finding 3: Inconsistent Attribute Mapping Application

File: span_attribute_customizer.py:219
Lines: 219–228
Severity: Medium
CWE: CWE-840 (Business Logic Errors)

Description

resource_pre_fetch resets span_attribute_mapping={} — attribute renaming never applies to resource spans. README/docs state mapping applies to all spans. Operators configuring renaming for privacy/compliance receive no sanitisation on resource spans — silent security-policy bypass.

Evidence

1. README.md (line ~40) claims:

attribute_mapping:
  # Rename plugin span attributes
  "plugin.name": "controls.artifact.name"
  "plugin.uuid": "controls.artifact.id"

Key Points:

• Applies to ALL spans (tools, resources, plugins)
• Original names are replaced with mapped names

2. Code reality (span_attribute_customizer.py:219):

async def resource_pre_fetch(...):
    # Reset all state keys to prevent stale tool-invocation state
    with self._state_lock:
        context.global_context.state["custom_span_attributes"] = custom_attrs
        context.global_context.state["remove_span_attributes"] = []
        context.global_context.state["span_attribute_mapping"] = {}  # ← EMPTY!

3. Security Impact:

• Operators configure attribute renaming for compliance (e.g., "user.email" → "user.id_hash")
• They believe mapping applies to ALL spans per README documentation
• Resource spans retain original attribute names, potentially exposing sensitive data
• Silent failure - no warning that mapping doesn't apply to resources
• This constitutes a security-policy bypass for compliance/privacy requirements

Remediation

Pass self._get_attribute_mapping() through resource_pre_fetch instead of {}, or document intentionally and test it.

Option 1 - Apply mapping to resource spans:

async def resource_pre_fetch(
    self, payload: ResourcePreFetchPayload, context: PluginContext
) -> ResourcePreFetchResult:
    with self._state_lock:
        self._custom_attrs = {}
        self._span_attribute_mapping = self._get_attribute_mapping()  # Keep mapping
        self._attributes_to_remove = set()

Option 2 - Document and test the intentional behavior:

• Update README to explicitly state: "Attribute mapping applies only to tool and prompt spans, not resource spans"
• Add test case verifying resource spans use original attribute names

---

Finding 4: Resource Exhaustion via Unbounded Configuration

File: config_schema.py:57
Lines: 57–70
Severity: Medium
CWE: CWE-400 (Uncontrolled Resource Consumption)

Description

tool_overrides, conditions, transformations, and attribute_mapping have no entry-count limits. Config-write access (admin role) can push thousands of entries, causing O(n²) per-span work — sustained DoS.

Remediation

Add count limits (≤ 200 tool_overrides, ≤ 50 conditions/transformations, ≤ 100 attribute_mapping).

Example Implementation:

@field_validator("tool_overrides")
@classmethod
def validate_tool_overrides_count(cls, v: List[ToolOverride]) -> List[ToolOverride]:
    """Limit tool_overrides to prevent resource exhaustion."""
    if len(v) > 200:
        raise ValueError(
            f"tool_overrides count ({len(v)}) exceeds maximum allowed (200)"
        )
    return v

@field_validator("conditions")
@classmethod
def validate_conditions_count(cls, v: List[ConditionalAttribute]) -> List[ConditionalAttribute]:
    """Limit conditions to prevent resource exhaustion."""
    if len(v) > 50:
        raise ValueError(
            f"conditions count ({len(v)}) exceeds maximum allowed (50)"
        )
    return v

@field_validator("transformations")
@classmethod
def validate_transformations_count(cls, v: List[AttributeTransformation]) -> List[AttributeTransformation]:
    """Limit transformations to prevent resource exhaustion."""
    if len(v) > 50:
        raise ValueError(
            f"transformations count ({len(v)}) exceeds maximum allowed (50)"
        )
    return v

@field_validator("attribute_mapping")
@classmethod
def validate_attribute_mapping_count(cls, v: Dict[str, str]) -> Dict[str, str]:
    """Limit attribute_mapping to prevent resource exhaustion."""
    if len(v) > 100:
        raise ValueError(
            f"attribute_mapping count ({len(v)}) exceeds maximum allowed (100)"
        )
    return v

[Lang-Akshay]

Thanks for the PR @vishu-bh

[brian-hussey]

Approved based on other reviews from the team.