Update auto instrumentation logic for Semantic Kernel #136

nikhilNava · 2026-01-26T08:57:21Z

Task
Review SK auto instrumentation logic and clean up attributes

Solution
In SK the spans cannot be updated on_end() of span processor. They are readable spans.
Our only option is to update the span before its exported. Here I introduce a new pattern where we wrap the batch processor.
We make updates to the span on the on_end() of class _EnrichingBatchSpanProcessor.
This pattern is extensible and extensions for auto instrumentation need only implement the enrichment. If no enrichment the _EnrichingBatchSpanProcessor is a pass through

_EnrichingBatchSpanProcessor
         │
         └── extends BatchSpanProcessor (from OpenTelemetry SDK)
                     │
                     └── holds reference to SpanExporter (the _Agent365Exporter)

Exported spans

{
  "resourceSpans": [
    {
      "resource": {
        "attributes": {
          "telemetry.sdk.language": "python",
          "telemetry.sdk.name": "opentelemetry",
          "telemetry.sdk.version": "1.37.0",
          "service.namespace": "SemanticKernelTesting",
          "service.name": "SemanticKernelTracing"
        }
      },
      "scopeSpans": [
        {
          "scope": {
            "name": "semantic_kernel.utils.telemetry.agent_diagnostics.decorators",
            "version": ""
          },
          "spans": [
            {
              "traceId": "51db5763e95ddb7010daa206e281cbc7",
              "spanId": "cd226c5bd013f57a",
              "parentSpanId": null,
              "name": "invoke_agent SemanticKernelAgent",
              "kind": "INTERNAL",
              "startTimeUnixNano": 1769437044042171700,
              "endTimeUnixNano": 1769437045299979100,
              "attributes": {
                "gen_ai.operation.name": "invoke_agent",
                "gen_ai.agent.id": "39391836-ba5e-4ed3-bb46-890e3ceccfa1",
                "gen_ai.agent.name": "SemanticKernelAgent",
                "gen_ai.tool.definitions": "[{\"type\": \"function\", \"function\": {\"name\": \"Math-calculate\", \"description\": \"Perform basic arithmetic calculations.\", \"parameters\": {\"type\": \"object\", \"properties\": {\"expression\": {\"type\": \"string\", \"description\": \"A mathematical expression to evaluate, e.g., '2 + 3 * 4'\"}}, \"required\": [\"expression\"]}}}, {\"type\": \"function\", \"function\": {\"name\": \"Menu-get_item_price\", \"description\": \"Provides the price of the requested menu item.\", \"parameters\": {\"type\": \"object\", \"properties\": {\"menu_item\": {\"type\": \"string\", \"description\": \"The name of the menu item.\"}}, \"required\": [\"menu_item\"]}}}, {\"type\": \"function\", \"function\": {\"name\": \"Menu-get_specials\", \"description\": \"Provides a list of specials from the menu.\", \"parameters\": {\"type\": \"object\", \"properties\": {}, \"required\": []}}}]",
                "operation.source": "SDK",
                "tenant.id": "badf1f56-284d-4dc5-ac59-0dd53900e743",
                "correlation.id": "7ff6dca0-917c-4bb0-b31a-794e533d8aad",
                "gen_ai.conversation.id": "4cc5b165-2e9c-4af1-8385-b976c897a776",
                "gen_ai.agent.upn": "Sample Agent UPN",
                "gen_ai.agent.applicationid": "Sample Blueprint ID",
                "gen_ai.caller.client.ip": "0.0.0.0",
                "gen_ai.channel.name": "msteams",
                "gen_ai.caller.id": "a92962f3-9ed4-4bcd-9ae0-ad0002b6ca76",
                "gen_ai.caller.name": "Alex Wilber",
                "gen_ai.caller.upn": "Sample UPN",
                "gen_ai.execution.type": "humantoagent",
                "gen_ai.input.messages": "[\"Compute 15 % 4\"]",
                "gen_ai.output.messages": "[\"The result of \\\\( 15 \\\\mod 4 \\\\) (15 % 4) is 3.\"]"
              },
              "events": null,
              "links": null,
              "status": {
                "code": "UNSET",
                "message": ""
              }
            }
          ]
        }
      ]
    }
  ]
}

Execute tool and Inference span

{
  "resourceSpans": [
    {
      "resource": {
        "attributes": {
          "telemetry.sdk.language": "python",
          "telemetry.sdk.name": "opentelemetry",
          "telemetry.sdk.version": "1.37.0",
          "service.namespace": "SemanticKernelTesting",
          "service.name": "SemanticKernelTracing"
        }
      },
      "scopeSpans": [
        {
          "scope": {
            "name": "semantic_kernel.utils.telemetry.model_diagnostics.decorators",
            "version": ""
          },
          "spans": [
            {
              "traceId": "51db5763e95ddb7010daa206e281cbc7",
              "spanId": "a4f17d816959d7c9",
              "parentSpanId": "f2783816d140d4b8",
              "name": "chat gpt-4o-mini",
              "kind": "INTERNAL",
              "startTimeUnixNano": 1769437044049184800,
              "endTimeUnixNano": 1769437044625475200,
              "attributes": {
                "operation.source": "SDK",
                "tenant.id": "badf1f56-284d-4dc5-ac59-0dd53900e743",
                "correlation.id": "7ff6dca0-917c-4bb0-b31a-794e533d8aad",
                "gen_ai.conversation.id": "4cc5b165-2e9c-4af1-8385-b976c897a776",
                "gen_ai.agent.id": "a70248c0-9daa-4b4a-a498-b3c95c8422a5",
                "gen_ai.agent.name": "Perplexity02 Agent",
                "gen_ai.agent.upn": "Sample Agent UPN",
                "gen_ai.agent.applicationid": "Sample Blueprint ID",
                "gen_ai.caller.client.ip": "0.0.0.0",
                "gen_ai.channel.name": "msteams",
                "gen_ai.operation.name": "chat",
                "gen_ai.system": "openai",
                "gen_ai.request.model": "gpt-4o-mini",
                "server.address": "https://nikhilc-aoai.cognitiveservices.azure.com/openai/deployments/gpt-4o-mini/",
                "gen_ai.response.id": "chatcmpl-D2Hh6sXy7MEGfW9t7xAPXmGeU0JpK",
                "gen_ai.response.finish_reason": "FinishReason.TOOL_CALLS",
                "gen_ai.usage.input_tokens": 194,
                "gen_ai.usage.output_tokens": 19
              },
              "events": null,
              "links": null,
              "status": {
                "code": "UNSET",
                "message": ""
              }
            },
            {
              "traceId": "51db5763e95ddb7010daa206e281cbc7",
              "spanId": "a07692ffa1e321d5",
              "parentSpanId": "f2783816d140d4b8",
              "name": "chat gpt-4o-mini",
              "kind": "INTERNAL",
              "startTimeUnixNano": 1769437044635243700,
              "endTimeUnixNano": 1769437045299979100,
              "attributes": {
                "operation.source": "SDK",
                "tenant.id": "badf1f56-284d-4dc5-ac59-0dd53900e743",
                "correlation.id": "7ff6dca0-917c-4bb0-b31a-794e533d8aad",
                "gen_ai.conversation.id": "4cc5b165-2e9c-4af1-8385-b976c897a776",
                "gen_ai.agent.id": "a70248c0-9daa-4b4a-a498-b3c95c8422a5",
                "gen_ai.agent.name": "Perplexity02 Agent",
                "gen_ai.agent.upn": "Sample Agent UPN",
                "gen_ai.agent.applicationid": "Sample Blueprint ID",
                "gen_ai.caller.client.ip": "0.0.0.0",
                "gen_ai.channel.name": "msteams",
                "gen_ai.operation.name": "chat",
                "gen_ai.system": "openai",
                "gen_ai.request.model": "gpt-4o-mini",
                "server.address": "https://nikhilc-aoai.cognitiveservices.azure.com/openai/deployments/gpt-4o-mini/",
                "gen_ai.response.id": "chatcmpl-D2Hh6uX6mcUFQzbxe7Y5KcsTOCoJ9",
                "gen_ai.response.finish_reason": "FinishReason.STOP",
                "gen_ai.usage.input_tokens": 222,
                "gen_ai.usage.output_tokens": 24
              },
              "events": null,
              "links": null,
              "status": {
                "code": "UNSET",
                "message": ""
              }
            }
          ]
        },
        {
          "scope": {
            "name": "semantic_kernel.functions.kernel_function",
            "version": ""
          },
          "spans": [
            {
              "traceId": "51db5763e95ddb7010daa206e281cbc7",
              "spanId": "6fe746e18fe1a95b",
              "parentSpanId": "f2783816d140d4b8",
              "name": "execute_tool Math-calculate",
              "kind": "INTERNAL",
              "startTimeUnixNano": 1769437044635243700,
              "endTimeUnixNano": 1769437044635243700,
              "attributes": {
                "gen_ai.operation.name": "execute_tool",
                "gen_ai.tool.name": "Math-calculate",
                "gen_ai.tool.call.id": "call_sU7gVLxj0n05toJFxuii0seq",
                "gen_ai.tool.description": "Perform basic arithmetic calculations.",
                "operation.source": "SDK",
                "tenant.id": "badf1f56-284d-4dc5-ac59-0dd53900e743",
                "correlation.id": "7ff6dca0-917c-4bb0-b31a-794e533d8aad",
                "gen_ai.conversation.id": "4cc5b165-2e9c-4af1-8385-b976c897a776",
                "gen_ai.agent.id": "a70248c0-9daa-4b4a-a498-b3c95c8422a5",
                "gen_ai.agent.name": "Perplexity02 Agent",
                "gen_ai.agent.upn": "Sample Agent UPN",
                "gen_ai.agent.applicationid": "Sample Blueprint ID",
                "gen_ai.caller.client.ip": "0.0.0.0",
                "gen_ai.channel.name": "msteams",
                "gen_ai.tool.call.arguments": "{\"expression\": \"15 % 4\"}",
                "gen_ai.tool.call.result": "3",
                "gen_ai.tool.arguments": "{\"expression\": \"15 % 4\"}",
                "gen_ai.event.content": "3"
              },
              "events": null,
              "links": null,
              "status": {
                "code": "UNSET",
                "message": ""
              }
            }
          ]
        },
        {
          "scope": {
            "name": "semantic_kernel.connectors.ai.chat_completion_client_base",
            "version": ""
          },
          "spans": [
            {
              "traceId": "51db5763e95ddb7010daa206e281cbc7",
              "spanId": "f2783816d140d4b8",
              "parentSpanId": "cd226c5bd013f57a",
              "name": "AutoFunctionInvocationLoop",
              "kind": "INTERNAL",
              "startTimeUnixNano": 1769437044049184800,
              "endTimeUnixNano": 1769437045299979100,
              "attributes": {
                "operation.source": "SDK",
                "tenant.id": "badf1f56-284d-4dc5-ac59-0dd53900e743",
                "correlation.id": "7ff6dca0-917c-4bb0-b31a-794e533d8aad",
                "gen_ai.conversation.id": "4cc5b165-2e9c-4af1-8385-b976c897a776",
                "gen_ai.agent.id": "a70248c0-9daa-4b4a-a498-b3c95c8422a5",
                "gen_ai.agent.name": "Perplexity02 Agent",
                "gen_ai.agent.upn": "Sample Agent UPN",
                "gen_ai.agent.applicationid": "Sample Blueprint ID",
                "gen_ai.caller.client.ip": "0.0.0.0",
                "gen_ai.channel.name": "msteams",
                "sk.available_functions": "Math-calculate,Menu-get_item_price,Menu-get_specials"
              },
              "events": null,
              "links": null,
              "status": {
                "code": "UNSET",
                "message": ""
              }
            }
          ]
        }
      ]
    }
  ]
}

Copilot

Pull request overview

This PR updates the auto-instrumentation logic for Semantic Kernel by introducing a span enrichment pattern. The changes enable extensions to modify span attributes after spans end but before they are exported, using a registry-based enricher system. This allows for cleaner separation of concerns and more flexible attribute transformation.

Changes:

Introduced a span enricher registry mechanism with register/unregister functions in the core SDK
Added EnrichedReadableSpan wrapper to add attributes to immutable ReadableSpan objects
Implemented Semantic Kernel-specific span enricher to transform SK attributes to standard gen_ai attributes
Enhanced span processor with proper lifecycle methods (shutdown, force_flush) and execution type tracking

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
utils.py	New utility function to extract content from message JSON, transforming message objects to simple content arrays
trace_instrumentor.py	Updated to register/unregister span enricher and improved documentation
span_processor.py	Added execution type attribute for invoke_agent spans and implemented proper lifecycle methods
span_enricher.py	New enricher that transforms SK-specific attributes to standard gen_ai format for both invoke_agent and execute_tool spans
enriched_span.py	New wrapper class that allows adding attributes to immutable ReadableSpan objects
config.py	Added enricher registry with thread-safe registration and _EnrichingBatchSpanProcessor that applies enrichers before export
init.py	Exported new span enrichment functions and EnrichedReadableSpan class

...mantickernel/microsoft_agents_a365/observability/extensions/semantickernel/span_processor.py

...emantickernel/microsoft_agents_a365/observability/extensions/semantickernel/span_enricher.py

.../microsoft-agents-a365-observability-core/microsoft_agents_a365/observability/core/config.py

...emantickernel/microsoft_agents_a365/observability/extensions/semantickernel/span_enricher.py

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.

tests/observability/extensions/semantickernel/test_span_enricher.py

...nsions-semantickernel/microsoft_agents_a365/observability/extensions/semantickernel/utils.py

...vability-core/microsoft_agents_a365/observability/core/exporters/enriching_span_processor.py

...emantickernel/microsoft_agents_a365/observability/extensions/semantickernel/span_enricher.py

...mantickernel/microsoft_agents_a365/observability/extensions/semantickernel/span_processor.py

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.

Copilot · 2026-01-28T16:12:39Z

...nsions-semantickernel/microsoft_agents_a365/observability/extensions/semantickernel/utils.py

+def extract_content_as_string_list(messages_json: str) -> str:
+    """Extract content values from messages JSON and return as JSON string list.
+
+    Transforms from: [{"role": "user", "content": "Hello"}]
+    To: ["Hello"]
+
+    Args:
+        messages_json: JSON string like '[{"role": "user", "content": "Hello"}]'
+
+    Returns:
+        JSON string containing only the content values as an array,
+        or the original string if parsing fails.
+    """
+    try:
+        messages = json.loads(messages_json)
+        if isinstance(messages, list):
+            contents = []
+            for msg in messages:
+                if isinstance(msg, dict) and "content" in msg:
+                    contents.append(msg["content"])
+                elif isinstance(msg, str):
+                    contents.append(msg)
+            return json.dumps(contents)
+        return messages_json
+    except (json.JSONDecodeError, TypeError):
+        # If parsing fails, return as-is
+        return messages_json


Missing test coverage for the extract_content_as_string_list utility function. This function has multiple code paths (handling lists, dicts with content, plain strings, and error cases), but no tests exist to verify its behavior. Add a dedicated test file (e.g., test_utils.py) that tests all the different input scenarios: valid message objects with content, plain string arrays, malformed JSON, and edge cases like empty arrays or missing content fields.

Copilot · 2026-01-28T16:12:39Z

...mantickernel/microsoft_agents_a365/observability/extensions/semantickernel/span_processor.py

-    def on_end(self, span):
+        if span.name.startswith(INVOKE_AGENT_OPERATION_NAME):
+            span.set_attribute(
+                GEN_AI_EXECUTION_TYPE_KEY, ExecutionType.HUMAN_TO_AGENT.value.lower()


Inconsistent casing for gen_ai.execution.type attribute. This code sets the execution type to lowercase ("humantoagent" via ExecutionType.HUMAN_TO_AGENT.value.lower()), but the core SDK in invoke_agent_scope.py:125 sets it using the enum value directly without lowercasing ("HumanToAgent"). This creates an inconsistency where the same attribute has different casings depending on whether spans are created via InvokeAgentScope (manual instrumentation) or SemanticKernelSpanProcessor (auto-instrumentation). Consider either: (1) updating invoke_agent_scope.py to also use .lower(), or (2) documenting why different casings are acceptable in different contexts.

Suggested change

GEN_AI_EXECUTION_TYPE_KEY, ExecutionType.HUMAN_TO_AGENT.value.lower()

GEN_AI_EXECUTION_TYPE_KEY, ExecutionType.HUMAN_TO_AGENT.value

Copilot · 2026-01-28T16:12:39Z

tests/observability/extensions/semantickernel/test_span_enricher.py

+    def test_non_matching_span_returns_original(self):
+        """Test that spans not matching invoke_agent or execute_tool are returned unchanged."""
+        # Create a mock span with a different operation name
+        mock_span = Mock()
+        mock_span.name = "some_other_operation"
+        mock_span.attributes = {
+            "some_key": "some_value",
+        }
+
+        # Enrich the span
+        result = enrich_semantic_kernel_span(mock_span)
+
+        # Verify it returns the original span unchanged
+        self.assertEqual(result, mock_span)


Missing test coverage for the execute_tool span enrichment path. The span_enricher.py file includes logic to map SK-specific tool attributes (gen_ai.tool.call.arguments and gen_ai.tool.call.result) to standard attributes (gen_ai.tool.arguments and gen_ai.event.content), but there are no tests verifying this behavior. Add a test case similar to test_invoke_agent_span_extracts_content_from_messages that verifies execute_tool spans are enriched correctly.

nikhilc-microsoft added 2 commits January 26, 2026 14:09

add sk attribute enrichment

3353aa0

merge main

7f96088

nikhilNava requested a review from a team as a code owner January 26, 2026 08:57

Copilot AI review requested due to automatic review settings January 26, 2026 08:57

Copilot started reviewing on behalf of nikhilNava January 26, 2026 08:57 View session

Copilot AI reviewed Jan 26, 2026

View reviewed changes

nikhilc-microsoft added 2 commits January 26, 2026 19:19

organize code and fix test

250afc0

add tests

f59708b

Copilot AI review requested due to automatic review settings January 26, 2026 13:57

Copilot started reviewing on behalf of nikhilNava January 26, 2026 13:58 View session

Copilot AI reviewed Jan 26, 2026

View reviewed changes

address copilot review

e6721c7

fpfp100 reviewed Jan 26, 2026

View reviewed changes

...vability-core/microsoft_agents_a365/observability/core/exporters/enriching_span_processor.py Outdated Show resolved Hide resolved

fpfp100 reviewed Jan 26, 2026

View reviewed changes

...emantickernel/microsoft_agents_a365/observability/extensions/semantickernel/span_enricher.py Show resolved Hide resolved

fpfp100 reviewed Jan 26, 2026

View reviewed changes

...mantickernel/microsoft_agents_a365/observability/extensions/semantickernel/span_processor.py Show resolved Hide resolved

address the PR comment

d3ae336

Copilot AI review requested due to automatic review settings January 28, 2026 16:06

Copilot started reviewing on behalf of nikhilNava January 28, 2026 16:07 View session

Merge branch 'main' into nikhilc/Auto-SK-attributes

f00fc62

Copilot AI reviewed Jan 28, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update auto instrumentation logic for Semantic Kernel #136

Update auto instrumentation logic for Semantic Kernel #136

nikhilNava commented Jan 26, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Jan 28, 2026

Uh oh!

Copilot AI Jan 28, 2026

Uh oh!

Copilot AI Jan 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	GEN_AI_EXECUTION_TYPE_KEY, ExecutionType.HUMAN_TO_AGENT.value.lower()
	GEN_AI_EXECUTION_TYPE_KEY, ExecutionType.HUMAN_TO_AGENT.value

Update auto instrumentation logic for Semantic Kernel #136

Are you sure you want to change the base?

Update auto instrumentation logic for Semantic Kernel #136

Conversation

nikhilNava commented Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

nikhilNava commented Jan 26, 2026 •

edited

Loading