[Runtime Configuration][Telemetry] Dynamic telemetry detail and enrichment policy #589

Parent tracker: #262

Analysis / implementation baseline: dev.

Context

SharpLink already exposes process-wide OpenTelemetry-compatible ActivitySource / Meter instrumentation. Runtime dynamic configuration should not duplicate responsibilities that already belong to OpenTelemetry / Hosting, such as sampling, exporter replacement, or replacing the telemetry source objects themselves.

The useful SharpLink-owned dynamic surface is narrower:

control how much optional diagnostic detail SharpLink adds to telemetry for future calls/events, without changing instrument identity or taking ownership of the OpenTelemetry pipeline.

This issue groups three closely related concerns under one shared runtime policy:

Telemetry Detail Policy
    ├─ expensive / high-cardinality tags and enrichment
    └─ optional diagnostic events

They should share one immutable policy snapshot / generation instead of introducing three independent dynamic configuration systems.

Goal

Allow a running Client / Server to atomically replace one SharpLink-owned telemetry detail policy that controls optional diagnostic richness for subsequent telemetry operations.

The intended semantics are broadly next-call / next-event:

policy generation N
    ↓
RPC A captures / observes N
    ↓
control path publishes generation N+1
    ↓
RPC A remains internally consistent
future RPC / diagnostic event uses N+1

Exact capture points should follow current telemetry call paths and avoid unnecessary per-RPC work.

Scope

1. Expensive / high-cardinality tags and enrichment

Audit and classify existing tags / enrichment by cost and cardinality.

Examples may include endpoint/selection details, retry/compression/connection diagnostics, detailed failure classifications, or other contextual tags whose value is useful for troubleshooting but should not necessarily be emitted on every call by default.

Support a runtime policy that can enable/disable appropriate optional enrichment without changing stable core semantic tags required for normal interoperability/diagnostics.

2. Optional diagnostic events

Audit optional ActivityEvent / equivalent detailed diagnostics that are not required for baseline call accounting.

Allow suitable optional events to be enabled/disabled for future calls/events through the same telemetry detail policy.

Do not turn required correctness/accounting events into optional behavior merely to fit this issue.

3. Shared telemetry detail policy

Use one immutable SharpLink-owned policy snapshot rather than separate mutable booleans scattered through Client / Server / Runtime.

The final public shape is intentionally not fixed here. The implementation may converge on a small mode/flags model such as conceptually:

Basic
Detailed

or a narrow immutable option set if real requirements need more granularity.

Do not introduce a large general-purpose telemetry configuration object unless the audit demonstrates a concrete need.

Ownership boundary

This issue must keep the following responsibilities outside SharpLink runtime dynamic configuration unless a separate design proves otherwise.

OpenTelemetry / Hosting owned

• tracing sampling rate / sampler replacement;
• exporter replacement or destination changes;
• processor/provider pipeline replacement;
• application-level OpenTelemetry resource configuration;
• listener/provider lifecycle.

Stable SharpLink instrumentation identity

Do not dynamically replace or rename:

• ActivitySource instances / names;
• Meter instance / name;
• metric instrument names / identities;
• stable baseline semantic conventions expected by dashboards/tests/integrations.

A listener deciding not to consume traces/metrics is not itself a SharpLink runtime-config problem.

Runtime model

Default model:

immutable TelemetryDetailPolicy N
        ↓
atomic publication
        ↓
new logical call / diagnostic emission observes N

update

immutable TelemetryDetailPolicy N+1
        ↓
atomic publication
        ↓
future logical call / event observes N+1

For one logical RPC, avoid mixing incompatible detail generations when that would create inconsistent traces. If the cheapest correct implementation captures once at call start, prefer that model.

Performance requirements

Telemetry policy must preserve the existing disabled / unconsumed fast paths.

• no request-path lock;
• no per-call allocation merely to support dynamic policy when optional detail is disabled;
• no expensive enrichment work before confirming the relevant telemetry consumer / policy needs it;
• avoid evaluating high-cardinality strings / endpoint descriptions / failure-detail formatting when they will not be emitted;
• update path may allocate and validate because it is control-plane work;
• if hot-path branches materially change, record focused BenchmarkDotNet evidence.

The preferred shape is conceptually:

no listener / metric disabled
    -> existing very cheap fast path

listener active + Basic
    -> stable baseline telemetry only

listener active + Detailed
    -> optional enrichment/events

Audit work

Before finalizing the API, classify current telemetry sites into:

A. stable baseline semantic data
B. optional low-cost enrichment
C. optional expensive/high-cardinality enrichment
D. optional diagnostic events
E. OpenTelemetry/Hosting-owned behavior

For each B/C/D site, record:

• owner and call site;
• whether it is Client / Server / shared Runtime;
• computation/allocation cost;
• likely cardinality;
• appropriate capture point;
• whether one logical call must keep one policy generation;
• whether disabled behavior is already efficiently skipped.

Validation

At minimum cover:

• Basic -> Detailed update affects subsequent logical calls/events;
• Detailed -> Basic update stops future optional enrichment/events;
• an in-flight call does not produce internally inconsistent mixed policy output if the chosen model is call-captured;
• invalid candidate does not partially publish;
• repeated updates are race-safe with Client/Server Stop/Dispose;
• telemetry listeners being added/removed continues to work independently of SharpLink policy;
• stable metric/activity names remain unchanged;
• sampling/exporter behavior is not reimplemented inside SharpLink;
• optional expensive tag formatting is skipped when not required;
• disabled/basic path does not gain measurable allocation or lock contention;
• NativeAOT / trimming behavior remains compatible where affected.

Explicitly out of scope

• OpenTelemetry sampler / sampling-rate configuration;
• exporter destination/runtime exporter replacement;
• replacing TracerProvider, MeterProvider, processors, or listeners;
• replacing or renaming ActivitySource / Meter;
• renaming metric instruments dynamically;
• dynamically changing metric schema/identity;
• logging configuration unless separately designed;
• payload / arbitrary user metadata capture;
• automatically adding unbounded/high-cardinality business values;
• support snapshot behavior tracked separately by diagnostics work such as [diagnostics][release] 提供可导出的脱敏 Client/connection support snapshot #575;
• a giant unified RuntimeOptions / telemetry control surface before [Runtime Configuration] 动态配置 / Hot Reload 长期跟踪 #262's final control-surface audit.

Completion criteria

• Existing telemetry sites are classified into baseline / optional enrichment / expensive-high-cardinality / optional events / external OTel ownership.
• One shared immutable telemetry detail policy owns the SharpLink-specific dynamic behavior that is actually justified.
• Suitable optional enrichment and diagnostic events can change with next-call / next-event semantics.
• Stable instrumentation identity remains immutable.
• Sampling/exporter/provider responsibilities remain with OpenTelemetry / Hosting.
• Logical-call generation consistency is explicitly defined.
• Disabled/basic hot paths remain allocation-free / lock-free apart from pre-existing telemetry costs, with benchmark evidence if materially touched.
• Documentation explains what SharpLink telemetry detail policy controls and what must instead be configured through OpenTelemetry.

If the audit finds that some proposed detail is already naturally controlled by ActivitySource / Meter listener semantics and adds no independent SharpLink value, leaving that item external is a valid completion outcome.