The platform emits NO metrics of its own — zero custom meters in src/, so answering any question about live hub/stream state requires a heap dump that restarts the replica #3488

The platform emits no metrics of its own — so every question about live state costs a replica restart

Measured 2026-09-06, twice independently:

grep -rn "new Meter(" src/                     → 0 results
grep -rln "System.Diagnostics.Metrics" src/    → 0 results

Not one custom meter in the entire codebase, and the namespace is not referenced. Every counter
any investigation has relied on is a .NET runtime built-in (dotnet.timer.count and friends), used as
a proxy for what we actually want to know.

What this costs, measured tonight

#3432 asks why 6,925 Started sync/ hubs hold ~2.7 GB of a 3.5 GB heap. Nothing reports:

• live hub count, or the sync/-vs-node split
• the RunLevel histogram (Started / Dead / Starting)
• stream totals versus disposed
• anything that would show the curve moving after a fix

So the only way to answer is a heap dump against a live production replica — and 🚨 that
restarts the replica:

21:37:38Z  dotnet-dump collect -p 1 --type Heap    (process suspended)
21:39:17Z  kubelet: liveness probe failed, will be restarted
21:39:26Z  Complete                                (106 s of freeze)
21:42:18Z  pod 1/1 Running, RESTARTS 1

/alive is 15 s × failureThreshold 6 ≈ 90 s. And the freeze is not write-bound — dd oflag=direct
measured 169 MB/s on the same node, so a 5.5 GB dump is ≈33 s of the 106; the rest is createdump
walking ≈50 M objects with threads suspended.

🚨 The freeze therefore scales with heap size, which produces a catch-22: the population under
investigation only exists on a replica fat enough that dumping it restarts the replica — and the
restart destroys the state being measured. A freshly restarted portal was back to 2 977 MiB within
five minutes, so ~3 GiB is the FLOOR, not the accumulation.

Why this is worth fixing rather than tolerating

1. It makes a whole class of question unaskable in production. Not merely expensive —
   self-defeating, because the measurement destroys its own subject.
2. It is why 6,925 Started sync/ hubs hold ~2.7 GB of a 3.5 GB live heap — population MEASURED, cause NOT established (split from #3321) #3432 says "cause NOT established" after two sessions. That title is honest, and the
   honesty is forced by the absence of instrumentation rather than by the difficulty of the defect.
3. The data mostly already exists in-process. HostedHubsCollection holds the hubs; RunLevel is
   a field; the streams are known. This is exposing what the runtime already has, not computing
   anything new.
4. It closes the loop on fixes. Portal managed heap leaks ~215 MB/h through 267 gen2 collections, with ZERO sync/bake events — the cause #2194 is not tracking #3321 step 3 released 1,495 Dead hubs and its own author said to
   expect the curve to flatten, not vanish — and nobody can see the curve. A merged fix whose
   effect cannot be observed is indistinguishable from one that did nothing.

Suggested v1 — deliberately small

A single meter with the counters the last two investigations actually needed:

• live MessageHub count, tagged by address kind (sync/ vs node vs other) and RunLevel
• streams created / disposed / live
• (worth considering) live IoPool slots, since a parked pool has been a root cause before

🚨 Constraints this repo will hold you to: a meter is process-wide state, so it must be an
instance on a mesh-scoped singleton, never static (NoStaticState.md); nothing async in
hub-reachable code; and 🚨 an observable gauge that enumerates a live collection on the metrics
thread must not park the action block — the cure would otherwise be a smaller version of the
disease.

🚨 And a counter that cannot distinguish "zero hubs" from "the meter never ran" is the night's
recurring defect. Six instruments were measured today returning confident wrong answers rather than
no answer. Emit a denominator, or make absence explicit.

Scope

Filed rather than actioned: instrumenting the platform is a maintainer-level scope call, not
something to slip into an investigation. Recorded here so the cost is visible and the decision is
deliberate.

Related: #3432 (the investigation this blocks), PR #3481 (Doc/Architecture/PortalHeapIsHubs — the
dump hazard and the ClrMD walk recipe), #3325.