[rbuergi]

Refs #7082

Why

The governed AnalyzeDump read (deploy/aks/operator/bin/hosting-dump-analyze) runs a fixed set of thread and lock commands. For the out-of-memory dump in #7082 (14.6 GB, result at Ops/Dumps/i7082-analyzedump-hwx6z-pod on the control instance), it named every thread but could not say what filled the heap, because it ran no heap statistics.

What

• Two new commands in the fixed set, both read-only and both bounded by the existing per-command timeout:
  • gcheapstat gives generation sizes per GC heap. It is kept as printed, under the normal text cap.
  • dumpheap -stat prints one row per type (count, total bytes, name) and never prints an object. A plain dumpheap object listing, dumpobj and !env stay excluded.
• In the pod, dumpheap -stat keeps the TAIL of the output. SOS lists types smallest first and prints the Total … objects, … bytes line last. Cutting from the head, as every other section does, would keep only the rows that do not matter. The cut starts at a whole line, and the end sentinel still reports the full byte count.
• Condensation turns the table into the N largest types, largest first. Each row carries bytes, object count and its share of the heap total. N starts at 40 and shrinks with the other caps until the fact fits the lane's budget; the table is never left out. The output says how many types it shows out of how many it found, and says when the pod cut rows (the type count is then a lower bound). The Free row is kept, because free space means fragmentation, not retention. A heap walk stopped by its time bound prints no total, so the table says the walk did not finish rather than reporting an empty heap.
• The table lands as a dumpheap entry in sections, which the existing Plugins DumpAnalysis.Markdown page renders as is (no Plugins change needed), and also as a structured heap field. The Plugins reader ignores fields it does not know.
• Docs: Doc/Architecture/DebuggingNativeCrashes → "Reading a production dump", and the AnalyzeDump row in OperatingFromThePortal.

Proof

• deploy/aks/operator/test/run-tests.sh: 1074 passed, 0 failed locally on macOS. New assertions pin:
  • the section list, including the two new commands;
  • the heap totals with thousands separators parsed, largest type first, share, and the Free row;
  • the rendered table text and the gcheapstat text;
  • a 20,000-type heap: the largest types are kept, shown < found is stated, and the fact is 1,936 bytes, inside the budget;
  • an unfinished walk (exit 124) is stated, with no total.
• The in-pod half needs GNU tools, so it does not run on macOS (CI runs it). I ran the same assertions in a Linux container (postgres:16, Debian) against the stub dotnet-dump, which now prints a ~174 KB dumpheap -stat against a 100 KB cap. All eight sections ran in order. The tail with Generated.Type003000 and the Total line was kept, the cut started at a whole line, and the full byte count was reported.
  • Negative control: with the dumpheap line switched to section head (how every other section is cut), the same assertions FAIL. The kept text is Loading core dump…, Statistics: and the smallest types, ending in a partial line, and the Total line is lost.
• shellcheck -S warning passes on the operator scripts, as CI runs it.

Bounds

The per-command bound (300 s) and the Job deadline (1800 s) are unchanged. dumpasync already walks the heap once and finished on the 14.6 GB dump; dumpheap -stat is one more walk of the same kind. If every command ran to its bound, the total would exceed the Job deadline. In that case the Job fails by name and nothing partial is reported, which is unchanged behaviour. This is stated in the script header.

After merge

The operator picks this up with its image. The #7082 dump is then re-read by filing a new AnalyzeDump against the same dump (dumpFile: memex-portal-deployment-8556887fd9-hwx6z.1792742631/coredump.1.1792742595). Nothing was run against the cluster for this PR.

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

SOS fragmentation output can corrupt or displace heap statistics, and capped heap sections report inaccurate completeness metadata.

No open issues 🎉

What changed in this PR

Extends the operator’s governed AnalyzeDump read with heap statistics to help investigate memory pressure such as #7082.

Changes:

• Adds gcheapstat and dumpheap -stat to the bounded command set.
• Condenses heap output into largest-type statistics, totals and percentage shares.
• Adds regression fixtures and documents the expanded analysis.

File	Description
src/​MeshWeaver.Documentation/​Data/​Architecture/​OperatingFromThePortal.md	Updates the supported analysis commands.
src/​MeshWeaver.Documentation/​Data/​Architecture/​DebuggingNativeCrashes.md	Explains heap reporting and its limits.
deploy/​aks/​operator/​test/​stubs/​dump-analyze/​dotnet	Generates heap command output for tests.
deploy/​aks/​operator/​test/​run-tests.sh	Adds heap parsing, output-cap and timeout assertions.
deploy/​aks/​operator/​test/​fixtures/​dump-analyze/​pod.txt	Adds representative heap statistics.
deploy/​aks/​operator/​bin/​hosting-dump-analyze	Parses and summarizes heap statistics.
deploy/​aks/​operator/​bin/​_dump_analyze_pod.sh	Runs heap commands and retains capped tail output.

_{Review effort: Balanced}

[github-actions]

Test Results

    29 files  ±0      29 suites  ±0   1h 8m 45s ⏱️ -16s
14 379 tests ±0  14 182 ✅ ±0  197 💤 ±0  0 ❌ ±0 
14 409 runs  ±0  14 212 ✅ ±0  197 💤 ±0  0 ❌ ±0 

Results for commit a965a66. ± Comparison against base commit b85e51b.

This pull request removes 44 and adds 18 tests. Note that renamed tests count towards both.

   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---)
   --- End of inner exception stack trace ---, expected: True)
   --- End of inner exception stack trace ---, isDenial: True)
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---
 ---> (Inner Exception #1) MeshWeaver.Messaging.Hub.Test.InfrastructureFaultTest+ProviderException (0x80004005): Failed to connect to 10.42.18.4:5432<---
 ---> (Inner Exception #1) System.ArgumentException: Value does not fall within the expected range.<---
 ---> (Inner Exception #1) System.InvalidOperationException: boom<---
 ---> (Inner Exception #1) System.InvalidOperationException: source B is misconfigured<---
…

Memex.Portal.Shared.Test.InstanceIdRulesMatchTheRegistryTest ‑ TheSetupHostAgreesWithTheRegistry(candidate: "9b3fa5e5-00c5-45b8-8c6d-43955c388a2d")
Memex.Portal.Shared.Test.PublishStorageFaultIsNotAConflictTest ‑ AStorageFault_IsA500_NamingTheError(fault: System.AggregateException: One or more errors occurred. (Access to the path '/data/modules/activation.d/MeshWeaver.Markdown.Export.json' is denied.)
 ---> System.UnauthorizedAccessException: Access to the path '/data/modules/activation.d/MeshWeaver.Markdown.Export.json' is denied.
   --- End of inner exception stack trace ---)
Memex.Portal.Shared.Test.PublishStorageFaultIsNotAConflictTest ‑ AStorageFault_IsA500_NamingTheError(fault: System.InvalidOperationException: landing faulted
 ---> System.UnauthorizedAccessException: Access to the path '/data/modules/activation.d/MeshWeaver.Markdown.Export.json' is denied.
   --- End of inner exception stack trace ---)
Memex.Portal.Shared.Test.SessionDenialIsAnAnswerTest ‑ OnlyAVerdictReadsAsADenial(shape: "the same verdict nested, as a late denial dispatch"···, failure: System.InvalidOperationException: write failed
 ---> System.UnauthorizedAccessException: Access denied
   --- End of inner exception stack trace ---, isDenial: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall as an aggregate's FIRST member", fault: System.AggregateException: One or more errors occurred. (Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.)
 ---> MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
   --- End of inner exception stack trace ---, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall as an aggregate's SECOND member", fault: System.AggregateException: One or more errors occurred. (The operation has timed out.) (Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.)
 ---> System.TimeoutException: The operation has timed out.
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---
, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall nested in an aggregate inside a wrapper", fault: System.InvalidOperationException: outer
 ---> System.AggregateException: One or more errors occurred. (Value does not fall within the expected range.) (Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.)
 ---> System.ArgumentException: Value does not fall within the expected range.
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---

   --- End of inner exception stack trace ---, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall wrapped as InnerException", fault: System.InvalidOperationException: outer
 ---> MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
   --- End of inner exception stack trace ---, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "an aggregate of unrelated faults", fault: System.AggregateException: One or more errors occurred. (The operation has timed out.) (Value does not fall within the expected range.)
 ---> System.TimeoutException: The operation has timed out.
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) System.ArgumentException: Value does not fall within the expected range.<---
, expected: False)
MeshWeaver.Messaging.Hub.Test.InfrastructureFaultTest ‑ Classifies(shape: "a MIXED aggregate — one transient branch, one genu"···, exception: System.AggregateException: One or more errors occurred. (Failed to connect to 10.42.18.4:5432) (source B is misconfigured)
 ---> MeshWeaver.Messaging.Hub.Test.InfrastructureFaultTest+ProviderException (0x80004005): Failed to connect to 10.42.18.4:5432
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) System.InvalidOperationException: source B is misconfigured<---
, expected: False)
…

♻️ This comment has been updated with latest results.

[Copilot]

🟢 Approval recommended

The bounded, read-only changes preserve the reporting contract, with focused regression coverage and no identified blocking defects.

No open issues 🎉

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Output caps can discard the documented Free statistic, leaving fragmentation information out of the analysis.

No open issues 🎉

_{Review effort: Balanced}