Skip to content

Dump analysis: the thread that owns a contended monitor is never cut - #6490

Closed
rbuergi wants to merge 2 commits into
mainfrom
fix/6391-dump-analysis-names-lock-owners
Closed

rbuergi wants to merge 2 commits into
mainfrom
fix/6391-dump-analysis-names-lock-owners

Conversation

@rbuergi

@rbuergi rbuergi commented Oct 11, 2026

Copy link
Copy Markdown
Contributor

What changed

deploy/aks/operator/bin/hosting-dump-analyze (the AnalyzeDump instance action's one step) condenses clrstack -all into stack groups and keeps as many as fit the operations lane's log tail. It now reads syncblk first:

  • Each OS thread that owns a monitor other threads wait for is a group of its own. A waiter with identical frames is no longer folded into it.
  • Owner groups sort first, the most-waited-for first, then the rest by size as before.
  • An owner's stack is kept at four times the frame cap, and its first frame line is a label: [LOCK OWNER: holds the monitor 21 thread(s) wait for (syncblk owner System.Object)]. The label rides in frames, which the control plane already carries, so the page at Ops/Dumps/<action> shows it without a paired change.
  • The fact gains lockOwners: [{thread, waiters, object, stack}], with stack one of shown, cut, no managed stack. The run log prints one line per owner.

Why

The first production read of a wedged portal (Ops/Actions/analyzedump-memex-cloud-20261011-ltlwc-2, the pod behind MeshWeaver#6391) answered everything except the one question it was filed for:

  • syncblk: sync block 2, MonitorHeld 43 (21 waiters), owner OS thread 0x1e; sync block 169, MonitorHeld 9 (4 waiters), owner 0xf1.
  • 16 of the 40 groups shown (16 threads) wait in Autofac.Core.Lifetime.LifetimeScope.CreateSharedInstance, among them the stalled create (MeshExtensions.BuildPostCreationHandlers), two grain activations (MessageHubGrain.ResolveHubConfigurationObservable), Orleans codec creation and ASP.NET's endpoint matcher.
  • Thread 0x1e is in none of the 40 groups. Its stack is shared with nobody, 86 groups were found, and the cut keeps the largest.

So the reading names the lock and its owner's thread id, and not what that thread is doing.

How tested

deploy/aks/operator/test/run-tests.sh (the Operator scripts (shellcheck + behaviour) lane):

  • The existing fixture: thread 0x2c owns the monitor and 0x2d has the same frames. 0x2c is now group 0, alone, labelled, and lockOwners says shown.
  • The 400-thread budget case gains the production shape: one syncblk owner with MonitorHeld 43, the highest-numbered of 400 single-thread groups. Asserted: it is group 0, deeper than the frame cap, 21 waiters named, lockOwners[0].stack == "shown", and the fact still fits the 14,000-byte budget.

With the fix: 907 passed, 0 failed. With bin/hosting-dump-analyze reverted and the new tests kept: 902 passed, 5 failed (stack grouping, lock owner first, lockOwners, the log line, the owner surviving the cut). shellcheck on the script reports only the pre-existing SC1091 note. The in-pod half (_dump_analyze_pod.sh) is unchanged and does not run on macOS; CI runs it.

Not covered

  • Only Monitor locks appear in syncblk. A System.Threading.Lock owner is not found this way.
  • This does not fix the wedge. It makes the next AnalyzeDump of the same dump (it is still on the volume) show what OS thread 0x1e holds the lock for.

Refs #6391

🤖 Generated with Claude Code

rbuergi and others added 2 commits October 11, 2026 13:26
…ended monitor

hosting-dump-analyze grouped clrstack threads by identical stack and kept the
largest groups. The thread syncblk names as the owner of a monitor other
threads wait for has a stack of its own, so the cut dropped exactly the stack
a wedge is read from. On the first production read 21 threads waited on one
monitor and its owner was among the 46 of 86 groups cut.

Each lock owner is now a group of its own, sorted first (most waiters first),
kept at four times the frame cap and labelled in its first frame line. The
fact carries lockOwners with shown / cut / no managed stack, and the run log
prints it.

Refs #6391

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Uncontended monitors are incorrectly treated as lock owners, and two new stack-status branches lack coverage.

0 open findings

What changed in this PR

Improves dump analysis by preserving and identifying monitor-owner stacks during deadlock diagnosis.

Changes:

  • Parses syncblk and prioritizes lock-owner stacks.
  • Adds lock-owner metadata, labels, logging, and tests.
  • Documents the enhanced analysis output.
File Description
deploy/​aks/​operator/​bin/​hosting-dump-analyze Extracts and prioritizes monitor owners.
deploy/​aks/​operator/​test/​run-tests.sh Tests lock-owner grouping and retention.
src/​MeshWeaver.Documentation/​Data/​Architecture/​DebuggingNativeCrashes.md Documents lock-owner analysis.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@rbuergi

rbuergi commented Oct 11, 2026

Copy link
Copy Markdown
Contributor Author

Closing as a duplicate of #6484: two sessions fixed the same truncation in hosting-dump-analyze (the lock owner's stack being cut) within minutes of each other. #6484 covers the same script and test file and also updates the eviction-reading doc, so it is the one that lands. Refs #6391, Refs #6432.

@rbuergi rbuergi closed this Oct 11, 2026
auto-merge was automatically disabled October 11, 2026 11:43

Pull request was closed

@rbuergi
rbuergi deleted the fix/6391-dump-analysis-names-lock-owners branch October 24, 2026 16:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants