Repository navigation
docs: three gotchas that cost real time reading a CI core dump - #776
Merged
Merged
Conversation
Complements DebuggingNativeCrashes.md (which already covers the macOS refusal and the Colima $HOME mount) with what is still missing: 1. Use the curl single-file dotnet-dump, NOT `dotnet tool install -g` - the installer fails under qemu emulation with "error reflecting type '…DotNetCliTool'", and the resulting "command not found" reads like a PATH problem rather than what it is. 2. A `strings` pass answers two questions with no container and no DAC: AccessViolation+FailFast classifies it as a runtime memory fault rather than an uncaught managed exception, and the loaded libcoreclr path reveals which runtime patch CI actually ran. On 2026-08-03 that was 10.0.10 against 10.0.9 locally - a candidate explanation for "only fails on CI" that is worth eliminating before blaming load or shard composition. 3. A frame appearing TWICE in one stack is re-entrancy, and re-entrancy IS the finding. Scanning for repeats first would have short-circuited the 2026-08-03 investigation, where three theories (teardown, ALC unload, scope registry) were each argued from circumstantial evidence and each wrong before the dump showed CreateHub → Build → … → CreateHub → Build nesting an Autofac ComponentRegistryBuilder.Build inside the in-progress one (fixed in #774). Verified: DocumentationLinkIntegrityTest passes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the native-crash debugging documentation to capture additional real-world CI minidump/core-dump triage lessons from the 2026-08-03 SIGSEGV investigation, complementing the earlier container/emulation guidance.
Changes:
- Documents why
dotnet tool install -g dotnet-dumpcan fail under qemu emulation and reinforces using the single-filecurldownload. - Adds “free triage” steps using
stringsto quickly identify FailFast/AccessViolation markers and the exact runtime patch version embedded in the dump. - Adds a heuristic for spotting re-entrancy via repeated frames in a stack, with an example tied to the #774 root cause.
Suppressed comments (1)
src/MeshWeaver.Documentation/Data/Architecture/DebuggingNativeCrashes.md:79
- The example commands use
dump.dmp, but earlier in the doc the dump is referenced as/dumps/<name>.dmp. Using the same placeholder here makes it clearer that the filename should match the artifact's actual.dmpname.
strings -a dump.dmp | grep -oE "AccessViolation|FailFast|SIGSEGV" | sort | uniq -c
strings -a dump.dmp | grep -oE "/usr/share/dotnet/shared/Microsoft.NETCore.App/[0-9.]+/lib[a-z]+\.so" | sort -u
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| ## Free triage before you start a container | ||
|
|
||
| `strings` on the raw core answers two questions in seconds, with no DAC and no emulation: |
Contributor
Contributor
Contributor
Contributor
Contributor
Contributor
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Complements
DebuggingNativeCrashes.md— which landed earlier today and already covers the macOS refusal and the Colima$HOMEmount — with the three things still missing, each of which cost real time during the 2026-08-03FutuRe.TestSIGSEGV investigation.1.
dotnet tool install -g dotnet-dumpfails under emulationIt dies with
There was an error reflecting type '…DotNetCliTool', and the resultingdotnet-dump: command not foundreads like a PATH problem rather than what it is. The doc'scurlsingle-file download is correct — this says why, so nobody "fixes" it back to the tool installer.2.
stringstriage before any containerTwo questions answered in seconds with no DAC and no emulation:
AccessViolation+FailFast⇒ a runtime memory fault, not an uncaught managed exceptionlibcoreclr.sopath ⇒ which runtime patch CI actually ranOn 2026-08-03 that was
10.0.10on CI against10.0.9locally — on its own a candidate explanation for "only fails on CI", and worth eliminating before blaming load or shard composition.3. A repeated frame is re-entrancy, and re-entrancy is the finding
Scanning the stack for repeats first would have short-circuited the whole investigation. Three theories — teardown, ALC unload, a scope registry — were each argued from circumstantial evidence and each wrong, before the dump showed:
Hub construction re-entering hub construction, nesting an Autofac container build inside the in-progress one. Fixed in #774.
Verification
DocumentationLinkIntegrityTestpasses.Release note
Skipped — internal documentation.
🤖 Generated with Claude Code