Crossgen2 work for .NET 8

[trylek]

Real bugs and deficiencies

• RVA fields don't work properly in composite mode for Managed C++ apps.
  • [Crossgen2] Fix RVA field access in composite mode #78723.
  • Cf. .NET 6 - R2R Composite image crashes the app #65879.
  • This is likely also related to RVAOrderingTest fails under composite R2R testing #55966.
  • Fixed with Fix RVA field access in composite Crossgen2 build (#78723) #85800.
• Generic cycles cause infinite expansion in Crossgen2 and subsequent OOMs / crashes.
  • Compiling application with LanguageExt as a dependency and ReadyToRun enabled using .NET 6 #66079.
  • Fixed with Adapt Tarjan generic cycle detector for use in Crossgen2 #71426.
• R2R code doesn't work with DOTNET_EnableSSE=0.
  • Ready To Run not working properly with DOTNET_EnableSSE=0 #85364.
  • @davidwrighton seems to be looking into that.
  • This is only broken on .NET 6.0, and is not an issue for .NET 8
• Problems with internationalization and Crossgen2.
  • Error in fresh Ubuntu 22.04.2 install: setlocale: LC_ALL: cannot change locale (en_US.UTF-8) crossgen-corelib.proj #82873.
  • This was filed by @carlossanlop, it's unclear whether it's a Crossgen2 / infra / documentation issue and who is the right owner.
• Type load for GDV guesses have observable behavior.
  • Type load for GDV guesses in R2R code has observable behavior #78461
  • It was filed by @jakobbotsch, I have no idea what's the severity and priority of this bug, @davidwrighton might know.
• --compilebubblegenerics produces code that hangs on startup.
  • crossgen2's --compilebubblegenerics causes live lock at app start (100% CPU usage) #78117
  • @davidwrighton would be probably best suited for an initial guess as to what might be happening here.
  • This was already fixed early in the NET 8 cycle, but the fix missed port to .NET 7 as compielbubblegenerics is not an officially supported scenario.
• crossgen2determinism on Win/Arm64.
  • Test failure readytorun\\determinism\\crossgen2determinism\\crossgen2determinism #84752
  • This issues is only reporting arm32 issues. Those are no longer relevant., and the issue is closed.
  • This one is a nullref in Crossgen2 that was also reproing in other tests and seems to be fixed now but recently we saw failures in the tests due to real diffs:
  • Four crossgen-comparison jobs are failing regularly in runtime-coreclr crossgen2 outerloop runs #74842, having the compiler output non-deterministic is worrisome.
  • A remaining source of issues was fixed by Adjust determinism tests to store away the 2 different binaries that are generated #87852
  • Most non-determinism has been fixed. There was still a case of non-determinism found in the Linux Arm target, but that issue appears to be rare. We will monitor this.
• Incorrect authoring for composite packages.
  • Microsoft.NETCore.App.Composite packaging misauthored - currently only win-x64 packages get published #72828
  • Not really a Crossgen2 issue per se, it's apparently about the publishing infra.
  • @hoyosjs filed and has driven this, we need to better understand the severity of the problem and its potential impact on Crossgen2 scenarios (e.g. our current work on composite container publishing).
• Crossgen2 fails for assemblies with embedded types.
  • crossgen2 fails on assemblies that include embedded types #67855
  • Not sure if it's still relevant, @davidwrighton should know for sure, in the issue thread he claims to be working on a fix.
  • That particular customer is no longer in need of the fix, but the issue was still present. This is now fixed with the addition of partial type equivalence support in Crossgen2. The new support should be sufficient such that we only have bugs to fix in the future, and not entire features to implement.
• Crossgen2 fails when a Managed C++ file is specified as just a reference.
  • crossgen2 fails when a Managed C++ dll is passed as a reference assembly #67849
  • We don't generate R2R code for references so it shouldn't bother us that they happen to be Managed C++.
• Segfault on Linux when running Crossgen2.
  • ReadyToRun (crossgen2) should respect the debuggable and optimization attributes of the assembly #67541
  • Potentially something around compiler versions, needs auditing to see if this is still relevant.
• Crossgen2 debug assert failure in owning type comparison.
  • crossgen2 asserts: "OwningType == other.OwningType" in MethodWithToken.CompareTo #61732
• dotnet-pgo dump crash.
  • "dotnet-pgo dump" throws "Same type X used as both ValueType and non-ValueType" #61604
  • I don't actually see why this is tracked as a Crossgen2 bug but I guess we need to find the right owner.
• Composite R2R build FAIL_FAST on Windows 7 SP1.
  • ReadyToRun composite build does not work on Windows 7 SP1 - no crash, just 0xC0000602 #61544
  • There was some investigation going on but with no results according to the issue.
• Crossgen2 crashes open a pop-up window.
  • crossgen2 crashes open a pop-up window #47552
  • We need to assess whether it's something that should be fixed in Crossgen2 or just ambient system behavior dictated by WER and the like.
  • Bruce Forstall confirmed we're no longer seeing these failures in the lab so that they were likely caused by misconfiguration of WER or related settings, the issue is now closed.
• getHFAType fails on ARM64 and passes on X64 targeting ARM64.
  • crossgen2: getHFAType fails on arm64 and passes on x64 targetting arm64 #46126
  • A very old bug, possibly fixed since, we need to identify its current status.
  • @ivdiazsa note: I think it's still there. Currently investigating.
• Crossgen2 skips methods using Vector.
  • [armel] Crossgen2 skips methods like GetIndexOfFirstNonAsciiByte_Default that use Vector<T> #45794
  • This is likely by design unless a particular CPU extension is specified in Crossgen2 command line.
  • We just need to double-check and follow up on the issue as appropriate, most likely by closing it.
• AV when running JIT optimization tests.
  • outerloop crossgen2 access violation in jit.opt workitem  #87305

Static virtual methods

• getCallInfo for SVM throws RequiresRuntimeJitException so that generic math cannot be inlined / optimized by Crossgen2 JIT.
  • Casting via generic math doesn't always inline in R2R #84421
  • There's a very old WIP PR where I tried to implement that:
  • Crossgen2 support for static virtual method resolution #54063
  • My current plan is to manually redo the PR against today codebase, addressing @MichalStrehovsky's feedback in the process.

Possible future work

• Managed C++ with native code is not supported
  • C++/CLI assemblies are not eligible for crossgen2 #83942
  • Priority is unclear, AFAIK there's been no recent escalation of this issue.
• Disable Crossgen2 for loongarch.
  • Loongarch add option to disable building Crossgen2  #78749
• Option for removing IL bodies from R2R images.
  • [feature proposal] Option for removing IL bodies from R2R images #52190
  • If I'm right to understand this is supposed to be a form of obfuscation, we need to discuss our stance in this area.
  • I think it basically violates @jkotas' defining principle that R2R is just a code cache.

Nice to have

• The option --pdb is very sensitive to the pdb file name.
  • crossgen2 --pdb does not work when the output PDB is not named the way MSDiaReader expects #78230
  • Ultimately it would be ideal to fix the various obscure checks in the DIA library but that's not a short-term solution.
  • We might want to look at the checks in DIA and play some tricks in Crossgen2 to name the output PDB file in a way that satisfies them.
• Big perf regression in debug mode.
  • Big crossgen2 perf regression in Debug mode #79561
  • At one point there was a speculation that the problem is Crossgen2 running under the CORE_ROOT debug build of coreclr and JIT but I haven't managed to observe that when trying to repro locally.
• Remove unused AllowEmptyTelemetry task from Crossgen2 usage.
  • Consider removing unnecessary AllowEmptyTelemetry task usage from Crossgen Targets #83290
  • I don't have too much context, @davidwrighton endorsed the proposal on the issue thread.
• Too many ValueTuple methods in R2R SPC.
  • R2R image of SPC has many costly ValueTuple methods #76513
  • Presumably boiling down to PGO partial R2R according to @jkotas' feedback on the issue thread.
  • We need to decide what to do with the issue.
• Crossgen2 library testing.
  • [Quality Week] Improve quality of Crossgen2 testing, most notably in context of library tests #74681
  • I believe @ivdiazsa has taken over work from @davidwrighton on making that work (at least to some extent).
• Some problem with libc version in musl nuget for Crossgen2.
  • CrossGen2 musl nupkg embeds the libc host #73908
  • @janvorli might have more context here.
• Dump missing dependent assemblies.
  • Crossgen2: Add Option to Report Missing Assemblies #68683
  • This would be a real nice to have ideal for an intern.
• Improve Crossgen2 nuget generation.
  • Follow-up improvements for #67636 #68519
  • This might result in a slight perf improvement but I doubt it's a real game changer.
• Improve documentation on --callchain-profile.
  • crossgen2 --callchain-profile usage #68491
  • Not sure if we want to do this as the feature is still mostly experimental.
• Respecting 'debuggable' and 'optimization' assembly flags.
  • ReadyToRun (crossgen2) should respect the debuggable and optimization attributes of the assembly #67541
  • I suspect this is already fixed or partially fixed, we just need to double-check and / or correct the remaining inconsistencies and close this, it should be a 1 day work at most.
  • This is probably also related to R2R jobs should not pass -O to crossgen2 #56148.
• Some bizarre corner case involving internal overrides.
  • Devirtualization in CrossGen2 for internal virtual overrides mismatches runtime behaviour  #60969
• Don't compile inlineable internal methods.
  • Crossgen2 shouldn't compile inlineable internal methods  #54781
• Lazy string optimization
  • [crossgen2] Lazy string optimization #48580
  • A real nice to have filed by JanK, should be less than a week of work.
  • Analysis of startup has shown that this will not produce a significant startup improvement, so I've closed the issue.
• Crossgen2 skips many methods.
  • [armel] Crossgen2 skip many methods during Version Bubble compilation #44948
  • This ties to our overall focus on improving the performance of Crossgen2-produced code.
• Crossgen2 doesn't have enough PDB coverage.
  • Crossgen /createpdb option doesn't have enough test coverage #13134
  • I have a WIP PR open for this but it hits some infra issues I haven't yet been able to resolve.
  • WIP: Initial provisions for semantic Crossgen2 PDB validation #75172
• Composite SDK compilation doesn't pass the global MIBC file to Crossgen2.
  • X64 - Too many methods JITted on startup for blank console application #85791
  • Fixing this should be a very easy way to slightly improve the performance of R2R composite published apps.

Individual test bugs

• EHPatternTests on Win/Arm64.
  • Test failure baseservices\\exceptions\\unittests\\EHPatternTests\\EHPatternTests.cmd #84815
  • One repro, possibly GC, not understood.
• slowpatheltenter / slowpatheltleave.
  • Test failure profiler\\elt\\slowpatheltenter\\slowpatheltenter.cmd #84750
  • Frankly speaking I don't see anything in the bug that would suggest it's a Crossgen2 bug.
  • I discussed this with @davmason in mid April but the issue is still marked as untriaged so I suspect we basically need to find the right owner.
• Composite R2R fails when RelativePath has a path component.
  • Composite ReadyToRun fails when RelativePath has a path component #79902
  • As of now it's unclear whether the customer is doing something they shouldn't or whether it's a real bug.
• Problems with NFloat._value in R2R publishing.
  • [S.P.CoreLib]System.Runtime.InteropServices.NFloat._value errors when publish with PublishReadyToRun #79965
  • I have no idea whether this is a framework, runtime or Crossgen2 bug.
• OOM in some cctor.
  • Test failure Regressions\\coreclr\\GitHub_34094\\Test34094\\Test34094.cmd #79582
  • Possibly fixed by @markples in Fix type initialization ordering under R2R #84275.
• hugeexpr1 test build failing on x86.
  • Test failure JIT\\jit64\\opt\\cse\\hugeexpr1\\hugeexpr1.cmd #74555
  • We need to identify the balance between further optimizing compiler memory footprint and just stating that there are limitations to code size compilable on 32-bit architectures.
• OSX x64 failures in composite GC stress due to "external disassembler not available".
  • [crossgen2-composite gcstress] OSX x64 failures with "External disassembler is not available" #71931
  • It's unclear to me whether it's a Crossgen2 bug, we need to find the right owner here.
• JITLoad_CustomALC failure due to wrong bind count for AssemblyToLoad,
  • Test JITLoad_CustomALC failed: TestLibrary.AssertTestException: Assert.IsTrue: Bind event count for AssemblyToLoad #57786
  • I saw some failures in these tests myself, @janvorli might know how to continue the investigation.

/cc @dotnet/crossgen-contrib

[trylek]

I have made two small fixes to the list - yesterday I missed the generic cycle bug as the open issue has somewhat cryptic title, I have also fixed the link to Mark's PR fixing type initialization ordering, yesterday I pasted the wrong link to a related issue. For commenting on individual issues feel free to use the original issue threads, my expectation is that this issue is only going to be used to prioritize and drive fixing the issues and to track progress of that effort (and possible overflow work to .NET 9 as not all above bullet points are critical for .NET 8 and, as you can see, there's just a lot of stuff).

[trylek]

As this is just an overarching issue summarizing Crossgen2 work and we're close to wrapping up .NET 8, I'm closing this issue for now; I assume that as part of .NET 9 planning we'll consider picking up some of the remaining work for .NET 9.