Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions Directory.Build.props
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,19 @@
toward leaving a dead package listed. Opting a project back in means intending to publish
it, and means adding it to .github/workflows/publish-packages.yml. -->
<IsPackable>false</IsPackable>
<!-- 🚨 #890: every executable here runs with Tiered PGO OFF (runtimeconfig System.Runtime.TieredPGO).
On .NET 10 linux-x64 the Tier-1 JIT, in roughly one process in three of the in-mesh compile
workload, inlines Roslyn's SourceMemberContainerTypeSymbol.ContainingType
(`_containingSymbol as NamedTypeSymbol`) into NamedTypeSymbol.AsNestedTypeDefinitionImpl WITHOUT
the isinst — the namespace container of a top-level type then reads as its containing type, and
every later Emit in the process dies in get_ContainingTypeDefinition. Measured, not inferred: the
JIT listing of that method has no isinst in 4 of 4 poisoned processes and has it in 10 of 10
clean ones, and with TieredPGO off 0 of 14 processes poisoned (Doc/Architecture/NodeTypeCompilation,
"The defect, in one listing"). This is a runtime defect worked around at the narrowest supported
switch; remove it only with a measurement on the runtime that fixes it. The test tree sets the
same property in test/Directory.Build.props, which does not import this file;
BakeHostRunsWithTieredPgoOffTest pins that this one reaches mw-plugin-test. -->
<TieredPGO>false</TieredPGO>
<GeneratePackageOnBuild>false</GeneratePackageOnBuild>
<!-- `dotnet pack` over the solution warns NU5128-style once per non-packable project; with the
default now false that is every project, and the warnings are noise, not signal. -->
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2144,6 +2144,64 @@ per-process rate above is a test-host rate and says nothing about a portal eithe
harness null a statement about `linux-x64`: it ran on arm64, and a codegen hypothesis is
architecture-specific.

#### The defect, in one listing — the JIT drops the `isinst`, and Tiered PGO off removes it

The capture above was finally read on core, where a workload change made the condition frequent:
the class split that #6128 shifted moved `InstallNeverInheritsRepoCompileVerdictTest` into
`Memex.Portal.Shared.Test` part 1/2, behind the stand-in emits of `ModulePlatformMemberLinkTest`,
`ModuleLinkVersionTest` and friends. From then on that part poisoned in **4 of 12** `main` runs
(`InstallNeverInherits…` and both `ModulesUpdateIndependentlyOfThePlatformTest` cases, or the whole of
`ModuleLinkVersionTest` when the onset came earlier) — no diff in those pull requests was involved.

**The reproduction in CI's own shape** (experiment branch `exp/890-tieredpgo-split-arm`, run
[`37647154163`](https://github.com/Systemorph/MeshWeaver/actions/runs/37647154163)): the very
`build-output-3` artifact of a poisoned pull-request run, part 1/2 launched exactly as the shard launches
it (`dotnet Memex.Portal.Shared.Test.dll -class …`), 14 runners per arm, on whatever `ubuntu-latest` hosts
GitHub assigned (EPYC 7763 without AVX-512, EPYC 9V74/9V45 and Xeon with it):

| arm | poisoned | clean |
|---|---|---|
| defaults + `DOTNET_JitDisasm` capture | **4** | 10 |
| `DOTNET_TieredPGO=0` | **0** | 14 |

**And the mechanism, 14 of 14.** The Tier-1 listing of
`NamedTypeSymbol:AsNestedTypeDefinitionImpl` inlines `SourceMemberContainerTypeSymbol.ContainingType`
(`_containingSymbol as NamedTypeSymbol`) behind a guarded devirtualisation. In all **10** clean hosts the
inlined body keeps the cast — a likely-class compare, then `CORINFO_HELP_ISINSTANCEOFCLASS`. In all **4**
poisoned hosts it is gone:

```text
G_M000_IG07:
mov rax, gword ptr [rbx+0x38] ; _containingSymbol, NO isinst
jmp SHORT G_M000_IG04 ; → test rax, rax / jne: "is nested"
```

So for a top-level type the namespace container reads as a non-null containing type, the guard admits
it, and `get_ContainingTypeDefinition` — whose own, separately compiled `ContainingType` still casts —
returns null and the writer dereferences it. That one listing explains every reading this page
collected: `dissect=READS-HEALTHY` (a direct call goes through the correct standalone method),
`flat=SAME-FRAME` (one member-less top-level class is enough), `compiler=PRIVATE-COPY-EMITS` (a cold copy
runs tier-0 code), the acquired onset (the method reaches tier 1 after enough emits) and the per-process
lottery (which code tier 1 produces depends on the profile that process collected). It is independent
of the ISA (both VEX and EVEX hosts poisoned). The standalone `ContainingType` in a poisoned host still
casts, so this is the inliner's copy only. *Why* the cast is folded is not established — the listing
shows the result, not the JIT phase; the profile-driven cast expansion is the suspect, because the
dropped cast and the surviving one differ only in which PGO data fed them.

**The remedy is the supported runtime switch, at the narrowest place that covers every compiling host:**
`<TieredPGO>false</TieredPGO>` in the root `Directory.Build.props` (every executable under `src/`,
`tools/` and `memex/`, including the bake host `mw-plugin-test`) and in `test/Directory.Build.props`
(which does not import the root). It reaches the runtime as `System.Runtime.TieredPGO: false` in each
host's runtimeconfig — the file a `dotnet <name>.dll` launch reads, which is how CI's shards start
hosts. Two guards pin that it arrives, one per props file: `TieredPgoIsOffInEveryCompilingHostTest`
(a test host, `test/Directory.Build.props`; negative control: a build with `-p:TieredPGO=true` fails
both of its assertions) and `BakeHostRunsWithTieredPgoOffTest` (the runtimeconfigs of `mw-plugin-test`
and `mw-combo-verify`, which inherit the root; negative control: deleting the root property alone turns
both cases red while the test tree's copy stays). This is a workaround for a runtime defect, stated as
one: the defect is in the .NET 10 JIT, not in this repository, and the switch is removed only with a
measurement on a runtime that fixes it — the arm above, re-run. The portal hosts live in
MeshWeaver.Plugins and need the same property there.

### Framework-version freezing

A compiled NodeType DLL references the MeshWeaver framework assemblies present
Expand Down
6 changes: 6 additions & 0 deletions test/Directory.Build.props
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,12 @@
the central src policy here so the test tree tracks it uniformly. Vulnerability handling
(bump/replace) is a deliberate dependency decision, not a per-build -warnaserror gate. -->
<WarningsNotAsErrors>NU1901;NU1902;NU1903;NU1904;$(WarningsNotAsErrors)</WarningsNotAsErrors>
<!-- 🚨 #890: test hosts run with Tiered PGO OFF, exactly as the root Directory.Build.props sets it for
src/ (this file does not import the root). Under Tiered PGO the .NET 10 linux-x64 Tier-1 JIT can
drop the isinst inside Roslyn's inlined SourceMemberContainerTypeSymbol.ContainingType, after
which no Emit in that process succeeds — Doc/Architecture/NodeTypeCompilation, "The defect, in
one listing". TieredPgoIsOffInEveryCompilingHostTest pins that the setting reaches the host. -->
<TieredPGO>false</TieredPGO>
</PropertyGroup>
<ItemGroup>
<!-- Security override, not a dependency we use: Testcontainers (any published version, incl.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
using System;
using System.IO;
using System.Text.Json;
using Xunit;

namespace Memex.Portal.Shared.Test;

/// <summary>
/// #890: a process that compiles in-mesh C# must run with Tiered PGO OFF.
///
/// <para>On .NET 10 linux-x64 the Tier-1 JIT, in roughly one test host in three of this suite's
/// workload, inlines Roslyn's <c>SourceMemberContainerTypeSymbol.ContainingType</c>
/// (<c>_containingSymbol as NamedTypeSymbol</c>) into <c>NamedTypeSymbol.AsNestedTypeDefinitionImpl</c>
/// without the <c>isinst</c>. A top-level type's namespace container then reads as its containing type,
/// and every later <c>Emit</c> in that process throws in <c>get_ContainingTypeDefinition</c> — which is
/// how <c>InstallNeverInheritsRepoCompileVerdictTest</c>, <c>ModulesUpdateIndependentlyOfThePlatformTest</c>
/// and <c>ModuleLinkVersionTest</c> went red on unrelated pull requests. Measured in CI's own shape: the
/// listing has no <c>isinst</c> in every poisoned host and has it in every clean one, and with Tiered PGO
/// off no host poisoned (Doc/Architecture/NodeTypeCompilation, "The defect, in one listing").</para>
///
/// <para>The defect itself is per-process and architecture-specific, so it cannot be reproduced on
/// demand in one test. What CAN be pinned deterministically is that the switch that removes it reaches
/// the host: the runtime reads <c>System.Runtime.TieredPGO</c> from this host's runtimeconfig, and the
/// build writes it there from <c>&lt;TieredPGO&gt;false&lt;/TieredPGO&gt;</c> in the Directory.Build.props
/// files. Negative control: building this project with <c>-p:TieredPGO=true</c> makes both assertions
/// fail.</para>
/// </summary>
public class TieredPgoIsOffInEveryCompilingHostTest

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, and fixed in 4182810. BakeHostRunsWithTieredPgoOffTest in MeshWeaver.PluginTester.Test parses the runtimeconfigs the SDK copies beside it from the referenced root-inheriting executables, mw-plugin-test.runtimeconfig.json and mw-combo-verify.runtimeconfig.json, and asserts "System.Runtime.TieredPGO": false in each.

Negative control, run locally: I deleted <TieredPGO>false</TieredPGO> from the root Directory.Build.props only and left the test tree's copy in place. Both cases went red (Total: 2, Failed: 2). With the property restored, both pass. The doc section and the root-props comment now name both guards.

{
private const string Switch = "System.Runtime.TieredPGO";

[Fact]
public void TheRuntimeIsHandedTieredPgoOff()
{
var value = AppContext.GetData(Switch);
Assert.True(value is not null,
$"{Switch} is not set for this host, so the runtime's default (Tiered PGO ON) applies and the "
+ "#890 miscompile can poison every Emit in the process. Set <TieredPGO>false</TieredPGO>.");
Assert.Equal("false", value!.ToString(), ignoreCase: true);
}

[Fact]
public void TheRuntimeConfigOnDiskCarriesIt()
{
var name = typeof(TieredPgoIsOffInEveryCompilingHostTest).Assembly.GetName().Name;
var path = Path.Combine(AppContext.BaseDirectory, $"{name}.runtimeconfig.json");
Assert.True(File.Exists(path), $"no runtimeconfig at {path}");
using var doc = JsonDocument.Parse(File.ReadAllText(path));
Assert.True(
doc.RootElement.GetProperty("runtimeOptions").TryGetProperty("configProperties", out var props)
&& props.TryGetProperty(Switch, out var pgo)
&& pgo.ValueKind == JsonValueKind.False,
$"{path} does not carry \"{Switch}\": false — the switch never reaches a host launched as "
+ "`dotnet <name>.dll`, which is how CI's shards launch it.");
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
using System;
using System.IO;
using System.Text.Json;
using Xunit;

namespace MeshWeaver.PluginTester.Test;

/// <summary>
/// #890, the production half: the executables built under the ROOT <c>Directory.Build.props</c> —
/// represented here by the bake host <c>mw-plugin-test</c>, which compiles every NodeType a CI bake
/// seals — must run with Tiered PGO OFF. Under Tiered PGO the .NET 10 linux-x64 Tier-1 JIT can drop the
/// <c>isinst</c> in Roslyn's inlined <c>SourceMemberContainerTypeSymbol.ContainingType</c>, after which
/// no <c>Emit</c> in that process succeeds (Doc/Architecture/NodeTypeCompilation, "The defect, in one
/// listing").
///
/// <para>This test host inherits <c>test/Directory.Build.props</c>, not the root, so it reads the
/// runtimeconfig the SDK copies beside it from the referenced executables — the files a
/// <c>dotnet mw-plugin-test.dll</c> launch actually reads. Deleting <c>&lt;TieredPGO&gt;false&lt;/TieredPGO&gt;</c>
/// from the root props turns it red; the test tree's own copy of the property cannot keep it green.</para>
/// </summary>
public class BakeHostRunsWithTieredPgoOffTest
{
[Theory]
[InlineData("mw-plugin-test")]
[InlineData("mw-combo-verify")]
public void TheRootInheritingExecutable_CarriesTieredPgoOff(string host)
{
var path = Path.Combine(AppContext.BaseDirectory, $"{host}.runtimeconfig.json");
Assert.True(File.Exists(path), $"no runtimeconfig for {host} at {path} — the referenced executable was not copied");
using var doc = JsonDocument.Parse(File.ReadAllText(path));
Assert.True(
doc.RootElement.GetProperty("runtimeOptions").TryGetProperty("configProperties", out var props)
&& props.TryGetProperty("System.Runtime.TieredPGO", out var pgo)
&& pgo.ValueKind == JsonValueKind.False,
$"{path} does not carry \"System.Runtime.TieredPGO\": false — {host} inherits the root "
+ "Directory.Build.props, and without the property it compiles under the #890 miscompile.");
}
}