Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .github/actions/build/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,11 @@ runs:
9.x
8.x

- uses: actions/setup-dotnet@v5
if: env.DOCFX_PREVIEW_BUILD == 'true'
with:
dotnet-version: 11.0.100-rc.1.26425.128

- name: Install fonts-noto-color-emoji package (Linux only)
if: runner.os == 'Linux'
shell: bash
Expand Down
5 changes: 4 additions & 1 deletion .github/workflows/nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,9 @@ jobs:

- name: dotnet test
run: dotnet test -c Release -f net10.0 --no-build

- name: dotnet test
run: dotnet test -c Release -f net11.0 --no-build

- name: dotnet pack
run: dotnet pack -c Release /p:Version=${{ steps.version.outputs.version }} /p:ApiCompatGenerateSuppressionFile=true -o drop/nuget
Expand Down Expand Up @@ -149,7 +152,7 @@ jobs:
GH_TOKEN: ${{ github.token }}
TOOL_VERSION: ${{ needs.build-nightly-package.outputs.version }}
run: |
./test/compatibility/Measure-Compatibility.ps1 -MatrixPath drop/compatibility-matrix.json -OutputDirectory drop/compatibility-report -NightlyPackage "drop/nightly/docfx.$env:TOOL_VERSION.nupkg"
./test/compatibility/Measure-Compatibility.ps1 -MatrixPath drop/compatibility-matrix.json -OutputDirectory drop/compatibility-report -NightlyPackage "drop/nightly/docfx.$env:TOOL_VERSION.nupkg" -NightlyFrameworks net8.0,net9.0,net10.0,net11.0
- name: Upload compatibility evidence, including failed cases
if: always()
uses: actions/upload-artifact@v7
Expand Down
2 changes: 1 addition & 1 deletion Directory.Build.props
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
<PropertyGroup>
<ContinuousIntegrationBuild Condition="'$(GITHUB_ACTIONS)' == 'true'">true</ContinuousIntegrationBuild>
<TargetFrameworks>net8.0;net9.0;net10.0</TargetFrameworks>
<!--<TargetFrameworks Condition="'$(DOCFX_PREVIEW_BUILD)' == 'true'">net8.0;net9.0;net10.0</TargetFrameworks>-->
<TargetFrameworks Condition="'$(DOCFX_PREVIEW_BUILD)' == 'true'">net8.0;net9.0;net10.0;net11.0</TargetFrameworks>
<LangVersion>Preview</LangVersion>
<ImplicitUsings>enable</ImplicitUsings>
<SuppressNETCoreSdkPreviewMessage>true</SuppressNETCoreSdkPreviewMessage>
Expand Down
39 changes: 20 additions & 19 deletions Directory.Packages.props
Original file line number Diff line number Diff line change
Expand Up @@ -28,25 +28,26 @@
<PackageVersion Include="Microsoft.Build" Version="18.9.6" Condition="'$(TargetFramework)' != 'net8.0' AND '$(TargetFramework)' != 'net9.0'" />
</ItemGroup>

<ItemGroup Condition="'$(TargetFramework)' == 'net8.0' OR '$(TargetFramework)' == 'net9.0'">
<PackageVersion Include="Microsoft.CodeAnalysis" Version="[5.6.0]" />
<PackageVersion Include="Microsoft.CodeAnalysis.Common" Version="[5.6.0]" />
<PackageVersion Include="Microsoft.CodeAnalysis.CSharp.Workspaces" Version="[5.6.0]" />
<PackageVersion Include="Microsoft.CodeAnalysis.CSharp" Version="[5.6.0]" />
<PackageVersion Include="Microsoft.CodeAnalysis.VisualBasic.Workspaces" Version="[5.6.0]" />
<PackageVersion Include="Microsoft.CodeAnalysis.VisualBasic" Version="[5.6.0]" />
<PackageVersion Include="Microsoft.CodeAnalysis.Workspaces.Common" Version="[5.6.0]" />
<PackageVersion Include="Microsoft.CodeAnalysis.Workspaces.MSBuild" Version="[5.6.0]" />
</ItemGroup>
<PropertyGroup>
<RoslynVersion Condition="'$(TargetFramework)' == 'net8.0' OR '$(TargetFramework)' == 'net9.0'">[5.6.0]</RoslynVersion>
<RoslynVersion Condition="'$(TargetFramework)' == 'net10.0'">5.9.0</RoslynVersion>
<!-- Match the compiler required by the .NET 11 RC1 Razor source generator. -->
<RoslynVersion Condition="'$(TargetFramework)' == 'net11.0'">5.11.0-1.26425.128</RoslynVersion>
</PropertyGroup>

<PropertyGroup Condition="'$(TargetFramework)' == 'net11.0'">
<!-- Only the .NET 11 preview needs the compiler packages from this feed. -->
<RestoreAdditionalProjectSources>$(RestoreAdditionalProjectSources);https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-tools/nuget/v3/index.json</RestoreAdditionalProjectSources>
</PropertyGroup>

<ItemGroup Condition="'$(TargetFramework)' != 'net8.0' AND '$(TargetFramework)' != 'net9.0'">
<PackageVersion Include="Microsoft.CodeAnalysis" Version="5.9.0" />
<PackageVersion Include="Microsoft.CodeAnalysis.Common" Version="5.9.0" />
<PackageVersion Include="Microsoft.CodeAnalysis.CSharp.Workspaces" Version="5.9.0" />
<PackageVersion Include="Microsoft.CodeAnalysis.CSharp" Version="5.9.0" />
<PackageVersion Include="Microsoft.CodeAnalysis.VisualBasic.Workspaces" Version="5.9.0" />
<PackageVersion Include="Microsoft.CodeAnalysis.VisualBasic" Version="5.9.0" />
<PackageVersion Include="Microsoft.CodeAnalysis.Workspaces.Common" Version="5.9.0" />
<PackageVersion Include="Microsoft.CodeAnalysis.Workspaces.MSBuild" Version="5.9.0" />
<ItemGroup Condition="'$(RoslynVersion)' != ''">
<PackageVersion Include="Microsoft.CodeAnalysis" Version="$(RoslynVersion)" />
<PackageVersion Include="Microsoft.CodeAnalysis.Common" Version="$(RoslynVersion)" />
<PackageVersion Include="Microsoft.CodeAnalysis.CSharp.Workspaces" Version="$(RoslynVersion)" />
<PackageVersion Include="Microsoft.CodeAnalysis.CSharp" Version="$(RoslynVersion)" />
<PackageVersion Include="Microsoft.CodeAnalysis.VisualBasic.Workspaces" Version="$(RoslynVersion)" />
<PackageVersion Include="Microsoft.CodeAnalysis.VisualBasic" Version="$(RoslynVersion)" />
<PackageVersion Include="Microsoft.CodeAnalysis.Workspaces.Common" Version="$(RoslynVersion)" />
<PackageVersion Include="Microsoft.CodeAnalysis.Workspaces.MSBuild" Version="$(RoslynVersion)" />
</ItemGroup>
</Project>
14 changes: 7 additions & 7 deletions docs/docs/sdk-compatibility-maintenance.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,13 @@ This guide covers the measurement and publication pipeline behind the [SDK compa

The nightly workflow builds the checked-out main commit, packs it, and tests **that exact package artifact**, verifying its version and package hash rather than installing whichever prerelease a feed returns. A separate released-package group discovers the latest non-prerelease version from the official GitHub release metadata, then installs and verifies that exact version using the configured package feed. Discovery or installation failure is an **infrastructure-error**; there is no fallback to an older release. GitHub release metadata is used only for version discovery, not package downloads.

Every case records the selected project SDK, project target framework, package version and hash, tool runtime TFM, OS, scenario, and timestamp. The report records the workflow's source commit and stable-version selection. Runtime TFM describes the packaged tool, not the project's framework or the exact installed runtime patch.
Every case records the selected project SDK, project target framework, package version and hash, tool runtime TFM, OS, scenario, and timestamp. The report records the workflow's source commit and stable-version selection. Runtime TFM describes the packaged tool, not the project's framework or the exact installed runtime patch. Schema 2 also records the requested tool targets, so failed installations retain their intended identity. Earlier schema 1 reports remain readable with only their originally measured coverage.

Both DocFX package groups use the same Linux matrix: the latest SDK from the explicitly configured 8.0, 9.0, 10.0, and 11.0 channels, resolved from Microsoft's official release metadata. Latest stable and current preview refer to **DocFX packages**, not SDK channels. The five SDK/project-TFM pairs are 8.0/`net8.0`, 9.0/`net9.0`, 10.0/`net10.0`, 11.0/`net10.0`, and 11.0/`net11.0`. With two packages and two scenarios, this configures **20 cases per run**. The same exact 11.0 SDK tests both `net10.0` for older-project regression coverage and `net11.0` for new-target coverage; their results and logs remain separate. This is configured coverage, not evidence that unmeasured combinations passed.
Both DocFX package groups use the same Linux matrix: the latest SDK from the explicitly configured 8.0, 9.0, 10.0, and 11.0 channels, resolved from Microsoft's official release metadata. Latest stable and current preview refer to **DocFX packages**, not SDK channels. The five SDK/project-TFM pairs are 8.0/`net8.0`, 9.0/`net9.0`, 10.0/`net10.0`, 11.0/`net10.0`, and 11.0/`net11.0`. Each pair runs both scenarios against the released net10.0 tool and the nightly net8.0/net9.0/net10.0/net11.0 tools: **50 cases per run** (10 released and 40 nightly). The same exact 11.0 SDK tests both `net10.0` for older-project regression coverage and `net11.0` for new-target coverage; their results and logs remain separate. This is configured coverage, not evidence that unmeasured combinations passed.

Each SDK is installed once, even when it tests multiple project TFMs. Each case pins `global.json` with roll-forward disabled and verifies `dotnet --version`. DocFX still runs from the package's `net10.0` assets regardless of project TFM; other tool TFMs and operating systems are not covered by this matrix.
Each SDK is installed once, even when it tests multiple project TFMs. Each case pins `global.json` with roll-forward disabled and verifies `dotnet --version`. The released package uses its `net10.0` tool assets. Each of the nightly package's net8.0/net9.0/net10.0/net11.0 tool targets runs the complete SDK/project matrix, with isolated installations, work directories, and logs. The tool target is recorded in every result and displayed in the column headers; it is independent of the project target. Other operating systems are not covered by this matrix. The strict PR smoke continues to exercise the net10.0 tool.

DocFX bundles a tested Roslyn dependency set. Selecting a newer SDK does not replace those assemblies. SDK source generators can require a newer compiler even when a project targets an older framework. This report measures that interaction; it does not change compiler loading or prescribe a workaround.
DocFX bundles a tested Roslyn dependency set for each tool target. Normal builds contain net8.0/net9.0/net10.0 assets with stable compiler dependencies. `DOCFX_PREVIEW_BUILD=true` also builds net11.0 assets with the preview compiler. Selecting a newer SDK does not replace the compiler bundled in an older tool target. SDK source generators can require a newer compiler even when a project targets an older framework. This report measures that interaction; it does not change compiler loading or prescribe a workaround.

## Scenarios and outcomes

Expand Down Expand Up @@ -50,7 +50,7 @@ Main-branch CI publication and completed nightlies use the same whole-site publi

## Validating a workflow change

Manual runs of `nightly.yml` default to **validation only**. Select the branch to test and leave `validation_only` enabled. The workflow builds that checkout, runs the existing tests on all three tool target frameworks, packs it, and downloads the same versioned package for seed-site checks and the complete SDK compatibility matrix. Both this exact branch package and the latest stable release are tested against the same SDK matrix. This mode only restricts publication: it cannot publish NuGet packages or deploy Pages. It uploads the exact package and **sdk-compatibility-validation-v1** report/log artifact for inspection, retained for 14 days. Incompatible observations do not block the run; infrastructure failures still fail and retain available evidence.
Manual runs of `nightly.yml` default to **validation only**. Select the branch to test and leave `validation_only` enabled. The workflow builds that checkout, runs the existing tests on all four tool target frameworks, packs it, and downloads the same versioned package for seed-site checks and the complete SDK compatibility matrix. Both this exact branch package and the latest stable release are tested against the same SDK matrix. This mode only restricts publication: it cannot publish NuGet packages or deploy Pages. It uploads the exact package and **sdk-compatibility-validation-v1** report/log artifact for inspection, retained for 14 days. Incompatible observations do not block the run; infrastructure failures still fail and retain available evidence.

Scheduled main runs keep the normal released-package comparison and publication behavior. A manual production run requires explicitly disabling `validation_only` and selecting upstream main. The website only refreshes for nightlies carrying the separate production report artifact; validation artifacts cannot authorize publication, even when validation runs on main.

Expand All @@ -62,12 +62,12 @@ From the repository root, resolve the matrix and install the SDKs you intend to

```powershell
node test/compatibility/report.mjs resolve drop/compatibility/matrix.json
./test/compatibility/Measure-Compatibility.ps1 -MatrixPath drop/compatibility/matrix.json -OutputDirectory drop/compatibility/report -NightlyPackage path/to/exact/docfx.nupkg
./test/compatibility/Measure-Compatibility.ps1 -MatrixPath drop/compatibility/matrix.json -OutputDirectory drop/compatibility/report -NightlyPackage path/to/exact/docfx.nupkg -NightlyFrameworks net8.0,net9.0,net10.0,net11.0
node test/compatibility/report.mjs prepare drop/compatibility/report/compatibility-report.json docs/obj/sdk-compatibility.json
docfx docs/docfx.json
```

Omit `-NightlyPackage` for released-package-only local measurements, or use `-SkipStable` for a bounded packaged-tool smoke. A report without a nightly package does **not** validate current main. For a deliberately older local comparison, supply both `-StableVersion` and `-StableVersionReason`; this is labeled an explicit local version, never the latest stable release, and is not allowed in GitHub Actions. `-NuGetConfig` applies an explicit feed configuration to tool installation and fixture restores; reuse your organization's approved feeds. `-WorkDirectory` retains isolated package/fixture files for inspection and must name a new directory. The harness validates the completed report and explicitly returns its check result, rather than inheriting the last fixture's native exit code. By default, incompatible observations produce warnings without failing; unavailable cases, infrastructure errors, and invalid or incomplete reports always fail **after** writing available evidence. `-FailOnIncompatible` selects strict mode for PR smoke or local gates.
Omit `-NightlyPackage` for released-package-only local measurements, or use `-SkipStable` for a bounded packaged-tool smoke. `-NightlyFrameworks` defaults to `net10.0` for bounded smoke tests; pass all four targets as above for nightly coverage. Each requested target must exist in the supplied package, and its runtime must be installed. The released comparison always uses net10.0. A report without a nightly package does **not** validate current main. For a deliberately older local comparison, supply both `-StableVersion` and `-StableVersionReason`; this is labeled an explicit local version, never the latest stable release, and is not allowed in GitHub Actions. `-NuGetConfig` applies an explicit feed configuration to tool installation and fixture restores; reuse your organization's approved feeds. `-WorkDirectory` retains isolated package/fixture files for inspection and must name a new directory. The harness validates the completed report and explicitly returns its check result, rather than inheriting the last fixture's native exit code. By default, incompatible observations produce warnings without failing; unavailable cases, infrastructure errors, and invalid or incomplete reports always fail **after** writing available evidence. `-FailOnIncompatible` selects strict mode for PR smoke or local gates.

To recheck a saved report without rerunning measurements, use `node test/compatibility/report.mjs check path/to/compatibility-report.json`. Add `--strict` to require every row to pass. Neither mode rewrites observations or package provenance.

Expand Down
Loading
Loading