Skip to content

Slai Agent Runtime:Guard unrepresented Windows .NET test patch regressions - #88

Open
Slai.life (huong8373tt-beep) wants to merge 1 commit into
microsoft:mainfrom
huong8373tt-beep:slai-windows-dotnet-test-patch-contract
Open

Slai.life (huong8373tt-beep) wants to merge 1 commit into
microsoft:mainfrom
huong8373tt-beep:slai-windows-dotnet-test-patch-contract

Conversation

@huong8373tt-beep

Copy link
Copy Markdown

Summary

This adds a narrow fail-closed guard for Windows rows where the official C#/.NET test_patch adds xUnit-style regression methods, but published FAIL_TO_PASS metadata names none of those methods.

In that condition, exact-name grading has no stated target for the official regression. A full-suite result, including unrelated GPU, Appium, font, locale, or parser outcomes, must not be converted into a candidate success or failure.

The evaluator writes:

{
  "resolved": null,
  "grading_status": "unavailable",
  "failure_class": "unrepresented_windows_dotnet_test_patch_regressions"
}

The guard does not repair test names, infer replacement metadata, modify commands or patches, start a task container, or mark a candidate solved.

Reproduction: AvaloniaUI__Avalonia-19538

The Windows row is based on Avalonia upstream #19538, the FontFamily / FontManager fix for resolving ordinary system font names with a base URI.

Its official test_patch adds these attributed C# regressions:

Avalonia.Base.UnitTests.Media.FontFamilyTests.Should_Parse_FontFamily_With_BaseUri
Avalonia.Base.UnitTests.Media.FontFamilyTests.Should_Parse_Relative_Path
Avalonia.Skia.UnitTests.Media.FontManagerTests.Should_Get_SystemFont_With_BaseUri

But the published FAIL_TO_PASS list contains six unrelated rendering/parser identities, for example:

Avalonia.Skia.RenderTests.PathTests.Path_With_PenLineCap
Avalonia.Markup.UnitTests.Parsers.SelectorGrammarTests.Is_Name
Avalonia.Skia.RenderTests.PathTests.CubicBezier_Relative

None names any added FontFamily / FontManager regression. Replaying the exact published test patch against this guard extracts the three methods above and returns unavailable before runtime startup.

The completed Windows run also showed why unrelated full-suite outcomes cannot stand in for the missing contract: Direct2D tests require an unavailable graphics interface, Appium requires a missing service on port 4723, and VSTest output is localized while the published parser recognizes English detailed lines only. Those observations are supporting evidence, not part of this guard's predicate.

Guard boundary

It applies only when all of these hold:

  1. platform is Windows;
  2. the official patch adds one or more attributed C# test methods in a conventional test-source path;
  3. the attribute is an xUnit-style Fact, Theory, Win32Theory, StaFact, or StaTheory form; and
  4. no published FAIL_TO_PASS identity ends in any added method name (with optional parameter text).

Rows outside that shape continue on the existing evaluation path. Test helpers, production methods, deleted lines, context lines, non-test paths, Linux rows, and rows that represent at least one added method are unchanged.

Relationship to existing PRs

This shares the dataset-integrity principle of #77: an official regression added by a Windows test patch must be represented in published FAIL_TO_PASS metadata. It is intentionally separate because #77 parses the distinct Go shape (func Test... in *_test.go), while this PR handles attributed C#/.NET test methods.

It is also separate from #74, which rejects visibly malformed expected-test strings. Avalonia's published identities are syntactically well-formed; their defect is that they do not represent the methods introduced by the official patch.

The durable resolution is revalidation and republication of AvaloniaUI__Avalonia-19538 with the actual regression identities and a runnable Windows test environment. This evaluator guard only prevents an unrelated suite outcome from becoming a misleading candidate verdict in the meantime.

Validation

python -m py_compile \
  evaluation/evaluation.py \
  evaluation/windows_dotnet_test_patch_contract.py \
  evaluation/test_windows_dotnet_test_patch_contract.py

python -m unittest evaluation.test_windows_dotnet_test_patch_contract -v
# Ran 5 tests — OK

git diff --check
# passed

The tests cover extraction of added attributed methods; a represented identity; non-Windows behavior; non-test and unattributed no-op behavior; an unrepresented C# regression contract; and the no-runtime-start unavailable report.

@huong8373tt-beep Slai.life (huong8373tt-beep) changed the title Guard unrepresented Windows .NET test patch regressions Slai Agent Runtime:Guard unrepresented Windows .NET test patch regressions Oct 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant