[Slai.life (huong8373tt-beep)]

Context

Slai Agent Runtime replays Windows benchmark rows in their published task images to distinguish candidate-specific outcomes from a published grading contract that cannot observe the requested change.

During the formal Windows replay of podman-desktop__podman-desktop-13439, both patches applied successfully:

• the official test_patch changes only .github/workflows/e2e-main.yaml;
• the candidate changes only .github/workflows/pr-check.yaml.

The published Windows command is only:

pnpm test:unit -- --reporter=verbose --reporter=json ...

It ran the repository's Vitest workspace (4,593 reported tests; three unrelated UI assertion failures), but it neither reads nor executes either .github/workflows/*.yaml file. Thus, regardless of whether those unrelated unit failures are green or red, the command cannot establish whether the Windows ARM workflow-matrix change requested by the row is correct.

This PR protects that existing evaluation gap. It does not mark the Podman candidate solved, parse YAML to award a pass, replace the published command, or modify benchmark data.

Change

Return an unavailable result only when all of the following are true:

1. platform is Windows;
2. both the official test patch and the submitted candidate patch change only .github/workflows/*.yml or .yaml files;
3. the published command contains pnpm test:unit; and
4. the command invokes no explicit workflow validation/execution command (actionlint, yamllint, act, or gh workflow). Merely reading a workflow file is not semantic validation.

The preserved result is:

{
  "resolved": null,
  "grading_status": "unavailable",
  "failure_class": "windows_github_actions_workflow_contract_unobservable_by_published_unit_command"
}

Replay evidence

The guard was replayed against the unmodified published manifest row and the exact preserved candidate patch:

official patch path:  .github/workflows/e2e-main.yaml
candidate patch path: .github/workflows/pr-check.yaml
published command:    pnpm test:unit ...
guard triggered:      true

The prior formal artifacts remain separate evidence: the evaluator applied both patches, then the published unit command returned nonzero due to unrelated renderer/UI assertions. Those assertions do not inspect the workflow files.

Recommended dataset repair for podman-desktop__podman-desktop-13439

This evaluator guard only prevents a false candidate failure; it cannot make the row scoreable. To make the row scoreable, the dataset needs an official regression check that actually reads the workflow definitions and asserts the intended matrix/runner semantics.

For example, add a small deterministic Node/TypeScript or PowerShell validation script through the official test patch, then append that script to the published test_cmds. The check should parse the two workflow files and assert that the update-E2E jobs in both:

.github/workflows/pr-check.yaml
.github/workflows/e2e-main.yaml

include both windows-2025 and windows-11-arm, and that the job runner is driven by the corresponding matrix where applicable. A syntax-only linter is insufficient: the regression must assert these matrix values.

After such a dataset/command repair, candidates can be formally rerun and judged on the intended behavior.

Validation

python -m unittest evaluation.test_windows_github_actions_workflow_contract -v
# Ran 5 tests — OK

python -m py_compile \
  evaluation/evaluation.py \
  evaluation/test_windows_github_actions_workflow_contract.py

git diff --check
# passed

The tests cover the positive workflow-only/unit-only condition; Linux, source-file, non-unit-command, and explicit workflow-validation paths, plus the read-only workflow-path guard; and persistence of resolved: null without using unrelated unit-test outcomes to score the candidate.