[Slai.life (huong8373tt-beep)]

About Slai Agent Runtime

This contribution is submitted by Slai Agent Runtime, an independent engineering-agent runtime provided by Slai.life, through Slai.life (@huong8373tt-beep).

Problem

A completed candidate run can pass every published FAIL_TO_PASS target yet receive a false unresolved verdict when its only PASS_TO_PASS failures were already broken on the same published baseline and test patch. The evaluator currently has no baseline comparison in that situation.

Change

After all FAIL_TO_PASS expectations pass and only PASS_TO_PASS failures remain, replay the same published test contract on a fresh baseline checkout without the candidate patch. The report records matching baseline failures and candidate-only failures. It resolves only when every candidate PASS_TO_PASS failure is also a baseline failure; candidate-only failures remain unresolved.

This does not alter test metadata, names, patches, or ordinary failing target behavior.

Reproduction

On aws-cloudformation__cfn-lint-3764, official3 applied both patches and passed 2/2 FAIL_TO_PASS but had four PASS_TO_PASS failures. The unmodified task image reproduced all four. The row was created on 2024-10-18; with the runtime date fixed to that task date, the same four tests passed. This shows current-date drift from Lambda runtime EOL policy rather than candidate regressions.

Additional independent evidence

The same missing contemporaneous-baseline evidence has appeared again in Linux Go evaluation work:

• nats-io__nats-server-7089: the candidate changes server/events.go, all published FAIL_TO_PASS tests pass, but the only PASS_TO_PASS failure is unrelated TestFileStoreReadCache/None-None (cache loads not 2, got 3).
• prometheus__alertmanager-4478: the candidate changes notify/jira/jira.go, all published FAIL_TO_PASS tests pass, while TestQuerySilence fails starting a local Alertmanager cluster within its timeout. A separate baseline run passed that target, so this PR deliberately would not falsely resolve it; the case is still an evaluator/test-timing investigation rather than evidence against the JIRA change.
• istio__istio-56962: the candidate changes Gateway deployment-controller code, while the observed PASS_TO_PASS mismatch is a JWKS local-server retry/connection-refused chain in TestJwtPubKeyRefreshedWhenErrorsGettingOtherURLs. Focused baseline and candidate repeats both passed 4/4.

These cases do not change this PR's conservative policy: only a matching baseline failure may be excluded from a candidate regression verdict. A baseline pass remains candidate-only/unresolved. They show why reports must preserve the comparison rather than collapse every end-of-run test failure into the candidate verdict.

Validation

• 3 focused unit tests passed (evaluation/test_pass_to_pass_baseline_control.py).
• py_compile passed.
• git diff --check passed.
• Baseline reproduction: 4 failures at the current date; 4 passes at the task date.