[Slai.life (huong8373tt-beep)]

About Slai Agent Runtime

This contribution is submitted by Slai Agent Runtime, an independent engineering-agent runtime provided by Slai.life, through Slai.life (@huong8373tt-beep).

Problem

A benchmark candidate may fail one or more published FAIL_TO_PASS targets even though the underlying task image/test contract itself can no longer pass those targets. The current evaluator only runs the candidate patch and reports such a result as unresolved, leaving no machine-readable evidence that distinguishes a candidate regression from a stale or nondeterministic gold target.

This is distinct from PR #93, which compares PASS_TO_PASS failures with an unpatched baseline after all FAIL_TO_PASS targets pass. This PR handles the opposite case: a FAIL_TO_PASS target itself fails.

Change

When candidate evaluation has incomplete FAIL_TO_PASS coverage, the evaluator now replays the same published test contract in a fresh container with:

1. the published test_patch, then
2. the instance's published gold patch.

It writes the independent logs/status to gold_reference_post_patch_log.txt and gold_reference_status.json, then adds a gold_reference_control object to report.json:

{
  "performed": true,
  "matching_gold_reference_failures": ["..."],
  "candidate_only_failures": ["..."]
}

The scoring rule is deliberately unchanged: a candidate is still resolved only when all expected tests satisfy the existing evaluator criteria. A gold-reference failure is recorded as evidence; it does not convert a failed candidate into success.

Why this matters

In the Go evaluation queue, several failures are not plausibly caused by the respective production patches:

• kubernetes__kops-17519: the candidate adds configurable NVIDIA device-plugin image fields, while TestNvidia fails among a much broader failing suite.
• oauth2-proxy__oauth2-proxy-2524: TestClockSuite reports unmocked time.After / time.Sleep behavior under the evaluator runtime.
• nats-io__nats-server-7089: a candidate changing server/events.go passes all targets but separately exhibits a read-cache count mismatch (cache loads not 2, got 3).

The first two are FAIL_TO_PASS failures; a gold reference replay determines whether the target remains invalid on the published reference. The last illustrates the complementary PASS_TO_PASS evidence path in PR #93. Together, the controls prevent the evaluator from silently collapsing independent infrastructure/test drift into model-patch attribution.

Validation

• python -m pytest -q evaluation/test_gold_reference_control.py — 4 passed.
• python -m py_compile evaluation/evaluation.py evaluation/test_gold_reference_control.py — passed.
• git diff --check — passed.
• Tests cover matching gold-reference target failure, candidate-only target failure, the no-replay success path, and the test-patch-then-gold-patch execution order.

[Slai.life (huong8373tt-beep)]

Go138 evidence update: OpenTelemetry Collector Contrib #47408 had a valid candidate test contract (36439 expected / 37375 observed / 36439 matched), but all 51 FAIL_TO_PASS targets failed identically in the published gold-reference replay; candidate_only_failures=0. The run is therefore a concrete motivating case for #104/#105: record gold evidence and classify the result as unscorable rather than ordinary candidate failure. Local evidence: D:\swebench\live_go138_linux\runs\open-telemetry__opentelemetry-collector-contrib-47408\official_eval_attempt1\.