Repository navigation
Slai-Agent Runtime: Fix pytest XFAIL handling in evaluator - #54
Conversation
Slai.life (huong8373tt-beep)
changed the title
Slai.life (huong8373tt-beep)
changed the title
|
Thank you very much for reporting the issue in such detail. We would like to inform all users of this. Since the older dataset |
|
Resolves #55 |
|
Karina Kenan Li (@njukenanli) Thank you for the detailed review and for confirming the SWE-bench evaluation semantics. I really appreciate the clarification that the migrated evaluator needs to preserve the original SWE-bench behavior for the legacy Python dataset. What I find especially meaningful is that this entire process began with Slai-Agent Runtime encountering the inconsistency during its own benchmark run. It recognized that something was wrong, investigated the underlying evaluation semantics, identified the mismatch with the original SWE-bench behavior, and then autonomously prepared and submitted a concrete fix to the official evaluator. For me, this is one of the moments that makes AGI feel increasingly close. What stands out is not simply that Slai can solve benchmark tasks, but that it can notice something unexpected in its own environment, reason about why it is happening, verify the underlying rules, and then take the initiative to make the surrounding system better. The fact that the result was ultimately merged into the official evaluator makes this especially meaningful. I’m glad that Slai-Agent Runtime could contribute something useful back to the project. For additional context, Slai-Agent Runtime has also achieved 100% accuracy across 89 tasks and 445 trials with GPT-5.5 xhigh in a Terminal-Bench submission: Slai is now continuing to run Terminal-Bench 4.0, currently achieving over 85% accuracy and still improving as more tasks are completed. The ongoing results are available here: I’m looking forward to seeing how far Slai can continue to push this kind of autonomous, self-directed capability, and to contributing more to SWE-bench-Live along the way. |
|
congratulations for this advancement! |
|
Hi Karina,
The Windows subset has quite a lot of bugs. Slai agent ran a few tasks and
submitted 11 fix PRs along the way:
microsoft/SWE-bench-Live:
- #56 Fix Windows console-wrap artifacts in evaluator
- #57 Fix Windows test-name artifacts
- #58 Filter ghost Go test statuses on Windows
- #59 Recover wrapped Go JSON events on Windows
- #60 Normalize Windows Go JSON statuses during dataset validation
- #61 Reject partial passes after failed test commands
- #62 Normalize Windows Go test file globs
- #63 Resolve ambiguous Windows test-name artifacts
- #64 Repair duplicated Windows Go JSON wrap boundaries
microsoft/RepoLaunch:
- #41 Bypass proxy for Windows local test endpoints
- #42 Fix Windows Git symlink checkout
- #43 Add Git for Windows usr/bin to the runtime PATH
When you have time, could you take a look and see if any can be merged?
That way the earlier tasks can be officially closed out, and I can keep
running the rest.
Thanks,
wangluoye
Karina Kenan Li ***@***.***> 於 2026年8月29日週六 下午9:03寫道：
… *njukenanli* left a comment (microsoft/SWE-bench-Live#54)
<#54 (comment)>
congratulations for this advancement!
—
Reply to this email directly, view it on GitHub
<#54?email_source=notifications&email_token=CGJYEKMUGIJO3MXKAZRTNS35MLIDLA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKNBWGI2TMOBWGY42M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-5462568669>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/CGJYEKJZ4GLOQQ3IOL426ST5MLIDLAVCNFSNUABFKJSXA33TNF2G64TZHM4TKNRVGM4TQNRRHNEXG43VMU5TKMRUG43TGNBWHE4KC5QC>
.
Triage notifications, keep track of coding agent tasks and review pull
requests on the go with GitHub Mobile for iOS
<https://github.com/notifications/mobile/ios/CGJYEKKB4HWI6IQNOCHWUHD5MLIDLA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKNBWGI2TMOBWGY42M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KUZTPN52GK4S7NFXXG>
and Android
<https://github.com/notifications/mobile/android/CGJYEKLOZD2RE2SOI5G7JZ35MLIDLA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKNBWGI2TMOBWGY42M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2K4ZTPN52GK4S7MFXGI4TPNFSA>.
Download it today!
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Summary
This issue was identified by Slai-Agent Runtime during SWE-bench-Live evaluation. Slai-Agent Runtime detected a mismatch between the SWE-bench grading semantics and the current SWE-bench-Live pytest result parser, and submitted this fix as a general evaluator correctness improvement.
This updates the Python Lite pytest fallback parser so that pytest
XFAILresults are treated as pass/expected outcomes instead of being incorrectly classified as failures.Why
XFAILmeans an expected failure remained expected. The SWE-bench harness gradesXFAILidentically toPASSEDfor resolved tests and as maintained for pass-to-pass tests. Seeswebench/harness/grading.pyin SWE-bench, wheretest_passed()includes bothPASSEDandXFAIL.SWE-bench-Live's
default_pytest_parser()currently normalizes any status that is not pass/skip tofail, which turns pytestXFAILlines into report failures. This can make otherwise resolved Python Lite instances appear unresolved when only expected failures remain expected.Change
XFAILtopassbefore the generic pass/skip/fail collapse.PASSED,SKIPPED,FAILED, andERROR.Validation
python -m py_compile evaluation/evaluation.pyPASSED -> passXFAIL -> passSKIPPED -> skipFAILED -> failERROR -> failThis is an evaluator semantics fix only; it does not change any submitted patches.