eval(headless): DeepSWE 1.1 discriminative-30 harness A/B — Maka vs Kimi Code and Codex CLI #1343

Following the Terminal-Bench 2.1 harness comparisons, compare Maka against two native CLI harness surfaces on the harder DeepSWE 1.1 software-engineering benchmark.

These are two independent paired system-level A/Bs, not causal component experiments.

Model strata

Kimi K3

• Arms: Maka vs Kimi Code
• Model: kimi-coding-plan/k3
• Reasoning effort: max

GPT-5.6-Sol

• Arms: Maka vs Codex CLI
• Model: openai-codex/gpt-5.6-sol
• Reasoning effort: xhigh
• Authentication: Codex OAuth account plan

Within each stratum, both arms use the same underlying model, task source, task order, reasoning effort, verifier, and attempt policy while retaining their native harness behavior.

Results must be analyzed separately by model stratum and must not be pooled into one pass rate.

Benchmark scope

Run the frozen 30-task discriminative subset introduced by #1398 through DeepSWE 1.1 and Pier:

• 11 tasks with public K3 seed pass rate 0.25
• 17 tasks with public K3 seed pass rate 0.50
• One all-fail and one all-pass anchor

The same frozen tasks are used for both model strata.

This subset is intentionally selected to increase paired discrimination and does not estimate the full 113-task DeepSWE score. Because selection used public K3 outcomes, the GPT-5.6-Sol result is conditional on a K3-informed subset and must be reported as such. No model-specific subset may be selected after observing these runs.

Harness implementation

#1398 adds DeepSWE 1.1 as a benchmark profile of the existing harness A/B driver, paired with the Pier executor. It provides:

• Pier-compatible Maka, Kimi Code, and Codex CLI adapters
• Frozen subset task IDs and task-tree fingerprint
• Host-side credential proxying
• Structured verifier, trace, token, and duration accounting
• Dry-run and fake-backend end-to-end verification

No real-model Pier trial has yet been accepted.

Execution plan

1. Merge or otherwise freeze the accepted feat(headless): run DeepSWE via Pier as a benchmark axis of the harness A/B #1398 revision.
2. Run a two-task real-model paired canary for each model stratum.
3. Audit all eight accepted canary cells before expansion:
   • Official verifier outcome is present
   • Frozen task and execution identities match
   • Provider usage and tool execution settle completely
   • Infrastructure failures are separated from model outcomes
4. If a stratum's canary is valid, run its remaining tasks to exactly 30 accepted pairs / 60 accepted cells.
5. Stop at exactly 60 accepted pairs / 120 accepted cells across both strata.
6. Do not add repetitions. Rerun only cells demonstrated to be infrastructure-invalid, preserving the original evidence.
7. Publish the results under docs/eval/, with separate sections for K3 and GPT-5.6-Sol.

Reported outcomes

Report independently for each model stratum.

Effectiveness:

• Pass count and pass rate by arm
• Paired wins, losses, and ties
• Exact paired significance and uncertainty
• Deadline and budget-exhaustion outcomes
• Failure-mechanism classification for discordant pairs

Resource efficiency:

• Uncached input, cached input, output, and total tokens
• Wall-clock duration
• Tokens and duration per pass
• Incremental resource cost per additional pass, if either arm wins more tasks

The Kimi and Codex account-plan accounting surfaces are not directly comparable monetary costs. Do not report recorded $0 values as a cost advantage. Any hypothetical API pricing projection must be labeled separately.

Completion criteria

This issue is complete when:

• Both two-task real-model canaries are audited and valid
• Exactly 30 accepted task pairs are evaluated for each model stratum
• The paired performance and resource report is published

A full 113-task DeepSWE run is outside this issue and should be tracked separately if later justified.