Skip to content

fix(headless): run the DeepSWE Maka arm inside the Pier task container #1473

Closed

Description

@Astro-Han

Problem

The DeepSWE A/B in #1343 runs the two arms across different execution boundaries.

Codex CLI and Kimi Code run inside the Pier task container and execute their tools locally. Maka runs the full task-run controller on the host, then sends every Bash, Read, Write, Edit, Glob, and Grep command through _ToolExecutorServer and environment.exec.

That difference exposes only the Maka arm to Pier's command-output transport on every tool call. Pier 0.3.0 starts docker compose exec with stderr=asyncio.subprocess.STDOUT, then returns the merged bytes as stdout and stderr=None. The Maka bridge converts that into merged stdout plus empty stderr even though IsolatedCommandResult requires separate streams.

The recorded DeepSWE canaries contain real consequences:

  • GNU Awk warnings from Glob/Grep enter the Base64 stdout frame and produce payload is not canonical Base64.
  • Reads of missing paths lose the intended diagnostic and become transport-integrity failures.
  • Edit reads can fail closed for the same reason.
  • Write and Edit write failures lose their stderr diagnostics.
  • Bash stderr is reported as stdout.

Historical artifacts include 11 Read integrity failures and repeated Glob/Grep failures in both serial and parallel runs. The paired Codex arm has none because individual Codex tool results never cross environment.exec; Pier launches only the outer CLI process, and Codex writes its rollout and output as container artifacts.

#1343 describes a system-level comparison, but it also requires infrastructure failures to be separated from model outcomes. These failures are specific to the Maka-side bridge and cannot be accepted as model results.

Reproduction

Use current main, official datacurve-pier==0.3.0, and a DeepSWE 1.1 Linux/amd64 task image. Run the Maka arm in host task-run mode with --isolation harbor-http.

Two reliable triggers are:

  1. Run a command-backed Glob or Grep fallback whose generated Awk -v value contains \.. GNU Awk writes a warning to stderr; Pier merges it into the Base64 stdout frame; parseFileToolOutputFrame() rejects the payload.
  2. Read a path that does not exist. The file tool writes the path error to stderr. Pier merges it into stdout, so the model receives an integrity failure instead of the path diagnostic.

The same executor behavior can be observed directly with:

printf OUT
printf ERR >&2
exit 7

Pier returns both strings through stdout and leaves stderr empty.

Proposed change

Run the complete Maka Headless task-run controller inside the Pier task container, matching the container-CLI arms. Reuse the existing Headless path:

--mode task-run
--isolation harbor-local

Use a pinned Linux/x64 Maka and Node toolchain mounted read-only. Route provider calls through the existing host credential proxy, with only an attempt-scoped proxy token inside the container. Write task-run, session, runtime, usage, and terminal artifacts under /logs/agent, then hydrate them after the run as the Codex adapter does.

Pier should start and supervise the agent process, not carry structured results for every Maka tool call.

Acceptance criteria

  • The DeepSWE Maka arm runs the full task-run controller inside the official Pier task container.
  • DeepSWE tool calls use the existing harbor-local executor, not _ToolExecutorServer or harbor-http.
  • Maka and its competitor use the same task image, container user, workdir, network policy, deadline authority, and verifier.
  • Both arms reach the provider through the host credential proxy; real provider credentials stay on the host.
  • A pinned, fingerprinted Linux/x64 Maka and Node toolchain is mounted read-only and needs no container-side network installation.
  • Task-run results, runtime events, sessions, usage, and terminal state are recovered from container artifacts.
  • A missing-path Read returns the path diagnostic.
  • Glob/Grep remain correct when GNU Awk writes a warning to stderr.
  • Bash preserves distinct stdout and stderr.
  • A real-model two-task canary passes the artifact and execution-identity audit required by eval(headless): DeepSWE 1.1 discriminative-30 harness A/B — Maka vs Kimi Code and Codex CLI #1343 using unmodified official Pier.
  • Historical cells containing tool transport-integrity failures are excluded and rerun.
  • The run manifest and report record that both agents executed inside the Pier task container.

Scope

This change should not patch or fork Pier, loosen the existing integrity checks, filter known warning strings, or make the two agents use identical internal tools. Their tool implementations are part of the systems being compared; the task-container boundary is not.

The current host bridge may remain for a demonstrated non-benchmark caller. If no supported caller still needs it, remove it instead of maintaining two task-run execution paths.

Related

Activity

  1. Astro-Han commented on Jul 25, 2026

    @Astro-Han
    ContributorAuthor

    I’m working on this. I’ll update the existing Harbor adapter so the Maka task-run executes inside the Pier task container with harbor-local isolation, without modifying Pier. I’ll add regression coverage for the container placement and keep the existing host-side artifact flow.

  2. self-assigned this
    on Jul 25, 2026
  3. Astro-Han commented on Jul 25, 2026

    @Astro-Han
    ContributorAuthor

    Post-merge validation is complete against unmodified datacurve-pier==0.3.0.

    Run: issue1473-main-b198bb0-two-task-canary-v1

    • The manifest records both arms at task-container; Maka uses harbor-local.
    • All four cells were attempted with zero infrastructure-failed cells and zero missing-final-usage cells.
    • The two Maka cells settled 317/317 and 292/292 tool call/result pairs.
    • A missing-path Read returned Read path does not exist: interp/exports.go.
    • The GNU Awk-sensitive Grep paths returned normal JSON results.
    • Bash preserved distinct stdout and stderr.
    • No transport integrity, canonical Base64, or incomplete-frame errors occurred.

    The Maka yaegi cell reached the 5400-second benchmark deadline, so the report contains one unscored cell. Its provider usage and terminal artifacts still settled, and the official verifier ran. This is a benchmark-budget outcome, not a transport or infrastructure failure.

    The transport-contaminated Maka cells from the v3/v4 runs are excluded from #1343 analysis and preserved as historical evidence. Both affected task IDs were rerun under the new manifest.

    The acceptance criteria for this infrastructure issue are satisfied. Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions