[agronskiy]

Our GDPval prompt and container had drifted apart and neither matched the
published GDPval-AA v2 environment.

The prompt advertised torch 2.10.0, keras, jax, dlib, mtcnn, pygraphviz,
pdfkit, imgkit and fuzzywuzzy. gdpval.def installed none of them, and none
of them are in the v2 dependency set. That listing has been there since the
original Stirrup import (#1090); #1714 dropped torch from the container but
the prompt never got the matching edit. A model that believes it has PyTorch
plans a PyTorch solution and only finds out mid-trajectory.

The interpreter was also wrong: the published apt closure pins
python3.13=3.13.5, but basing on the docker python:3.13 image gave 3.13.15.

What this does

• Vendors the two dependency snapshots Artificial Analysis publishes for
  GDPval-AA v2: 419 pinned Python packages and the 762-package apt closure.
• Bases on debian:trixie and installs Debian's python3.13, asserting the
  micro version in %post. The closure is trixie-pinned (+deb13uN, t64
  names), so the old bookworm base also shipped older libreoffice/ffmpeg.
• Installs Python from the pin file verbatim instead of resolving loose names,
  and checks afterwards that every pin landed at the published version.
• Replaces the fabricated package listing in the prompt with the v2 stack
  summary, and says explicitly that there is no deep learning framework.
• Installs the Office metric-substitute fonts (Carlito, Caladea) and Noto.
  --no-install-recommends had dropped them, which silently reflows every
  Office document written in Calibri or Cambria and renders CJK as tofu.
• Adds verify_gdpval_sandbox.py, staged into the image at /opt/gdpval,
  which audits a built sif against both snapshots and then exercises the
  toolchain end to end.

Deliberate divergences

The only runtime divergence from the published prompt is the per-command time
limit: 5 minutes here against their 10, matching SHELL_TIMEOUT.

An earlier revision of this PR also claimed the working directory and
environment variables carry over between calls. That was wrong and is
fixed in 74b8765, thanks to @spanev. Three things in
apptainer_provider.py mean no shell state survives: every call is prefixed
with cd <working_dir>, the command runs inside bash -c, and that is
wrapped in a ( ... ) subshell. Files persist; directories, exported
variables, functions and aliases do not. So the published wording was
accurate for this backend all along, and the prompt now matches it.

The test for that claim originally asserted the prompt's wording, which
locked the error in rather than catching it. _build_command_script is now
extracted from _exec so the test asserts against the script the provider
actually builds.

arm64 builds drop five pins that publish x86_64 wheels with no sdist
(aspose-words, cadquery-ocp, cadquery_vtk, nlopt, and cadquery which depends
on the first). The list is closed and the verifier fails if a build drops
anything else.

Verification

Built and verified on SVG (x86_64) and HSG (aarch64). Verifier runs twice per
build, once with the production apptainer exec flags and once with a roomy
scratch dir.

Cluster	Arch	Interpreter	Pins	apt missing	Failures
SVG	x86_64	3.13.5	419/419	0	0
HSG	aarch64	3.13.5	414/419	3	0

Images published to .../eval/gdpval/sif/python-3.13.gdpval.gym-80e4fc.sif on
both clusters with a BUILD_INFO file recording arch and deviations.

Stacked on #3319, so the first commit here is that PR's. It drops out once
#3319 merges.

[agronskiy]

Good catch, and correct — thank you. Fixed in 74b8765; the PR description is updated too, since it advertised the same wrong claim.

You were right about the subshell, and there is a stronger reason next to it. Three independent things in apptainer_provider.py mean no shell state survives, any one of which alone kills a cd:

script = (
    f"cd {self._working_dir} && "     # 1. the directory is reset before EVERY call
    f"( {wrapped_cmd} ) 2>..."        # 3. outer subshell
)
# wrapped_cmd = f"timeout -k 10 {timeout} bash -c {shlex.quote(cmd)}"   # 2. inner subshell

Line 360 is the one I should have seen: an earlier cd is overwritten before the next command runs, independently of the subshells. Files persist, since it is the same filesystem; directories, exported variables, functions and aliases do not — exactly as you describe.

How I got it wrong: I saw the provider keeps one long-lived bash process and concluded state persists, without reading how commands are fed into it. I then treated the published Artificial Analysis wording as E2B-specific and "corrected" it, when it was accurate for this backend all along. The prompt now matches it again.

Your second point is the sharper one and I have acted on it. The test asserted the prompt's wording, so it locked the error in instead of catching it — wording can always be made to agree with itself. _build_command_script is now extracted from _exec, and the test asserts against the script the provider actually builds:

assert script.startswith("cd /workspace &&")
assert "bash -c " in script

The wording test stays, but only as a pair with the behavioural one.

Two consequences worth flagging, since they reach past this PR. The Super-3.5 evidence run (be3caf0d493d5f6b) ran on df3e201d9 and so used the bad prompt, and EFB main currently pins that SHA. It may also explain the 31-point Elo drop I flagged against the previous run — a model told it can cd, then silently working in the wrong directory, loses deliverables without crashing. I will move the pin to a tag on the fixed commit and repeat the run.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.