[chrisknvidia]

Closes #79

Summary

This PR moves SkillEvaluator's optional Tier 3 backend from Harbor 0.13.2 to Harbor 0.24.0, the current stable release. Release notes: v0.24.0, v0.23.0.

It adapts SkillEvaluator to Harbor's current agent, environment, task, verifier, reward and trajectory contracts, without weakening SkillEvaluator's isolation, credential handling or scoring guarantees.

What changes for users

• Environments: --env-mode selects any of 23 Harbor-native backends or SkillEvaluator's own local mode.
• Backend settings: operator-only --environment-kwarg / --ek passes non-secret backend constructor options, with runtime-policy fields reserved.
• Native tasks: native Harbor tasks (BYOT) support multi-step tasks, step-local tests and Harbor 0.24's separate-verifier rules.
• Reports: reports keep task selectors, logical case IDs and display names distinct. They carry real token totals and correct attempt accounting, and multimodal tool output is rendered as text.
• Wandb mode: --env-mode wandb keeps working. It now runs Harbor's merged cwsandbox backend with W&B authentication.
• Codex: Codex keeps running with reasoning_effort=high, Harbor 0.22's default, so scores stay comparable.

How SkillEvaluator runs Harbor

flowchart LR
    subgraph SE["SkillEvaluator (operator process)"]
        CMD["tier3 evaluate / validate / doctor"] --> STAGE["Stage tasks<br/>generated or native"]
        STAGE --> ENV["Allowlisted child environment<br/>PYTHON_DOTENV_DISABLED=1<br/>HARBOR_TELEMETRY=0"]
        ENV --> RUN["harbor run<br/>from an empty SE-owned directory<br/>absolute paths, --ak / --ek"]
    end
    RUN --> H["Harbor 0.24.0"]
    H --> DOCKER["Hardened Docker<br/>Docker runtime pinned<br/>no SSH stream or GPU reservations"]
    H --> LOCAL["Local mode<br/>Seatbelt or bubblewrap"]
    H --> CLOUD["Harbor-native backends<br/>Daytona, Modal, E2B, ...<br/>wandb runs cwsandbox"]
    DOCKER --> ART["Job and trial artifacts<br/>rewards, ATIF v1.8"]
    LOCAL --> ART
    CLOUD --> ART
    ART --> COL["Collector<br/>identity, rewards, failures"] --> REP["JSON / HTML / Markdown reports"]

Loading

Harbor 0.23 and later load .env.local from their working directory, and Harbor and LiteLLM call load_dotenv() on import. Every Harbor process (runs, runtime preflight, harbor view and the ACK probe) therefore starts in an empty directory that SkillEvaluator owns.

• Dotenv loading and telemetry are disabled at launch.
• The tier3 extra now requires python-dotenv 1.2.0, the first release that honors PYTHON_DOTENV_DISABLED.
• Relative host paths such as KUBECONFIG are anchored, so they keep their meaning.

Changes

Harbor 0.24 integration

• Pin and lock:
  • harbor==0.24.0 is pinned and locked;
  • the lock change is harbor plus semver 3.1.0;
  • the environment-kwarg table matches the installed Harbor sources, and an AST parity test enforces this.
• Unified selectors:
  • agents and environments use Harbor's unified --agent / --env flags;
  • --agent-import-path is gone;
  • only the OpenAI-compatible gateway Codex wrapper may run on Harbor-native cloud backends.
• Codex: every Codex path gets --ak reasoning_effort=high.
• OpenCode: remote MCP servers set oauth: false, matching Harbor.

Environments

• Hardened Docker:
  • pins the Docker runtime;
  • reports stream=False and gpus=False;
  • downloads through its own Compose copy path, never Harbor's raw docker cp fallback.
• Reserved kwargs:
  • stream, enable_environment_dir_upload and the cwsandbox auth selector are reserved;
  • SkillEvaluator sets auth=wandb itself for the alias;
  • Modal's removed modal_sandbox_v2 is rejected.
• cwsandbox: SkillEvaluator checks the cwsandbox SDK and credentials itself, because Harbor 0.24 removed that preflight.

Native tasks and identity

• Identity:

  • selectors, logical dataset IDs, display names and structural attempt ordinals stay separate;
  • without a runner-owned selector map, Harbor's persisted task path outranks task names;
  • grader-authored rewards cannot set collector-owned identity fields.

• Paired-baseline isolation: these are rejected wherever they could contaminate standard grading:

  • skill-controlled setup and healthchecks;
  • prior trajectories;
  • step workdirs and step tests;
  • verifier controls, including task-authored judge retry and evidence-budget values;
  • collect hooks.

  custom_only keeps Harbor-native ownership.

Verifiers and rewards

• Native tasks follow Harbor 0.24's separate-verifier resolution, shown below.
• Harbor 0.24 requires numeric reward.json values. SkillEvaluator's verifier and wrapped graders already write them. A native task's own test script must too, and the docs say so.
• When Harbor records a failure on a step, the failed job now names that exception instead of reporting only a trial count.

flowchart TD
    subgraph Wrapped["Generated tasks, or native tasks with tests/grader.py"]
        EVAL["SE verifier: standard metrics + LLM judges"] --> SIDE["skill_evaluator_reward.json<br/>rich details"]
        GRADER["Custom grader via custom_grader_runner.py"] --> CUST["custom_reward.json"]
        SIDE --> NUM["reward.json: numeric values only"]
        CUST --> NUM
    end
    subgraph Own["custom_only native task with its own test script"]
        SCRIPT["Task test.sh"] --> OWN["reward.json: must be numeric"]
    end
    NUM --> PARSE["Harbor verifier<br/>rejects non-numeric values"]
    OWN --> PARSE
    PARSE --> COLLECT["Collector: numeric keys other than overall become custom metrics<br/>failed jobs name Harbor's recorded exception"]

Loading

flowchart TD
    STEP["Native task step"] --> Q{"Verifier image, or a Dockerfile or<br/>docker-compose.yaml in the step or task tests/?"}
    Q -- yes --> DED["Dedicated verifier<br/>runs its bundled tests/test.sh<br/>never receives SE's grader"]
    Q -- no --> FB["Verifier built from environment/<br/>tests uploaded<br/>Compose model checked against the verifier allowlist"]
    YML["docker-compose.yml only"] --> REJ["Rejected: Harbor ignores .yml"]

Loading

Collection, reports and redaction

• Collection:
  • accepts Harbor 0.24 job, trial and reward artifacts and ATIF v1.8;
  • multimodal tool output reaches judges and reports as text with [image]/[audio] markers;
  • merged trajectories carry the newest source schema version.
• Redaction and bounds:
  • diagnostics and artifacts are hardened against credential and URI user-info leakage;
  • report and error payloads are bounded;
  • timeout values must be finite;
  • mixed metric contracts and unavailable telemetry are explicit;
  • short credential-named flags no longer erase counts in summaries.

Merge with current main

• The 16 conflicting files were rebuilt function by function. Decisions:
  • the runner's selector map stays authoritative for case IDs;
  • fix(tier3): propagate Harbor dual-arm task suffixes and normalize canonical case ID resolution #161's resolver applies only without that map;
  • main's structured-output judges (feat(tier3): enforce structured JSON schemas and rate-limit retries in LLM judges #145) and retry and budget controls (feat(tier3): make evidence bundle budgets and final response limit configurable #160) are kept;
  • task-authored values for those controls stay rejected in native tasks under standard grading;
  • native task TOML edits stay structural.
• Unreleased CHANGELOG entries were moved back under Unreleased; released sections are identical to main.

Supported environments

Mode	Status in this PR
docker	Default. Hardened Docker backend. Live-tested on Harbor 0.24.
local	SkillEvaluator host mode with an OS sandbox. Live-tested on Harbor 0.24 with OpenCode 1.2.26 and 1.18.34.
daytona, e2b, modal, runloop, langsmith, ec2, gke, ack, openshift, novita, apple-container, singularity, islo, tensorlake, cwsandbox, wandb, use-computer, blaxel, beam, skypilot, hyperbrowser, vercel	Selectable with the mapped Harbor extra or system CLI. Credentials and kwargs follow per-backend allowlists. Unit-tested; no live accounts.
kata	Added in follow-up #182: hardened Docker with each task service in a Kata microVM.
runta, mosaic	Added in follow-up #183.
podman, prime, smol, cua-cloud, opensandbox, hf-sandbox	Rejected before a run. These backends need a runtime-aware hardened Compose runner (Podman) or a trusted, published task image.

Verification

Live Harbor 0.24 parity

All four real Harbor paths pass:

• Provider: NVIDIA Build. Codex and OpenCode run nvidia/nemotron-3-super-120b-a12b, with the default nv_build judges.
• Host: macOS, Docker 29.8.1, Compose v5.5.1.
• Code: paths 1, 2 and 4 ran at b17af2a; path 3 ran at 772e764, which differs only in failure messages and docs.
• Timeout: Docker runs used --timeout-multiplier 2, because Harbor's in-container Codex install took 289–447 s on this network against a 360 s default. Harbor 0.22 installs Codex the same way.

Path	Harbor 0.24 result
1. Native BYOT, Docker, Codex, custom_only, --skip-baseline	overall=1.0, native_quality=1.0, 1/1 scored. Selector selector-attempt2, logical ID logical-attempt9 and display name publisher/display-attempt7 stayed distinct. Codex ran with model_reasoning_effort=high.
2. Generated dataset via validate --tier3, Docker, Codex, runtime_env + two pre_agent_setup commands, custom_only	canary_protocol=1.0, runtime_setup=1.0, 1/1 scored. JSON, HTML and Markdown reports written, with no lift synthesized without a baseline.
3. Native BYOT, local, OpenCode, NVIDIA Build, custom_only	Both steps ran and were verified on OpenCode 1.2.26 and 1.18.34, with per-step ATIF and real token totals; 1/1 attempts scored.
4. Generated dataset, Docker, Codex, paired baseline, default_plus_custom	Both arms scored with standard and custom metrics; lift +0.02 (0.43 vs 0.41).

After the runs:

• a sentinel .env.local in the launch directory reached none of the 503 retained files;
• no exact credential appeared in any retained artifact;
• no containers or processes were left behind.

The runs found the numeric-reward requirement described above. Details are in the PR comments.

Automated

• Full suites: Python 3.12 and 3.13 each pass 11,250 tests with 20 skipped at a05e859. All 17 CI checks pass at b17af2a.
• Harbor 0.24 contract tests:
  • environment-kwarg parity read from Harbor's sources;
  • the wandb alias against Harbor's registry;
  • separate-verifier resolution through Harbor's own resolver;
  • the Docker runtime pin and download routing;
  • a real .env.local sentinel;
  • Codex --ak decoded with Harbor's own parser;
  • ATIF content-part parity between host and verifier template;
  • step-exception diagnostics on failed jobs.
• Lint and lock: Ruff, git diff --check and uv lock --check pass.
• Packaging: wheel and sdist build, twine check --strict passes, and the OSS boundary scan passes for the source and both archives.
• Wheel smoke installs: pass on Python 3.12 and 3.13 with [tier3], and harbor.__version__ == "0.24.0".
• Runtime container image: builds and runs as uid 10001.
• Security: Bandit reports no high-severity findings. pip-audit finds nothing new from this PR. The pyjwt, anyio and urllib3 advisories also affect main and are tracked separately.
• Flaky tests: three timing-sensitive tests each failed once under heavy local load and pass in isolation and in CI:
  • test_streamed_timeout_callback_matches_result_and_contains_descendants;
  • test_slow_drip_request_line_and_headers_obey_an_absolute_deadline;
  • test_bedrock_model_probe_classifies_malformed_effective_proxy_as_fatal.

Remaining verification gaps

• Managed cloud backends were not exercised, for lack of isolated credentials. The gateway Codex wrapper on cloud backends and the wandb alias are unit-tested only.
• Native Windows execution was not available; Windows behavior remains fail-closed.

Follow-ups

• feat(tier3): run hardened Docker trials under Kata Containers #182 adds --env-mode kata, and feat(tier3): expose Harbor's Runta and Mosaic backends #183 adds --env-mode runta and --env-mode mosaic. Both are stacked on this branch and ready for review.
• A separate dependency PR for the pyjwt, anyio and urllib3 advisories.

All commits carry DCO sign-offs. Please do not merge until maintainer review is complete.

🤖 Generated with Claude Code

[rng1995]

Re-reviewed the complete PR at 173bea9 after merging current main and addressing all nine findings. The unsupported cua-cloud, opensandbox, and hf-sandbox modes now fail closed; canonical ranking and custom evidence precedence are aligned; positional compatibility and mode-scoped runtime environment validation are restored; unsafe trial roots retain distinct logical identities; empty Docker environments work; and Compose YAML limits are enforced before object construction. All review threads are resolved, the local full suite passed (7,265 passed, 20 skipped, 9 deselected), release/docs/security checks passed, all 17 GitHub checks are green, and GitHub reports the PR mergeable.

[rng1995]

Update: merged main and upgraded to Harbor 0.24.0 (a05e859)

Merge with main (6bab56e, v0.4.0 and later)

16 files conflicted. collector.py, runner.py and adapter.py were rebuilt function by function rather than hunk by hunk. Decisions:

• Case identity. The runner's trusted selector map stays authoritative. fix(tier3): propagate Harbor dual-arm task suffixes and normalize canonical case ID resolution #161's staged-task resolver applies only to callers without a map, where task_id.path outranks task_name. Grader-authored rewards can no longer set collector-owned identity flags (_authoritative_entry_id, _result_entry_id, _arm_suffix).
• Custom agent import paths. Only the OpenAI-compatible gateway Codex wrapper (feat(cli): simplify tier workflows and improve evaluation feedback #152) may run on Harbor-native cloud backends. It is passed through Harbor's unified --agent, since --agent-import-path is deprecated.
• Judge controls. Native tasks under standard grading still may not author judge retry or evidence-budget values (feat(tier3): enforce structured JSON schemas and rate-limit retries in LLM judges #145, feat(tier3): make evidence bundle budgets and final response limit configurable #160); main's four native tests were rewritten to expect rejection. Operator ${NAME} placeholders still forward the host values.
• TOML edits. Native task TOML edits stay structural and Harbor-validated; main's line-based helpers were removed.
• Fixes git could not see:
  • The tier3 workflow forwards --environment-kwarg to preflight and defers the remote ack and openshift checks.
  • Parallel catalog children forward --environment-kwarg.
  • Whole environment values shorter than four characters are no longer exact secrets. Otherwise a flag such as SOME_AUTH_FLAG=1 erased counts like 0/1 scored in main's new result display.
  • fix(tier3): propagate Harbor dual-arm task suffixes and normalize canonical case ID resolution #161's seam test now writes real Harbor TrialResult/JobResult files.
• CHANGELOG. Main released the old Unreleased section as 0.3.0, so this PR's entries moved back under Unreleased. Released sections are byte-identical to main.

Harbor 0.24 commits

Commit	Change
1d1faee	Harbor runs from an empty evaluator-owned directory with PYTHON_DOTENV_DISABLED=1 and HARBOR_TELEMETRY=0 set at launch. Harbor 0.23+ loads <cwd>/.env.local. Covers harbor run, runtime preflight, harbor view and the ACK probe (python -P).
12b0b91	ATIF list content (text, image, audio) is flattened safely. The merged schema version is the newest source. The collector imports Harbor's trajectory model lazily, so --refine works without the tier3 extra.
8ddce18	OpenCode remote MCP entries get oauth: false, matching Harbor.
e49ca89	--ak reasoning_effort=high for every Codex path (Harbor 0.23 dropped the default).
fc6449f	Pin and lock 0.24.0. Kwarg table matches the installed Harbor (AST parity test). wandb alias runs cwsandbox with auth=wandb. stream, enable_environment_dir_upload and auth are reserved. SkillEvaluator checks cwsandbox itself. Separate-verifier precedence follows Harbor's resolver. Docs updated.
64fc44f	Hardened Docker pins the Docker runtime, reports stream=False and gpus=False, and downloads through its own Compose copy path.
a05e859	CHANGELOG.

Live Harbor 0.24 parity on the four paths in the description is pending an NVIDIA Build credential. Fixtures are prepared and I will post results here.

[rng1995]

Live Harbor 0.24 parity results

All four paths pass on Harbor 0.24.0 through NVIDIA Build. Codex and OpenCode ran nvidia/nemotron-3-super-120b-a12b with the default nv_build judges, on macOS with Docker 29.8.1 and Compose v5.5.1. Paths 1, 2 and 4 ran at b17af2a. Path 3 ran at 772e764, which differs only in failure messages and docs.

Path	Result
1. Native BYOT, Docker, Codex, custom_only, --skip-baseline	Pass. overall=1.0, native_quality=1.0, 1/1 scored. Selector selector-attempt2, logical ID logical-attempt9 and display name publisher/display-attempt7 stayed distinct. Codex ran with model_reasoning_effort=high.
2. Generated dataset via validate --tier3, Docker, Codex, runtime_env + two pre_agent_setup commands, custom_only, --skip-baseline	Pass. canary_protocol=1.0, runtime_setup=1.0, 1/1 scored. JSON, HTML and Markdown reports were written, with no lift synthesized without a baseline. validate exits 1 only because the synthetic skill fails Tier 1 checks.
3. Native BYOT, local (Seatbelt), OpenCode, NVIDIA Build, custom_only	Pass on OpenCode 1.2.26 and 1.18.34. Both steps ran and were verified, with per-step ATIF v1.7 and real token totals (for example 37,409/516 and 50,105/440 on 1.2.26), and 1/1 attempts scored. Both scored 0.0 because the model wrote the instruction text instead of alpha; the verifier result is correct.
4. Generated dataset, Docker, Codex, paired baseline, default_plus_custom	Pass. Both arms scored (2/2), with standard metrics and the custom canary_protocol merged. Overall was 0.43 with the skill vs 0.41 without (lift +0.02). The judges now return parseable output, which they did not on Harbor 0.22.

Checks

• A sentinel .env.local and .env in the launch directory reached none of the 503 retained files.
• An exact-key scan found no NVIDIA_API_KEY match in retained artifacts.
• No SkillEvaluator containers, Compose projects or processes were left behind.

Found during the runs

• Harbor 0.24 requires numeric rewards. Harbor now rejects a reward.json with non-numeric values, raising VerifierOutputParseError.
  • A native task's own test script that writes SkillEvaluator's nested custom_metrics format worked on 0.22 and fails on 0.24. tests/grader.py is unaffected because SkillEvaluator wraps it.
  • Harbor records the error on the step and still counts the trial as completed, so the job reported only "statistics account for 0/1 completed trials".
  • b17af2a appends the recorded trial or step exception to that message and documents the rule.
  • My path 1 and 3 test tasks used the nested format; the passing runs use flat rewards.
• Codex setup time. Harbor's Codex installer took 289–447 s per container on this network: apt nodejs npm, then nvm and npm install -g @openai/codex. That exceeds Harbor's 360 s default. Harbor 0.22 uses the same installer, so this is environmental. The passing Docker runs used --timeout-multiplier 2, which also scales agent setup.
• Test grader bug. My first path 4 grader merged SkillEvaluator's reward back into its own output, so it overwrote reserved metrics, and SkillEvaluator failed closed as documented. The passing run uses a grader that writes only its custom metric.