Repository navigation
[FEA]: Upgrade Tier 3 Harbor backend from 0.13.2 to 0.20+ #79
Description
Problem to solve
SkillEvaluator's live main branch still hard-pins the Tier 3 backend to harbor==0.13.2 in both pyproject.toml and uv.lock. The requested baseline, Harbor v0.20.0, is several releases newer; upstream is currently at v0.22.0.
This is not only a lockfile update. SkillEvaluator constructs Harbor CLI commands, imports Harbor agent/environment/model APIs directly, subclasses DockerEnvironment, imports a private Docker helper, generates Harbor task configuration, and consumes Harbor's persisted job/trial/reward artifacts. Those compatibility surfaces need to be checked across the version jump.
Goal: support Harbor >=0.20.0, preferring the latest stable compatible release available during implementation. If an exact 0.20.x pin is required, document the concrete incompatibility that prevents using the current stable release.
Proposed approach
- Update the Tier 3 dependency constraint and regenerate
uv.lock. - Audit and adapt the Harbor integration surface:
harbor runflags, custom agent/environment import paths, environment kwargs, verifier env, timeouts, resources, and non-interactive execution;- installed-agent wrappers, local environment support, environment preflight, and the secure Docker subclass/private helper boundary;
- generated and native task configuration, environment variables, health checks, MCP configuration, and custom Compose tasks;
- aggregate and per-trial result schemas, multi-step rewards, failure states, ATIF trajectories, artifact locations, retention, and report generation.
- Add focused regression coverage for every compatibility change.
- Update Tier 3 installation/backend documentation and the changelog for any changed behavior or supported-version constraint.
Acceptance criteria
-
pyproject.tomlanduv.lockresolve a supported Harbor version>=0.20.0; the selected pin/range and rationale are explicit. - Clean Python 3.12 and 3.13 installs work, including installation of the built wheel with
[tier3]. - The full unit, integration, packaging, and security suites pass with the upgraded dependency.
- Harbor CLI flags and direct Python API usages are reviewed, especially the private Docker helper and subclass methods.
- Generated
evals.jsontasks and native Harbor/BYOT tasks both validate and execute. - Real retained-output E2E runs complete through Harbor with Docker and local modes, at least two supported agents, with-skill/baseline and
--skip-baseline, plus multiple attempts with stop-on-pass. - E2E coverage exercises
runtime_env, pre-agent setup, artifact retention, and representative task/network configuration combinations. - Verification inspects scored results, aggregate and per-trial status, rewards, trajectories/token metrics, retained Harbor artifacts, JSON/HTML/Markdown report correctness, error handling, secret redaction, and cleanup—not only exit status.
- At least one Harbor-native cloud backend is exercised when credentials/infrastructure are available; otherwise the exact untested integration and reason are recorded.
- A final code-review pass checks edge cases, option interactions, output correctness, and regressions.
Additional context
- Current SkillEvaluator pin: pyproject.toml and uv.lock
- Requested upstream baseline: Harbor v0.20.0
- Current upstream release: Harbor v0.22.0
- Upgrade comparison: v0.13.2...v0.20.0
Before submitting
- I agree to follow this project's Code of Conduct.
- I searched open and closed issues and pull requests and found no duplicate.
Activity
chrisknvidia commentedon Aug 25, 2026
@rng1995 : FYI