[YMuskrat]

Summary

A generated answer such as 401 currently invalidates an otherwise completed, rewarded trial when optional final ATIF token totals are absent, despite positive per-step token usage. Treat step usage as evidence that the trajectory is not a tokenless startup failure. Explicit Harbor exceptions and structured provider error events still invalidate trials.

Closes #198

The focused collector module passes 129 tests, including 11 added regressions. Full suite: 11,204 passed, 53 skipped, 25 baseline failures.

Verification

• Familiar with CONTRIBUTING.md
• Added focused behavioral regression tests
• Updated CHANGELOG.md
• Ran make lint (passed)
• Ran make test
• Ran make build (wheel and source distribution passed)
• No credentials, private datasets, or proprietary benchmark content added

The full suite used the repository's locked dependencies on Python 3.12 and pytest-xdist with four workers. It has 25 failures, all reproduced against unchanged main at f32c884. The identical failing test IDs occur on all three independent fixes; none is in a changed module. These include restricted Unix sockets/local subprocesses and existing SDK/CLI/report contracts. The full suite is not claimed green.

Release Impact

• Updated CHANGELOG.md

[chrisknvidia]

Reviewed ce62cba1b449294cca1c9df091eda8334514d439. No actionable new code defect found in per-step token detection, explicit runtime-error precedence, or downstream score/report handling. There were no earlier review threads to address.

Not ready to merge yet:

• GitHub reports CONFLICTING / DIRTY against current main 098f96a0d169878ab85b6801a8f4609b29d11e50. A read-only merge-tree check isolates the conflict to CHANGELOG.md. Update the branch and resolve it.
• Current-head CI, DCO, and Security runs are all action_required; they have not executed. Authorize the updated-head workflows and require passing checks after conflict resolution.

Verification: 395 local tests passed with Harbor 0.24.0, covering runtime failures, multi-step trajectories, negative-control evidence, unified reports and HTML reporting; changed-file Ruff passed. These use constructed inputs and real-library/report integration. No live agent/provider E2E or full-suite pass is claimed; the Docker server probe timed out.

[chrisknvidia]

@YMuskrat, please update this branch with the latest main, resolve the merge conflict in CHANGELOG.md, and push the updated branch. GitHub still reports this PR as conflicting. Once the conflict is resolved, the updated head needs passing required checks before it is ready to merge.

[rng1995]

Reviewed the change and its tests; no findings. The new tests fail on main and pass here, and the related test modules pass. The branch needs main merged in to clear a CHANGELOG.md conflict before it can merge.