[kweinmeister]

Summary

When running Tier 3 Harbor dual-arm (with-skill / without-skill) evaluations, native staged task.toml files retained their base [task] name without the arm suffix, and result collection could fail to correlate paired trials when Harbor produced external namespace prefixes (repo/, org__), attempt suffixes (__attempt-N, -attempt-N), or arm suffixes (-with-skill, -without-skill) in varying orders.

This change:

• Propagates arm suffixes to task.toml: Updates _rewrite_task_toml and copy_native_tasks_with_skill_mode in src/skillevaluator/tier3/harbor/adapter.py (wired from src/skillevaluator/tier3/harbor/runner.py) so [task] name in staged native task.toml files includes -with-skill / -without-skill alongside the staged task directory name.
• Commutative canonical case ID resolution: Updates src/skillevaluator/tier3/harbor/collector.py (_strip_attempt_suffix, _strip_arm_suffix, _canonical_case_id) to strip external repository/namespace prefixes and commutatively strip attempt and arm suffixes in any order while preserving legitimate case IDs (such as those in expected_ids, retaining skillevaluator-, or ending in -with / -without).
• Fail-fast validation & tests: Validates arm_suffix inputs across adapter APIs and adds parameterized unit and integration tests in tests/test_tier3_public_runtime.py.

Verification

• I am familiar with the Contributing Guidelines
• Added or updated focused tests
• Updated documentation for user-visible changes
• Ran make lint
• Ran make test
• Ran make build
• Did not add credentials, private datasets, or proprietary benchmark content

Release Impact

• No user-visible release note needed
• Updated CHANGELOG.md

[rng1995]

@kweinmeister Thanks for the contribution. The current normalization introduces cross-case identity collisions, and the native name rewrite can corrupt valid TOML. Please address the three inline findings with regressions before approval.

Local validation: 418 focused runtime, case-ID, metrics, adapter, collector, and security-attribution tests passed; Ruff passed. Separate reproductions exposed the reported gaps. All 17 reported CI checks pass.

There is also a merge conflict with main in CHANGELOG.md. Could you please resolve it so the updated PR can complete verification and move toward merge?

[rng1995]

Resolved the merge conflict with main and pushed 4ff18f5. Both the PR's release note and the new upstream entries are preserved. All non-changelog files were verified unchanged from their respective PR/main versions.

Local verification: 48 case-ID/report dependency tests passed; diff checks and Fern validation passed (authenticated redirects check skipped). The three code-review findings remain open, so this is not an approval.

[kweinmeister]

Merged main and pushed updates for the review comments:

• Authoritative reward entry_id values are preserved without stripping arm suffixes or skillevaluator- prefixes, and _canonical_case_id only strips generated wrappers when matching against expected_case_ids, which is now passed through _annotate_security_attribution and _save_trials.
• Native staging always appends one arm suffix to [task].name and records [metadata].entry_id when absent, while reward extraction strips at most one known arm suffix from task_name.
• [task].name is updated via a token-aware TOML scanner and validated with tomllib.loads so triple-quoted strings, commented [task] headers, and [metadata].name fields are handled cleanly.

[rng1995]

@kweinmeister Thank you for your contribution and revisions. I verified and resolved two earlier findings: authoritative-ID attribution and authored arm-suffix handling. The inline/escaped TOML representation and metadata-versus-directory identity findings remain reproducible; I followed up in their existing threads.

Local validation: 382 focused tests passed, four skipped; diff checks passed. CI has 16 successful checks; Tier 2 macOS failed before tests during dependency installation because PyPI returned HTTP 503 for setuptools. Please rerun that job after addressing the remaining findings. No merge conflicts are reported, but the two code blockers still prevent approval.

[kweinmeister]

Merged the latest main and pushed fixes for the two remaining review threads:

• The TOML scanner now decodes quoted/escaped keys and tracks inline tables (task = { name = ... }), dotted keys, and metadata subtables, failing closed if a string task name or canonical metadata.entry_id cannot be persisted.
• Native task identity (metadata.entry_id when present, otherwise directory name) is now validated upfront during staging and aligned across the runner's expected case IDs, staged task.toml collection lookup, and the dataset snapshot.

[rng1995]

@kweinmeister Thank you for your patience through several review rounds, and for your contribution to SkillEvaluator!

I re-reviewed the branch, and both remaining findings are fixed:

• Native task.name rewrite: I staged 60+ valid TOML spellings in both arms. They covered inline task = { name = ... }, dotted task.name, quoted and unicode-escaped keys, commented headers, triple-quoted and literal strings, CRLF, and decoy keys. Every one either got exactly one arm suffix and round-tripped through tomllib and Harbor's TaskConfig, or was rejected before launch. With adjacent case-1 / case-1-with-skill tasks, dual-arm collection now has no Missing/Excess errors, and pass@k and lift are complete. Your new tests fail on 18cac4a and pass at head.
• metadata.entry_id vs directory identity: with physical-case + entry_id = "authored-entry", the runner's expected IDs, collection, persisted trials and the dataset snapshot now agree in single-arm, dual-arm and stop_on_pass runs. Duplicate and colliding entry IDs are rejected before Harbor runs.

One regression found during re-review, now fixed in the merge with main: a dual-arm set with directory y (with a [task] table) next to y-with-skill (without one) failed collection on the previous head. The error was Missing scored attempts for cases: y-with-skill / Excess scored attempts for cases: y; main handled the same set fine. Harbor names a task without [task] after its directory, which collided with y's suffixed name. The staged-name map then dropped that key as ambiguous, and the resolver stripped -with-skill from the task name.

#119 (merged into main) made Harbor result.json case IDs prefer the task_id.path directory over the repository-prefixed task_name, and that conflicted with your resolver. In 0805758 I kept your _resolve_harbor_result_entry_id and applied #119's rule inside it. When the result has a task path with no metadata.entry_id or map entry, the directory name is now returned before falling back to task_name. That also fixes the case above in every run mode. Please take a look at the collector.py part of 0805758 in case it conflicts with an intent I missed.

Validation on 0805758: Ruff is clean, the full suite passes (9,820 passed, 18 skipped, including #119's path-first test), and all CI checks pass.

Non-blocking, for a follow-up:

• CHANGELOG.md doesn't mention two behavior changes for native Harbor sets: duplicate or invalid metadata.entry_id values are now rejected at staging, and tasks whose entry_id differs from the directory are reported under the entry_id.
• _native_projection_entry_ids still treats entry_id = 0 as missing (a truthiness check), while _native_entry_id uses is not None.

Approved.