Skip to content

refactor: benchmark-driven general harness hardening - #71

Draft
dollspace-gay wants to merge 7 commits into
mainfrom
feature/benchmark-driven-general-harness-hardening
Draft

refactor: benchmark-driven general harness hardening#71
dollspace-gay wants to merge 7 commits into
mainfrom
feature/benchmark-driven-general-harness-hardening

Conversation

@dollspace-gay

Copy link
Copy Markdown
Member

Scope

Establish honest Harness-Bench and Terminal-Bench baselines, then repair broad harness defects without task-specific shortcuts.

Current delivered fixes:

  • honor documented global --dangerously-skip-permissions for ACP automation while preserving hard host safety boundaries
  • accept standard ACP text content-block prompts while retaining legacy string compatibility and bounded validation

Integrity constraints

  • upstream benchmark tasks, prompts, budgets, adapters, graders, and oracles remain unchanged
  • no task-ID detection, expected-output inspection, benchmark-specific routing, or score manipulation
  • fixes must improve the general harness and be validated outside the triggering benchmark task

Verification so far

  • Rust 1.98 formatting and strict all-target/all-feature Clippy pass
  • focused ACP, config, IDE, and CLI tests pass; known parallel fixture collisions remain tracked separately
  • pinned acpx@0.13.2 real smoke reaches the provider through the documented global flag and standard ACP prompt shape

Current blocker

The configured Claude Agent SDK reports an expired OAuth session whose refresh failed. This is tracked as Crosslink #1238 and will be repaired without routing around the documented automation path.

@dollspace-gay dollspace-gay self-assigned this Aug 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant