Activity

1. harperreed commented on Mar 24, 2026

   

   harperreed

   on Mar 24, 2026

   MemberAuthor

   More actions

   Updated Results: v1.12.0 with Extended Thinking + Multi-Model

   Following the initial v1.11.x baseline, we shipped v1.12.0 with significant improvements and ran evaluations across 4 model configurations.

   Score Progression

   Version	Model	Passed	Total	Rate	Approx Rank
   v1.11.0	Sonnet 4.5	34	88	38.6%	~#71
   v1.11.2	Opus 4.6	43	88	48.9%	~#53
   v1.12.0	Sonnet 4.6 + thinking	42	88	47.7%	~#54
   v1.12.0	Opus 4.6 + thinking	48	88	54.5%	~#42
   v1.12.1	GPT-5.4	28	88	31.8%	~#80

   Best result: 54.5% with Opus 4.6 + extended thinking — up 15.9 points from the initial 38.6% baseline. Gap to Claude Code (58.0%) narrowed from 20 points to 3.5.

   v1.12.0 Changes That Drove Improvement

   Hex-side (v1.11.3 → v1.12.0):

   • Graceful max-turns exit — exit 0 with best effort instead of crashing (was losing ~3-5 tasks)
   • Default max turns increased from 20 to 50
   • Tool result truncation at 15K chars — prevents context window blowout
   • MaxTokens increased from 4096 to 16384 — longer code in single response
   • Stronger verification assertion in system prompt
   • Warm context in Harbor adapter — file listing prepended to task

   mux v0.7.0 (critical):

   • Extended thinking support for Anthropic (ThinkingClient wrapper)
   • OpenAI tool_calls response format fix
   • API retry with exponential backoff
   • Base URL support for all providers

   Extended Thinking Impact

   Config	Without Thinking	With Thinking	Delta
   Opus 4.6	48.9% (v1.11.2)	54.5% (v1.12.0)	+5.6

   Tasks newly solved with thinking enabled (vs v1.11.2 Opus without):

   • extract-moves-from-video — video frame analysis
   • feal-linear-cryptanalysis — cryptanalysis (no model solved this before)
   • largest-eigenval — numerical computing
   • make-mips-interpreter — MIPS CPU emulator
   • path-tracing + path-tracing-reverse — ray tracing (heavy math)
   • torch-pipeline-parallelism — distributed ML
   • write-compressor — compression algorithm design
   • build-pmars — build systems
   • openssl-selfsigned-cert — crypto/security
   • sparql-university — knowledge graphs

   These are all deep-reasoning tasks where extended thinking provides the most value.

   Model Comparison: Tasks Solved

   Union of all models: 66 unique tasks solved (75.0%)

   Solved by all Anthropic configs (30 tasks): bn-fit-modify, cobol-modernization, code-from-image, constraints-scheduling, custom-memory-heap-crash, distribution-search, feal-differential-cryptanalysis, financial-document-processor, fix-code-vulnerability, fix-git, git-leak-recovery, headless-terminal, hf-model-inference, kv-store-grpc, large-scale-text-editing, llm-inference-batching-scheduler, log-summary-date-ranges, mcmc-sampling-stan, merge-diff-arc-agi-task, modernize-scientific-stack, multi-source-data-merger, nginx-request-logging, password-recovery, portfolio-optimization, prove-plus-comm, pypi-server, pytorch-model-cli, qemu-startup, reshard-c4-data, rstan-to-pystan

   Opus 4.6 + thinking only (unique): extract-moves-from-video, feal-linear-cryptanalysis, largest-eigenval, make-mips-interpreter, path-tracing, path-tracing-reverse, torch-pipeline-parallelism, write-compressor

   Sonnet 4.6 only (unique): adaptive-rejection-sampler, build-pov-ray, configure-git-webserver, qemu-alpine-ssh, tune-mjcf

   GPT-5.4 Analysis

   GPT-5.4 scored 31.8% — significantly below its capability (scores ~50%+ through Codex CLI). Root causes:

   1. No extended thinking — OpenAI rejects reasoning_effort with tools on /v1/chat/completions
   2. Anthropic-native tool patterns — hex's system prompt, tool descriptions, and agent spell are optimized for Claude's instruction-following style
   3. 48 wrong answers, only 4 timeouts — model uses tools correctly but produces wrong solutions, suggesting prompt/format mismatch rather than infrastructure issues

   Remaining Gap to Claude Code (3.5 points)

   The biggest remaining opportunities:

   1. Thinking budget tuning — current 10K tokens may be too low for hardest tasks; experiment with 20K-50K
   2. Multi-attempt — running failed tasks a second time with different strategy
   3. Better warm context — currently just file listing; adding file sizes and first lines would help
   4. OpenAI provider optimization — separate system prompt/tool format for OpenAI models
   5. mux orchestrator thinking support — currently using a client wrapper; native support would be cleaner

   Reproducing v1.12.0 Results

   # Opus 4.6 with thinking (best config)
   harbor run -d terminal-bench@2.0 \
     --agent-import-path hex_terminal_bench:HexAgent \
     -m anthropic/claude-opus-4-6 \
     --ak version=1.12.0 \
     --ae ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
     --ae ANTHROPIC_BASE_URL=$ANTHROPIC_BASE_URL \
     --jobs-dir ./results \
     --agent-setup-timeout-multiplier 3.0 \
     -n 4