Repository navigation
Conversation
Four deterministic predicates over a recorded InvestigationTrace: a source excerpt came back before the first proposal, every executed read and write stayed inside its boundary, executed usage stayed inside the budget the run was given, and the tool took exactly one proposal. One rule for all four: a predicate fires on what ran. A request the harness refused before it took effect (tool schema, tool budget, one-proposal cap) is counted as an observation, separately. Under that rule the call-count budget dimensions, the scope gate and a second proposal cannot fire on an executed call; only total tokens and wall clock, which are checked after a step, and zero proposals can. Generation 3 is published alongside 1 and 2 in every artifact. EXPERIMENT stays at 2, so no headline verdict moves; a fixture pins the v1 and v2 verdicts of all 39 runs from before generation 3, and none changed. A human-authored candidate has no trace, so its v3 verdict is its v2 verdict. Each predicate's tests were mutated to always-pass and went red (4 to 5 failures each) before being restored. lab-agent-report.ts prints the run table, k of n per cell, firings per predicate, refused requests apart from firings, and gateway-error runs as infrastructure failures. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The model agent.ts was written for, exercised for the first time: until the gateway account had paid credit it refused every Anthropic model with a free-tier 403, and the earlier sweep switched to other models without recording why. Six runs per cell, all committed: tight proposed 0 of 6, accepted 0 of 6 default proposed 0 of 6, accepted 0 of 6 generous-out proposed 6 of 6, accepted 0 of 6 All six proposals were near misses: five fail the same synthetic case, syn-relevant-not-bool, which none of the six agents inspected. Gateway meter for the cell set: $6.46, matching the trace sums. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
Item 5, the model
agent.tswas written for. 18 runs ofanthropic/claude-sonnet-4.5through the AI Gateway, six per budget cell, every run committed including the 12 that proposed nothing.Gateway meter for the 18 runs: $6.46, equal to the per-call costs summed from the traces.
The default model had never once run before this. The gateway refused every Anthropic model on the free tier with a 403, and the earlier sweep switched to kimi-k2 and gpt-4.1 without recording why.
All six Claude proposals were near misses (run 01: 59 of 60 applicable cases). Five fail the same synthetic case,
syn-relevant-not-bool, which none of the six agents inspected, though synthetic cases are inspectable.Item 6, trajectory verifiers, as spec generation 3. Four deterministic predicates over the recorded trace: read before proposing, inside the scope gate, inside budget, proposed once.
EXPERIMENTstays at 2.Over all 49 agent runs:
Refused before running, counted apart: 4 runs had a second proposal refused by the one-proposal cap, including both accepted kimi-k2 runs.
npx tsx web/scripts/lab-agent-report.tsreproduces every table from committed files.Web tests: 274 passed, 1 skipped.
🤖 Generated with Claude Code