Repository navigation
Reduce token usage through lower-tier models #49
Description
Pre-requisites
Maintaining an eval test harness on each prompt may be necessary before making these changes.
Issue
Some nodes such as prepare-foundry-harness and discover-base-test may not require the most advanced model due to their basic requirements for success.
Proposed Solution
The easiest stop-gap solution is to provide a config option where users can configure a lower-tier model on less important nodes conserving token usage for the critical nodes.
Rationale
If this tool is designed to be used by a protocol only once, this change will have little impact. However, if the protocol aims to use this tool continuously with every new deployed change, the token usage costs can add up. For security researchers who aim to use this tool beyond a single protocol, such optimizations can save money over time.
Activity
aviggiano commentedon Jul 15, 2026
Agreed, I'm currently working on evals and this could be an optimization target
aviggiano commentedon Jul 18, 2026
Suggested prerequisite for evaluating lower-tier model selection:
Synthetic acceptance tests should cover retries, checkpoint continuation, cache components, and incomplete pricing without relying on private execution data.
This makes model-selection decisions reproducible without depending on incomplete aggregate totals.
Focused accounting and terminal-metric work is tracked in #64, #65, #66, and #67.
aviggiano commentedon Aug 14, 2026
The model-profile mechanism requested here was implemented by PR #47. Remaining work is optimization rather than this issue's missing mechanism: token/time efficiency stays tracked in #515, with per-node/stage evaluation work in #80.
Closing as completed. The
help wantedlabel did not authorize additional implementation, and no optimization work is being started in this administrative pass.