You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Minimize token/time spend for the same deliverable #515
Closing as not planned: no study has run since this was filed, so this has become a stale umbrella.
The tools for it already exist:
ultrafuzz stats <run-id> shows each node's tokens, cost and time.
Per-node and per-group model_profiles and the audit profiles in packages/config/audit-profiles.yml let you try cheaper configurations.
ultrafuzz eval compare checks whether a cheaper variant still finds the same issues.
The largest known launch-time overhead is the sealed execution snapshot. It is copied and checked in full before every run (#921 measured about 51k files / 772 MB per generation). Removing it is tracked in #921. That only saves wall-clock time; it does not reduce tokens. Stage-level evals (#80), which would make these experiments cheaper, were closed in the same pass; reopen them together if cost work becomes a priority.
Reopen with a specific node or config change and its eval compare result against a baseline.
Activity
[-]Minimize token spend for the same deliverable[/-][+]Minimize token/time spend for the same deliverable[/+]aviggiano commentedon Oct 5, 2026
Closing as not planned: no study has run since this was filed, so this has become a stale umbrella.
The tools for it already exist:
ultrafuzz stats <run-id>shows each node's tokens, cost and time.model_profilesand the audit profiles inpackages/config/audit-profiles.ymllet you try cheaper configurations.ultrafuzz eval comparechecks whether a cheaper variant still finds the same issues.The largest known launch-time overhead is the sealed execution snapshot. It is copied and checked in full before every run (#921 measured about 51k files / 772 MB per generation). Removing it is tracked in #921. That only saves wall-clock time; it does not reduce tokens. Stage-level evals (#80), which would make these experiments cheaper, were closed in the same pass; reopen them together if cost work becomes a priority.
Reopen with a specific node or config change and its
eval compareresult against a baseline.