Repository navigation
Create eval per node or group of nodes (stages) #80
Description
Example:
We don't need to always re-run the whole "project setup" stage every time when what we care the most is the number of bugs found
It would be interesting to always start from the same baseline (eg BaseTest.t.sol)
We need to think about how to do evals that are not metric based
Activity
mrthankyou commentedon Jul 25, 2026
It would be good to know which metrics type a node has and how to properly evaluate them. So far we have three types of metrics:
I'm digging deep into this so hopefully will have something to share in a week or two (limited availability atm).
aviggiano commentedon Oct 5, 2026
Closing as not planned. No one has worked on this since the 2026-07-25 comment, and the metric-type analysis it promised was never posted. Nothing has been built for it either: eval variants (
EvalVariantin packages/evals/src/types.ts) can't start from a pinned baseline, so every eval row still runs the full campaign, setup included.The closest thing today is manual. Fork a finished baseline run at the checkpoint frame after setup with
ultrafuzz fork <run-id> --frame <n>(useultrafuzz timeline <run-id>to find the frame). Add--reset-node <node-id>to set one node back to pending before the fork starts. Forked runs are not part of an eval matrix, soultrafuzz eval scorewon't score them. Comparing them is up to you.#49 named this issue as the tracker for per-node/stage evals, and the trial loop in #84 (property-generation hill-climbing) would need it to avoid re-running setup on every trial. Reopen this if #84 is picked up or if full-campaign eval cost becomes a blocker.