You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[feat] Count a scorer's labels by value in the dashboard #1472
The scorer view of #1182 counts a measurement's labels as passed and failed. A scorer with more than two labels loses what told them apart. partial and bad both fail. For a routing question (#1469), missed and extra both fail, and fit and idle both pass, so its precision and recall cannot be read.
Count the labels of a measurement by value too.
Example
teamai:routing:ops:release · rubric 5b2d0a · session · claude / <recorded-model> · attended · 30 days
label score labels
fit 1 31
idle 1 112
extra 0 6
missed 0 9
158 labels · 143 passed · 15 failed · 91%
The view shows the counts. Precision and recall are read from them; the view does not compute them.
Scope
Under the pass rate of a measurement, one row per label value it holds, with the value, its score and its count. Rows go by score, highest first, then by value.
The same labels as the pass rate: after the newest-scoredAt dedup and the window, so the counts add up to the measurement's label count.
Values as the labels stored them. The scorer file is not read to rename, add or reorder them.
The read API returns the counts on the selected measurement (selected.cohort). Docs: the Scorers section of docs/guide/dashboard.md and docs/guide/zh-CN/dashboard.md.
Acceptance criteria
A measurement with fixture labels fit ×3, missed ×1 and extra ×2 shows those three rows, and they add up to its label count of 6.
A label rescored in a later month with another value counts once, under its newest value.
A value with no label in the window has no row.
Out of scope
Counts by value inside groups and in the split at a team commit.
Computing precision, recall or any other metric.
Labels from benchmark runs, which the view already leaves out.
Activity
feat(scorers): count a measurement's labels by value (Tencent#1472)
feat(dashboard): page the failing labels and list a measurement's lab…
docs(dashboard): labels by value, paged failing labels, and the user …
feat(dashboard): show scorer pass rates, what drives them, and labels…