Skip to content

[feat] Count a scorer's labels by value in the dashboard #1472

Closed

Description

@SaulMoro

Parent: #1137

Goal

The scorer view of #1182 counts a measurement's labels as passed and failed. A scorer with more than two labels loses what told them apart. partial and bad both fail. For a routing question (#1469), missed and extra both fail, and fit and idle both pass, so its precision and recall cannot be read.

Count the labels of a measurement by value too.

Example

teamai:routing:ops:release · rubric 5b2d0a · session · claude / <recorded-model> · attended · 30 days
label    score   labels
fit      1       31
idle     1       112
extra    0       6
missed   0       9
158 labels · 143 passed · 15 failed · 91%
precision(ops:release) ≈ 31 / (31 + 6) = 84%
recall(ops:release)    ≈ 31 / (31 + 9) = 78%

The view shows the counts. Precision and recall are read from them; the view does not compute them.

Scope

  • Under the pass rate of a measurement, one row per label value it holds, with the value, its score and its count. Rows go by score, highest first, then by value.
  • The same labels as the pass rate: after the newest-scoredAt dedup and the window, so the counts add up to the measurement's label count.
  • Values as the labels stored them. The scorer file is not read to rename, add or reorder them.
  • The read API returns the counts on the selected measurement (selected.cohort). Docs: the Scorers section of docs/guide/dashboard.md and docs/guide/zh-CN/dashboard.md.

Acceptance criteria

  1. A measurement with fixture labels fit ×3, missed ×1 and extra ×2 shows those three rows, and they add up to its label count of 6.
  2. A label rescored in a later month with another value counts once, under its newest value.
  3. A value with no label in the window has no row.

Out of scope

  • Counts by value inside groups and in the split at a team commit.
  • Computing precision, recall or any other metric.
  • Labels from benchmark runs, which the view already leaves out.

Dependencies

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions