Skip to content

Evals: write 30 tool calls from a domain you know, without reading our rules (a blind pool) #386

Closed as not planned

Description

@LoopGlitch26

Rules written by people who know the test set get tuned to it. E21 measures the packs on calls written without reading the rules: on the first blind pool, the default pack stopped 211 of 420 harmful calls. Outside contributors' blind pools are the most honest numbers we have (#357 added one). #401 added CI/CD and identity (#368); this issue is for any other domain you know well.

Several people can take this at once: each PR is one blind pool.

What to do

  1. Pick a domain you know: payments, databases, Kubernetes, cloud IAM, email, browsers, data pipelines, your own industry.
  2. Without reading tracekit/policy2/packs/, write 30–60 tool calls an agent could make there: about half harmful (they should be denied or held) and half harmless lookalikes. Use the format of the files in the existing blind pool (see feat: add E21 blind evaluation corpus (fixes #344) #357's files and eval/e21_policy_robustness.py).
  3. Open a PR that adds them as a new pool, and say in the PR that you didn't read the rules first.
  4. We run E21 on it and record the score, the misses and the false alarms, misses included. You don't need to fix the rules; misses become cases for Policy packs: add a harmful tool call the packs let through #258.

Commits need a DCO sign-off (git commit -s). Comment here with your domain before you start.

Activity

  1. added
    help wantedExtra attention is needed
    open to manySeveral people can take this at once: each PR or report is one contribution
    on Oct 25, 2026
  2. changed the title [-]Recurring: write a blind pool of tool calls for a domain you know (E21)[/-] [+]Evals: write 30 tool calls from a domain you know, without reading our rules (a blind pool)[/+] on Nov 10, 2026
  3. LoopGlitch26 commented on Nov 11, 2026

    @LoopGlitch26
    MemberAuthor

    Refreshed as #740 with current file paths; earlier discussion stays here.

  4. added a commit that references this issue on Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    good first issueGood for newcomershelp wantedExtra attention is neededopen to manySeveral people can take this at once: each PR or report is one contribution

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions