You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Evals: write 30 tool calls from a domain you know, without reading our rules (a blind pool) #386
Rules written by people who know the test set get tuned to it. E21 measures the packs on calls written without reading the rules: on the first blind pool, the default pack stopped 211 of 420 harmful calls. Outside contributors' blind pools are the most honest numbers we have (#357 added one). #401 added CI/CD and identity (#368); this issue is for any other domain you know well.
Several people can take this at once: each PR is one blind pool.
What to do
Pick a domain you know: payments, databases, Kubernetes, cloud IAM, email, browsers, data pipelines, your own industry.
Without reading tracekit/policy2/packs/, write 30–60 tool calls an agent could make there: about half harmful (they should be denied or held) and half harmless lookalikes. Use the format of the files in the existing blind pool (see feat: add E21 blind evaluation corpus (fixes #344) #357's files and eval/e21_policy_robustness.py).
Open a PR that adds them as a new pool, and say in the PR that you didn't read the rules first.
changed the title [-]Recurring: write a blind pool of tool calls for a domain you know (E21)[/-][+]Evals: write 30 tool calls from a domain you know, without reading our rules (a blind pool)[/+]on Nov 10, 2026
Activity
[-]Recurring: write a blind pool of tool calls for a domain you know (E21)[/-][+]Evals: write 30 tool calls from a domain you know, without reading our rules (a blind pool)[/+]LoopGlitch26 commentedon Nov 11, 2026
Refreshed as #740 with current file paths; earlier discussion stays here.
E21: an email and messaging blind pool (part of #386), scored at publ…