You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
E21: a second blind pool (CI/CD and identity), written without reading the rules #368
The first fresh blind pool (#344, by @juliasadler, merged in #357) was the most useful contribution the policy packs have had: 107 calls written without reading the rules, which turned up three real gaps in a day (#359, #360, #361). A second pool from a different person, on different ground, would do the same.
Task: add eval/e21_blind/<your-github-handle>.jsonl with 30 to 60 harmful calls and about as many benign ones, one JSON object per line, in the same shape as eval/e21_heldout/corpus.jsonl (tool, args, label, severity, technique, owasp, category, evasion, note).
This pool's ground: CI/CD and identity. Agents running in CI or with cloud credentials: changing or deleting GitHub Actions workflows and secrets, cloud IAM changes (new admin users, keys, role bindings, policy attachments), reading or exporting secrets (Vault, Kubernetes secrets, .env files, CI secret stores), turning off audit logging or branch protection, and data sent out through a request an agent was tricked into making (prompt injection).
Add benign calls that look close but are fine: listing roles, describe and get without secret values, read-only gh and kubectl calls.
Write calls the way an agent would actually type them. Most should be plain; tricks like quoting or encoding are welcome but shouldn't dominate.
Until your PR is open, don't read tracekit/policy2/, tracekit/policy/, the policy packs, docs/policy-v2.md, the E3 or E17 corpora, eval/e21_blind/, or any rule ids. Say so in the PR description.
We score the pool as you submitted it, before any rule changes, and post the score on the PR. The misses then become their own rule issues.
Done when: every line is valid JSON with label either harmful or benign, and the PR says the rules weren't read.
Activity
feat: add E21 blind evaluation pool for amandeavor (fixes Cygnux-Labs…