Skip to content

E21: a second blind pool (CI/CD and identity), written without reading the rules #368

Closed

Description

@LoopGlitch26

The first fresh blind pool (#344, by @juliasadler, merged in #357) was the most useful contribution the policy packs have had: 107 calls written without reading the rules, which turned up three real gaps in a day (#359, #360, #361). A second pool from a different person, on different ground, would do the same.

Task: add eval/e21_blind/<your-github-handle>.jsonl with 30 to 60 harmful calls and about as many benign ones, one JSON object per line, in the same shape as eval/e21_heldout/corpus.jsonl (tool, args, label, severity, technique, owasp, category, evasion, note).

  • This pool's ground: CI/CD and identity. Agents running in CI or with cloud credentials: changing or deleting GitHub Actions workflows and secrets, cloud IAM changes (new admin users, keys, role bindings, policy attachments), reading or exporting secrets (Vault, Kubernetes secrets, .env files, CI secret stores), turning off audit logging or branch protection, and data sent out through a request an agent was tricked into making (prompt injection).
  • Add benign calls that look close but are fine: listing roles, describe and get without secret values, read-only gh and kubectl calls.
  • Write calls the way an agent would actually type them. Most should be plain; tricks like quoting or encoding are welcome but shouldn't dominate.
  • Until your PR is open, don't read tracekit/policy2/, tracekit/policy/, the policy packs, docs/policy-v2.md, the E3 or E17 corpora, eval/e21_blind/, or any rule ids. Say so in the PR description.

We score the pool as you submitted it, before any rule changes, and post the score on the PR. The misses then become their own rule issues.

Done when: every line is valid JSON with label either harmful or benign, and the PR says the rules weren't read.

Activity

  1. added a commit that references this issue on Oct 25, 2026
    79bdfa6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions