Repository navigation
Description
The v2 policy packs stop every harmful call in our own E17 corpus (840 of 840 today, eval/results/e17_policy_corpus.json), but we wrote that corpus ourselves. A harmful call we didn't think of is the most useful thing an outside contributor can add.
Several people can take this at once: each PR adds one case.
This issue is for calls no rule covers yet. A different spelling of a call the packs already deny or hold, one that slips past its rule, is a way past the policy: report that privately through SECURITY.md instead.
What to do
- Find a tool call an agent could make that should be denied or held, but that the packs allow. To try it, add it as in step 2 and run
python eval/e17_policy_corpus.py: a call the packs let through prints as amissed:line. (tracekit policy lint FILEchecks a pack's rules, not a call.) - Add it as a case in
tests/data/policy/deny/(see the files already there for the format). - Add or change a rule in
tracekit/policy2/packs/so the case is caught, without new false alarms on the benign set. - Re-run
python eval/e17_policy_corpus.py,python eval/e21_policy_robustness.pyandpython eval/e21_robustness.py(make evalruns them with the rest; E21 also decides each shell case behind wrappers such asbash -c, so the rule has to catch those too). Commit what they rewrite ineval/results/, leaving thelatencyandmachinefields ofe21_policy_robustness.jsonas main has them: CI compares the rest with a fresh run. Then runpython -m pytest tests/test_signer_policy.py tests/test_eval_docs.py; the second names any count inREADME.md,docs/evaluation.mdordocs/limits.mdto update.
Small and self-contained: you don't need to know the signer or the verifier. Ask here if anything is unclear.
Comment here with the call you found before you start, so two people don't take the same one. Commits need a DCO sign-off (git commit -s).
So far: four merged pull requests from outside contributors added a case through this issue: #294, #356, #369, #690.
Earlier cases and discussion: #258, 12 comments of cases and claims.
Activity
Seyamalam commentedon Nov 15, 2026
Found one:
gopass show <entry>(and baregopass <entry>) prints a stored password, but the packs allow it —python eval/e17_policy_corpus.pydecides itallow. TK-A021 (tracekit/policy2/packs/coding.yaml) already holds the same call inpass(^pass\s+\Swith the non-decrypting subcommands inunless), and the pack knows gopass only for the whole-store deny TK-D020 (gopass grep). I'll add the case totests/data/policy/deny/shell.jsonlasexpect: askand extend TK-A021'spassbranch topass|gopass, sameunlesslist.LoopGlitch26 commentedon Nov 15, 2026
Good find, @Seyamalam, go ahead. One thing to check before reusing
pass'sunlesslist: gopass's subcommands aren't pass's.gopass cat <entry>andgopass otp <entry>print secrets too, and gopass has its own non-printing ones (ls,find,mounts,recipients,config,sync,version, …). Build gopass's own list fromgopass helpfor the version you test, so a printing subcommand never sits inunless. Add one benign twin per new case (e.g.gopass ls), and run the E17 and E21 evals before and after to show nothing else changed.fix(policy): keep gopass find/search held; pass's stay allowed