Repository navigation
Policy: hold other common ways to switch off AWS detection in TK-C010 - #543
Conversation
Signed-off-by: shrinivas-sn <shrinusn2001@gmail.com>
|
Thanks for these. Six policy PRs is more than we can review well at once, so we'll take them one at a time, in the order you opened them, starting with #543. They all edit the same pack, test corpus and E17 results, so after each one merges, the next will need main merged in and E17 re-recorded. #544 also needs a new rule ID: TK-A049 is taken by #513, which was opened first. Please don't claim more issues until these are in; the other good first issues are there for other first-time contributors. From now on, comment on an issue and wait to be assigned before you start. |
|
Review of this one. The look-alikes are good (
When CI is green and a second review passes, I'll merge it. |
Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
…ing about these packs Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
|
Thanks for this. I pushed two commits to this branch so it can land here, as your pull request: a merge of main with the review's changes, and one that keeps E21's recorded latency rows.
|
…-run, E21's recorded latency rows kept Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
Brings in #582 (TK-A052, TK-A053; coding pack 2026.10-v2-63), #543 (TK-C010), #612, #595, #621 and the commits before them. - coding.yaml: version 2026.11-v2-64, the next free one. server-17, strict-10, server-core-1 and dev-1 are numbers main hasn't used. - docs/policy-v2.md: merged word by word, main's TK-A052 and TK-C010 text with this branch's. - docs/evaluation.md: main's "refreshed on macOS" note with this branch's dev-row note. - eval results: main's files, then E17, E21 rewrites (no --rebaseline) and E21 policy robustness re-run. The recorded latency rows are main's, and this branch's dev row (5.44 / 6.28) as before.
|
Thanks again for this. If you use a coding agent (Claude Code, Codex, Cursor, Gemini CLI or OpenCode), would you run Tracekit on it for a week? After a week, |
What and why
Fixes #539. Follows up on #532.
#532madeTK-C010(tracekit/policy2/packs/server-cloud.yaml) hold AWS commands that turn off audit logging and threat detection. As noted in #539, several other common methods for switching off, disabling, or evading AWS detection and audit trails still ran without a hold.This extends
TK-C010to hold the following detection-disable commands across both regex engines:aws guardduty update-detector ... --no-enable: common method for switching GuardDuty offaws guardduty update-detector ... --features ... Status=DISABLED: disabling individual GuardDuty feature detectorsaws guardduty create-filter ... --action ARCHIVE: auto-archiving incoming findings to hide detectionaws cloudtrail update-trail ... --s3-bucket-name <bucket>and... --no-is-multi-region-trail: diverting audit trail destinations or turning off multi-region coverageaws configservice delete-delivery-channel(andaws config delete-delivery-channel): removing the audit stream delivery mechanismaws securityhub batch-disable-standards: turning off security standards compliance checksaws route53resolver delete-resolver-query-log-config: deleting DNS query log audit configsaws organizations leave-organization: detaching an account from organizational logging, GuardDuty, and SCPsOrdinary modifications remain allowed:
aws guardduty update-detector --enableaws guardduty update-detector ... Status=ENABLEDaws guardduty create-filter ... --action NOOPaws cloudtrail update-trail --enable-log-file-validationaws cloudtrail update-trail --s3-key-prefix <prefix>aws configservice describe-delivery-channelsaws securityhub get-enabled-standardsaws route53resolver list-resolver-query-log-configsaws organizations describe-organizationAs noted in #539,
aws guardduty create-ip-setis left unheld because trusted IP lists are also routine setup; a limit note is added todocs/limits.md.Pack bumped to
2026.10-server-cloud-9.Eval & corpus numbers:
tests/data/policy/{deny,benign}/cloud.jsonl.eval/e17_policy_corpus.py: 788 of 788 harmful calls stopped (100.0% recall), 1 false alarm in 852 (0.12%), 0 engine mismatches between RE2 and regex fallback.tests/test_eval_docs.pyandtests/test_docs.pypass.How I ran it as a user
Ran
Engineonserver.yamlagainst both benign and held calls:tracekit policy lintis clean forserver-cloud.yaml,server.yamlandstrict.yaml.Tests run on Windows 11, Python 3.13:
pytest tests/test_signer_policy.py tests/test_eval_docs.py tests/test_docs.py: 149 passed.pyflakes tracekit tracekit_sdk.py tests eval: clean (exit 0).Checklist
make checkpasses (lint, tests, build);make test-tstoo ifsdk/typescript/changedCHANGELOG.md)docs/limits.md,README.md,docs/evaluation.md)git commit -s, seeCONTRIBUTING.md)Security invariants
Evidence format
core.sig_message,ledger.make_record, the v1 schema, how the verifier treats existing bundles) or the published v2 format