Skip to content

Policy: hold other common ways to switch off AWS detection in TK-C010 - #543

Merged
LoopGlitch26 merged 3 commits into
Cygnux-Labs:mainfrom
shrinivas-sn:agent/issue-539
Nov 5, 2026
Merged

LoopGlitch26 merged 3 commits into
Cygnux-Labs:mainfrom
shrinivas-sn:agent/issue-539

Conversation

@shrinivas-sn

Copy link
Copy Markdown
Contributor

What and why

Fixes #539. Follows up on #532.

#532 made TK-C010 (tracekit/policy2/packs/server-cloud.yaml) hold AWS commands that turn off audit logging and threat detection. As noted in #539, several other common methods for switching off, disabling, or evading AWS detection and audit trails still ran without a hold.

This extends TK-C010 to hold the following detection-disable commands across both regex engines:

  • aws guardduty update-detector ... --no-enable: common method for switching GuardDuty off
  • aws guardduty update-detector ... --features ... Status=DISABLED: disabling individual GuardDuty feature detectors
  • aws guardduty create-filter ... --action ARCHIVE: auto-archiving incoming findings to hide detection
  • aws cloudtrail update-trail ... --s3-bucket-name <bucket> and ... --no-is-multi-region-trail: diverting audit trail destinations or turning off multi-region coverage
  • aws configservice delete-delivery-channel (and aws config delete-delivery-channel): removing the audit stream delivery mechanism
  • aws securityhub batch-disable-standards: turning off security standards compliance checks
  • aws route53resolver delete-resolver-query-log-config: deleting DNS query log audit configs
  • aws organizations leave-organization: detaching an account from organizational logging, GuardDuty, and SCPs

Ordinary modifications remain allowed:

  • aws guardduty update-detector --enable
  • aws guardduty update-detector ... Status=ENABLED
  • aws guardduty create-filter ... --action NOOP
  • aws cloudtrail update-trail --enable-log-file-validation
  • aws cloudtrail update-trail --s3-key-prefix <prefix>
  • aws configservice describe-delivery-channels
  • aws securityhub get-enabled-standards
  • aws route53resolver list-resolver-query-log-configs
  • aws organizations describe-organization

As noted in #539, aws guardduty create-ip-set is left unheld because trusted IP lists are also routine setup; a limit note is added to docs/limits.md.

Pack bumped to 2026.10-server-cloud-9.

Eval & corpus numbers:

  • 10 new deny cases and 9 new benign cases added to tests/data/policy/{deny,benign}/cloud.jsonl.
  • eval/e17_policy_corpus.py: 788 of 788 harmful calls stopped (100.0% recall), 1 false alarm in 852 (0.12%), 0 engine mismatches between RE2 and regex fallback.
  • tests/test_eval_docs.py and tests/test_docs.py pass.

How I ran it as a user

Ran Engine on server.yaml against both benign and held calls:

allow -        aws guardduty list-detectors
ask   TK-C010  aws guardduty update-detector --detector-id d-123 --no-enable
ask   TK-C010  aws guardduty update-detector --detector-id d-123 --features Name=S3_DATA_EVENTS,Status=DISABLED
ask   TK-C010  aws guardduty create-filter --detector-id d-123 --name filter1 --action ARCHIVE
allow -        aws cloudtrail update-trail --name prod-trail --enable-log-file-validation
ask   TK-C010  aws cloudtrail update-trail --name prod-trail --s3-bucket-name other-bucket
ask   TK-C010  aws cloudtrail update-trail --name prod-trail --no-is-multi-region-trail
allow -        aws configservice describe-delivery-channels
ask   TK-C010  aws configservice delete-delivery-channel --delivery-channel-name default
allow -        aws securityhub get-enabled-standards
ask   TK-C010  aws securityhub batch-disable-standards --standards-subscription-arns arn:aws:...
allow -        aws route53resolver list-resolver-query-log-configs
ask   TK-C010  aws route53resolver delete-resolver-query-log-config --resolver-query-log-config-id rqlc-123
allow -        aws organizations describe-organization
ask   TK-C010  aws organizations leave-organization

tracekit policy lint is clean for server-cloud.yaml, server.yaml and strict.yaml.

Tests run on Windows 11, Python 3.13:

  • pytest tests/test_signer_policy.py tests/test_eval_docs.py tests/test_docs.py: 149 passed.
  • pyflakes tracekit tracekit_sdk.py tests eval: clean (exit 0).

Checklist

  • make check passes (lint, tests, build); make test-ts too if sdk/typescript/ changed
  • a regression test covers the change
  • user-visible changes are described above (release notes are written at release time; don't edit CHANGELOG.md)
  • docs updated if behaviour or a security claim changed (docs/limits.md, README.md, docs/evaluation.md)
  • every commit is signed off (git commit -s, see CONTRIBUTING.md)

Security invariants

  • the agent still never holds a signing key or assigns sequence numbers
  • nothing an agent-controlled process supplies (isolation level, fail mode, gap/tamper records, approval identity) is newly trusted by the signer or verifier
  • gaps are still signed, never silent; a check that cannot run warns instead of passing
  • no verification code is shipped inside a bundle

Evidence format

  • this change does not touch the frozen formats: v1 (core.sig_message, ledger.make_record, the v1 schema, how the verifier treats existing bundles) or the published v2 format

Signed-off-by: shrinivas-sn <shrinusn2001@gmail.com>
@LoopGlitch26

Copy link
Copy Markdown
Member

Thanks for these. Six policy PRs is more than we can review well at once, so we'll take them one at a time, in the order you opened them, starting with #543. They all edit the same pack, test corpus and E17 results, so after each one merges, the next will need main merged in and E17 re-recorded. #544 also needs a new rule ID: TK-A049 is taken by #513, which was opened first.

Please don't claim more issues until these are in; the other good first issues are there for other first-time contributors. From now on, comment on an issue and wait to be assigned before you start.

@LoopGlitch26

Copy link
Copy Markdown
Member

Review of this one. The look-alikes are good (--enable, Status=ENABLED, NOOP, --s3-key-prefix). To merge, please:

  1. Merge main. 1.5.3 review: TK-M003 spellings, tool-name spellings two packs share, MCP bare names, F016/F017 case, A048 options #542 changed the E17 results, so the branch conflicts.

  2. Hold the other spellings of the same actions, with tests:

    • aws guardduty update-filter … --action ARCHIVE, which does the same as create-filter;
    • --action=ARCHIVE;
    • the JSON form update-detector --features '[{"Name":"S3_DATA_EVENTS","Status":"DISABLED"}]'. The shell rule sees the words without their quotes, so Status=DISABLED never appears in it.

    For example (create|update)-filter (.*? )?--action[ =]ARCHIVE, plus a "Status":"DISABLED" alternative. The (\S* )? after (.*? )? adds nothing and can go.

  3. Re-record E21 as well as E17. The new deny cases change eval/results/e21_robustness.json and e21_policy_robustness.json, and CI's offline evals job compares both (the same e21_* lines in ci.yml). Then update the README and docs/evaluation.md numbers that tests/test_eval_docs.py checks. Policy: turning off AWS audit logging or detection waits for a person #532 shows the full set.

  4. Docs.

    • Add the new commands to TK-C010's line in docs/policy-v2.md, as Policy: turning off AWS audit logging or detection waits for a person #532 did.
    • In the limits row, name what stays unheld: JSON or file:// features, --no-enable-log-file-validation, --no-include-global-service-events and route53resolver disassociate-resolver-query-log-config. Or hold the two trail options instead.
    • Write the row in words, not regex ((master|administrator)).
  5. Run make check before ticking it. The description lists three test files, run on Windows. Say in the PR what you ran.

When CI is green and a second review passes, I'll merge it.

Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
…ing about these packs

Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
@LoopGlitch26

Copy link
Copy Markdown
Member

Thanks for this. I pushed two commits to this branch so it can land here, as your pull request: a merge of main with the review's changes, and one that keeps E21's recorded latency rows.

  • Main merged in. The branch was behind main and conflicted in the pack, docs/limits.md and the E17 results.
  • The review's changes, from TK-C010: the other common ways to switch off AWS detection wait for a person #581. TK-C010 also holds update-filter … --action ARCHIVE, --action=ARCHIVE, a feature's Status set to DISABLED in JSON, the other update-trail --no- options and route53resolver disassociate-resolver-query-log-config, in a shell and through call_aws. Each option is read within sixteen words of its command, so the fallback engine stays linear. Your cases and look-alikes are kept, with the review's spellings beside them and a speed test in tests/test_policy2.py.
  • Docs. TK-C010's line in docs/policy-v2.md lists the new commands, and the limits row says in words what stays unheld (file:// or --cli-input-json input, --data-sources, create-ip-set, an option past sixteen words).
  • Pack and results. server-cloud is 2026.10-server-cloud-17. E17 and E21 were re-run on the merged tree: 806 of 806 harmful calls held, 1 false alarm in 866, all 2,297 shell rewrites contained. The README and docs/evaluation.md quote the new numbers.

@LoopGlitch26
LoopGlitch26 merged commit cbef793 into Cygnux-Labs:main Nov 5, 2026
59 of 60 checks passed
LoopGlitch26 added a commit that referenced this pull request Nov 5, 2026
…-run, E21's recorded latency rows kept

Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
LoopGlitch26 added a commit that referenced this pull request Nov 6, 2026
Brings in #582 (TK-A052, TK-A053; coding pack 2026.10-v2-63), #543 (TK-C010),
#612, #595, #621 and the commits before them.

- coding.yaml: version 2026.11-v2-64, the next free one. server-17,
  strict-10, server-core-1 and dev-1 are numbers main hasn't used.
- docs/policy-v2.md: merged word by word, main's TK-A052 and TK-C010 text
  with this branch's.
- docs/evaluation.md: main's "refreshed on macOS" note with this branch's
  dev-row note.
- eval results: main's files, then E17, E21 rewrites (no --rebaseline) and
  E21 policy robustness re-run. The recorded latency rows are main's, and
  this branch's dev row (5.44 / 6.28) as before.
@LoopGlitch26

Copy link
Copy Markdown
Member

Thanks again for this. If you use a coding agent (Claude Code, Codex, Cursor, Gemini CLI or OpenCode), would you run Tracekit on it for a week?

pip install tracekit-ai
tracekit audit --since 7d      # read-only: what it would have held in your last week, from Claude Code's history
tracekit init --dev --v2       # then record and guard from now on

After a week, tracekit report --share prints counts only (no commands, paths or content). Paste it into #385, with anything that got in your way. Those reports decide what we fix next.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TK-C010: hold the other common ways to switch off AWS detection

2 participants