CIS Benchmarks: Update macOS benchmarks - #49815
Conversation
4cd19b5 to
1dba221
Compare
1dba221 to
075075a
Compare
075075a to
52f3111
Compare
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (11)
🚧 Files skipped from review as they are similar to previous changes (10)
WalkthroughUpdates macOS CIS benchmark references to Sonoma v3.1.0, Sequoia v2.1.0, and Tahoe v1.1.0. Policy checks and remediation scripts now cover current-console-user hot corners, 15-minute screen saver inactivity, 30-day audit retention, Library exclusions, APFS encryption scope, and root secure-token removal. New pass/fail fixtures support the revised controls. SSH test probes now retry after timeouts. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Warning Review ran into problems🔥 ProblemsGit: Failed to clone repository. Please run the Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 8
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@ee/cis/macos-14/cis-policy-queries.yml`:
- Around line 2771-2772: Update the path exclusion condition in the CIS policy
query to match only the exact /Library/AppStore directory and descendants under
/Library/AppStore/. Replace the broad prefix pattern while preserving the
existing exclusion behavior for the intended directory and its children.
- Around line 2369-2374: The audit-control query must validate all active
day-based expire-after directives, not just accept any compliant line. Update
the EXISTS logic around the expire-after regex so it requires a directive of at
least 30 days while rejecting configurations containing any active day-based
directive below 30 days; preserve handling of non-day-based directives unless
the surrounding policy specifies otherwise.
In `@ee/cis/macos-14/test/scripts/CIS_2.7.1_fail.sh`:
- Around line 5-8: Update the console-user handling in CIS_2.7.1_fail.sh to exit
nonzero when stat returns an empty value or root, instead of silently
succeeding. Only invoke sudo/defaults for a valid non-root user, preserving the
existing failing-state setup in that case.
In `@ee/cis/macos-14/test/scripts/CIS_5.1.7_pass.sh`:
- Around line 12-13: Limit pass-fixture cleanup to the flagged test directory
instead of recursively scanning and modifying all of /Library. In
ee/cis/macos-14/test/scripts/CIS_5.1.7_pass.sh lines 12-13 and
ee/cis/macos-15/test/scripts/CIS_5.1.7_pass.sh lines 12-13, update the cleanup
command to remove /Library/CIS_Test_World_Writable or directly clear that
directory’s world-write bit without affecting unrelated paths.
In `@ee/cis/macos-14/test/scripts/not_always_working_CIS_2.10.1.sh`:
- Line 4: Replace the literal <username> argument in the screensaver defaults
command with the same console-user resolution used by the other updated
fixtures, then pass that resolved username to sudo while preserving the existing
idleTime setting.
In `@ee/cis/macos-15/cis-policy-queries.yml`:
- Around line 2247-2252: Update the retention-duration regex in the macOS 15
query at ee/cis/macos-15/cis-policy-queries.yml lines 2247-2252 to capture only
the day value immediately following expire-after:, then add a failing fixture
for expire-after:7d OR 30d. Apply the same anchored parsing rule and
mixed-duration failing fixture at ee/cis/macos-26/cis-policy-queries.yml lines
1982-1987.
In `@ee/cis/macos-26/cis-policy-queries.yml`:
- Around line 2367-2369: De-scope the internal-volume control in the APFS policy
queries because apfs_volumes cannot distinguish internal from external media. In
ee/cis/macos-26/cis-policy-queries.yml lines 2367-2369, replace the overbroad
query with a locality-aware data source or omit the control; apply the same
correction in ee/cis/macos-15/cis-policy-queries.yml lines 2784-2786 rather than
merely documenting the limitation.
In `@tools/cis/cis-test-runner.py`:
- Around line 913-922: Update wait_for_ssh so each retry calculates the
remaining deadline before calling ssh, uses the smaller of that value and the
existing 10-second timeout, and stops when no time remains. Also cap the retry
sleep to the remaining deadline so the function does not exceed the requested
timeout.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: d753a60e-3688-4d9b-b01f-ac0243c88284
⛔ Files ignored due to path filters (4)
ee/cis/CIS-BENCHMARKS.mdis excluded by!**/*.mdee/cis/macos-14/README.mdis excluded by!**/*.mdee/cis/macos-15/README.mdis excluded by!**/*.mdee/cis/macos-26/README.mdis excluded by!**/*.md
📒 Files selected for processing (27)
changes/45644-update-macos-cis-benchmarksee/cis/macos-14/cis-policy-queries.ymlee/cis/macos-14/test/profiles/not_always_working_2.10.1.mobileconfigee/cis/macos-14/test/scripts/CIS_2.7.1.shee/cis/macos-14/test/scripts/CIS_2.7.1_fail.shee/cis/macos-14/test/scripts/CIS_2.7.1_pass.shee/cis/macos-14/test/scripts/CIS_3.4_fail.shee/cis/macos-14/test/scripts/CIS_3.4_pass.shee/cis/macos-14/test/scripts/CIS_5.1.7_fail.shee/cis/macos-14/test/scripts/CIS_5.1.7_pass.shee/cis/macos-14/test/scripts/CIS_5.6.shee/cis/macos-14/test/scripts/not_always_working_CIS_2.10.1.shee/cis/macos-15/cis-policy-queries.ymlee/cis/macos-15/test/scripts/CIS_2.7.1_fail.shee/cis/macos-15/test/scripts/CIS_2.7.1_pass.shee/cis/macos-15/test/scripts/CIS_3.4_fail.shee/cis/macos-15/test/scripts/CIS_3.4_pass.shee/cis/macos-15/test/scripts/CIS_5.1.7_fail.shee/cis/macos-15/test/scripts/CIS_5.1.7_pass.shee/cis/macos-15/test/scripts/CIS_5.6.shee/cis/macos-26/cis-policy-queries.ymlee/cis/macos-26/test/scripts/CIS_2.7.1_pass.shee/cis/macos-26/test/scripts/CIS_3.4_fail.shee/cis/macos-26/test/scripts/CIS_3.4_pass.shee/cis/macos-26/test/scripts/CIS_5.6.shee/cis/macos-26/test/scripts/CIS_5.6_pass.shtools/cis/cis-test-runner.py
💤 Files with no reviewable changes (2)
- ee/cis/macos-14/test/scripts/CIS_2.7.1.sh
- ee/cis/macos-26/test/scripts/CIS_5.6_pass.sh
There was a problem hiding this comment.
Caution
Inline review comments failed to post. This is likely due to GitHub's internal server error or limits when posting large numbers of comments. If you are seeing this consistently it is likely a permissions issue. Please check "Moderation" -> "Code review limits" under your organization settings.
Actionable comments posted: 8
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@ee/cis/macos-14/cis-policy-queries.yml`:
- Around line 2771-2772: Update the path exclusion condition in the CIS policy
query to match only the exact /Library/AppStore directory and descendants under
/Library/AppStore/. Replace the broad prefix pattern while preserving the
existing exclusion behavior for the intended directory and its children.
- Around line 2369-2374: The audit-control query must validate all active
day-based expire-after directives, not just accept any compliant line. Update
the EXISTS logic around the expire-after regex so it requires a directive of at
least 30 days while rejecting configurations containing any active day-based
directive below 30 days; preserve handling of non-day-based directives unless
the surrounding policy specifies otherwise.
In `@ee/cis/macos-14/test/scripts/CIS_2.7.1_fail.sh`:
- Around line 5-8: Update the console-user handling in CIS_2.7.1_fail.sh to exit
nonzero when stat returns an empty value or root, instead of silently
succeeding. Only invoke sudo/defaults for a valid non-root user, preserving the
existing failing-state setup in that case.
In `@ee/cis/macos-14/test/scripts/CIS_5.1.7_pass.sh`:
- Around line 12-13: Limit pass-fixture cleanup to the flagged test directory
instead of recursively scanning and modifying all of /Library. In
ee/cis/macos-14/test/scripts/CIS_5.1.7_pass.sh lines 12-13 and
ee/cis/macos-15/test/scripts/CIS_5.1.7_pass.sh lines 12-13, update the cleanup
command to remove /Library/CIS_Test_World_Writable or directly clear that
directory’s world-write bit without affecting unrelated paths.
In `@ee/cis/macos-14/test/scripts/not_always_working_CIS_2.10.1.sh`:
- Line 4: Replace the literal <username> argument in the screensaver defaults
command with the same console-user resolution used by the other updated
fixtures, then pass that resolved username to sudo while preserving the existing
idleTime setting.
In `@ee/cis/macos-15/cis-policy-queries.yml`:
- Around line 2247-2252: Update the retention-duration regex in the macOS 15
query at ee/cis/macos-15/cis-policy-queries.yml lines 2247-2252 to capture only
the day value immediately following expire-after:, then add a failing fixture
for expire-after:7d OR 30d. Apply the same anchored parsing rule and
mixed-duration failing fixture at ee/cis/macos-26/cis-policy-queries.yml lines
1982-1987.
In `@ee/cis/macos-26/cis-policy-queries.yml`:
- Around line 2367-2369: De-scope the internal-volume control in the APFS policy
queries because apfs_volumes cannot distinguish internal from external media. In
ee/cis/macos-26/cis-policy-queries.yml lines 2367-2369, replace the overbroad
query with a locality-aware data source or omit the control; apply the same
correction in ee/cis/macos-15/cis-policy-queries.yml lines 2784-2786 rather than
merely documenting the limitation.
In `@tools/cis/cis-test-runner.py`:
- Around line 913-922: Update wait_for_ssh so each retry calculates the
remaining deadline before calling ssh, uses the smaller of that value and the
existing 10-second timeout, and stops when no time remains. Also cap the retry
sleep to the remaining deadline so the function does not exceed the requested
timeout.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: d753a60e-3688-4d9b-b01f-ac0243c88284
⛔ Files ignored due to path filters (4)
ee/cis/CIS-BENCHMARKS.mdis excluded by!**/*.mdee/cis/macos-14/README.mdis excluded by!**/*.mdee/cis/macos-15/README.mdis excluded by!**/*.mdee/cis/macos-26/README.mdis excluded by!**/*.md
📒 Files selected for processing (27)
changes/45644-update-macos-cis-benchmarksee/cis/macos-14/cis-policy-queries.ymlee/cis/macos-14/test/profiles/not_always_working_2.10.1.mobileconfigee/cis/macos-14/test/scripts/CIS_2.7.1.shee/cis/macos-14/test/scripts/CIS_2.7.1_fail.shee/cis/macos-14/test/scripts/CIS_2.7.1_pass.shee/cis/macos-14/test/scripts/CIS_3.4_fail.shee/cis/macos-14/test/scripts/CIS_3.4_pass.shee/cis/macos-14/test/scripts/CIS_5.1.7_fail.shee/cis/macos-14/test/scripts/CIS_5.1.7_pass.shee/cis/macos-14/test/scripts/CIS_5.6.shee/cis/macos-14/test/scripts/not_always_working_CIS_2.10.1.shee/cis/macos-15/cis-policy-queries.ymlee/cis/macos-15/test/scripts/CIS_2.7.1_fail.shee/cis/macos-15/test/scripts/CIS_2.7.1_pass.shee/cis/macos-15/test/scripts/CIS_3.4_fail.shee/cis/macos-15/test/scripts/CIS_3.4_pass.shee/cis/macos-15/test/scripts/CIS_5.1.7_fail.shee/cis/macos-15/test/scripts/CIS_5.1.7_pass.shee/cis/macos-15/test/scripts/CIS_5.6.shee/cis/macos-26/cis-policy-queries.ymlee/cis/macos-26/test/scripts/CIS_2.7.1_pass.shee/cis/macos-26/test/scripts/CIS_3.4_fail.shee/cis/macos-26/test/scripts/CIS_3.4_pass.shee/cis/macos-26/test/scripts/CIS_5.6.shee/cis/macos-26/test/scripts/CIS_5.6_pass.shtools/cis/cis-test-runner.py
💤 Files with no reviewable changes (2)
- ee/cis/macos-14/test/scripts/CIS_2.7.1.sh
- ee/cis/macos-26/test/scripts/CIS_5.6_pass.sh
🛑 Comments failed to post (1)
ee/cis/macos-14/test/scripts/CIS_5.1.7_pass.sh (1)
12-13: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Limit pass-fixture cleanup to the test directory. These scripts recursively modify every matching
/Librarytree, including files beneath the flagged directory, although the paired fail fixture creates only/Library/CIS_Test_World_Writable. This can alter unrelated system or application permissions during CIS tests.
ee/cis/macos-14/test/scripts/CIS_5.1.7_pass.sh#L12-L13: remove the test directory or clear its world-write bit directly instead of recursively modifying all matching paths.ee/cis/macos-15/test/scripts/CIS_5.1.7_pass.sh#L12-L13: apply the same narrowly scoped cleanup.📍 Affects 2 files
ee/cis/macos-14/test/scripts/CIS_5.1.7_pass.sh#L12-L13(this comment)ee/cis/macos-15/test/scripts/CIS_5.1.7_pass.sh#L12-L13🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@ee/cis/macos-14/test/scripts/CIS_5.1.7_pass.sh` around lines 12 - 13, Limit pass-fixture cleanup to the flagged test directory instead of recursively scanning and modifying all of /Library. In ee/cis/macos-14/test/scripts/CIS_5.1.7_pass.sh lines 12-13 and ee/cis/macos-15/test/scripts/CIS_5.1.7_pass.sh lines 12-13, update the cleanup command to remove /Library/CIS_Test_World_Writable or directly clear that directory’s world-write bit without affecting unrelated paths.
Resolves #45644 Bring the macOS CIS benchmark policies up to the current CIS releases: - macOS 14 Sonoma: v3.0.0 -> v3.1.0 - macOS 15 Sequoia: v2.0.0 -> v2.1.0 - macOS 26 Tahoe: v1.0.0 -> v1.1.0 Policy changes: - 2.7.1: rescope the screen saver hot corners check to the current console user and move it to Level 1, per the updated CIS audit. - 3.4: relax audit log retention to >= 30 days (no size requirement); parse the day value anchored to `expire-after:` so a mixed directive such as `7d OR 30d` is correctly rejected. - 5.1.7: exclude the non-accessible /Library/AppStore directory and its descendants from the world-writable Library check. - 5.6: update for the new secure-token audit (verify root has no AuthenticationAuthority). - 5.3.1: add the internal APFS volume encryption check (Manual -> Automated) on macOS 26; remove the deprecated CoreStorage 5.3.2 check on macOS 14/15. - 2.10.1: correct the screen saver inactivity threshold to <= 900s (15 minutes) to match the CIS audit. Limitations documented (no reliable fleetd data source): - 5.3.2 (external APFS/HFS+ encryption): apfs_volumes exposes no internal/external indicator. - 5.3.3 (FAT32/ExFAT): CIS Manual audit.
52f3111 to
c216816
Compare
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@ee/cis/macos-14/test/scripts/CIS_3.4_pass.sh`:
- Around line 18-20: Make the fixture setup fail fast by adding failure checks
to the mv, chown, and chmod commands in
ee/cis/macos-14/test/scripts/CIS_3.4_pass.sh lines 18-20,
ee/cis/macos-14/test/scripts/CIS_3.4_fail.sh lines 18-20,
ee/cis/macos-15/test/scripts/CIS_3.4_fail.sh lines 18-20, and
ee/cis/macos-15/test/scripts/CIS_3.4_pass.sh lines 18-20, ensuring any failed
replacement or metadata update causes the script to exit nonzero.
In `@ee/cis/macos-15/cis-policy-queries.yml`:
- Around line 1820-1845: Add cis_id "2.11.1" to the macOS 15 inactivity-interval
policy identified by its name, and add the corresponding MDM profile/script test
pair under the macOS 15 test fixtures. Ensure the test pair validates the
existing idleTime <= 900 requirement and is discoverable by cis-test-runner.py.
In `@ee/cis/macos-15/test/scripts/CIS_2.7.1_fail.sh`:
- Around line 12-13: Ensure the Dock preference write in CIS_2.7.1_fail.sh
propagates failure instead of being masked by the subsequent non-fatal cfprefsd
killall command. Apply this change at
ee/cis/macos-15/test/scripts/CIS_2.7.1_fail.sh lines 12-13 and
ee/cis/macos-26/test/scripts/CIS_2.7.1_fail.sh lines 12-13, preserving fail-fast
behavior when writing wvous-br-corner fails.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: f37b7ec9-5db1-4b98-9e7a-0d350d7c4174
⛔ Files ignored due to path filters (5)
ee/cis/CIS-BENCHMARKS.mdis excluded by!**/*.mdee/cis/macos-14/README.mdis excluded by!**/*.mdee/cis/macos-15/README.mdis excluded by!**/*.mdee/cis/macos-26/README.mdis excluded by!**/*.mdtools/cis/__pycache__/cis-test-runner.cpython-314.pycis excluded by!**/*.pyc
📒 Files selected for processing (31)
changes/45644-update-macos-cis-benchmarksee/cis/macos-14/cis-policy-queries.ymlee/cis/macos-14/test/profiles/not_always_working_2.10.1.mobileconfigee/cis/macos-14/test/scripts/CIS_2.7.1.shee/cis/macos-14/test/scripts/CIS_2.7.1_fail.shee/cis/macos-14/test/scripts/CIS_2.7.1_pass.shee/cis/macos-14/test/scripts/CIS_3.4.shee/cis/macos-14/test/scripts/CIS_3.4_fail.shee/cis/macos-14/test/scripts/CIS_3.4_pass.shee/cis/macos-14/test/scripts/CIS_5.1.7.shee/cis/macos-14/test/scripts/CIS_5.1.7_fail.shee/cis/macos-14/test/scripts/CIS_5.1.7_pass.shee/cis/macos-14/test/scripts/CIS_5.6.shee/cis/macos-14/test/scripts/not_always_working_CIS_2.10.1.shee/cis/macos-15/cis-policy-queries.ymlee/cis/macos-15/test/scripts/CIS_2.7.1_fail.shee/cis/macos-15/test/scripts/CIS_2.7.1_pass.shee/cis/macos-15/test/scripts/CIS_3.4_fail.shee/cis/macos-15/test/scripts/CIS_3.4_pass.shee/cis/macos-15/test/scripts/CIS_5.1.7_fail.shee/cis/macos-15/test/scripts/CIS_5.1.7_pass.shee/cis/macos-15/test/scripts/CIS_5.6.shee/cis/macos-26/cis-policy-queries.ymlee/cis/macos-26/test/scripts/CIS_2.7.1_fail.shee/cis/macos-26/test/scripts/CIS_2.7.1_pass.shee/cis/macos-26/test/scripts/CIS_3.4_fail.shee/cis/macos-26/test/scripts/CIS_3.4_pass.shee/cis/macos-26/test/scripts/CIS_5.1.7_pass.shee/cis/macos-26/test/scripts/CIS_5.6.shee/cis/macos-26/test/scripts/CIS_5.6_pass.shtools/cis/cis-test-runner.py
💤 Files with no reviewable changes (4)
- ee/cis/macos-14/test/scripts/CIS_5.1.7.sh
- ee/cis/macos-14/test/scripts/CIS_3.4.sh
- ee/cis/macos-14/test/scripts/CIS_2.7.1.sh
- ee/cis/macos-26/test/scripts/CIS_5.6_pass.sh
🚧 Files skipped from review as they are similar to previous changes (17)
- changes/45644-update-macos-cis-benchmarks
- ee/cis/macos-14/test/scripts/CIS_5.1.7_fail.sh
- ee/cis/macos-26/test/scripts/CIS_3.4_fail.sh
- ee/cis/macos-15/test/scripts/CIS_5.6.sh
- ee/cis/macos-14/test/scripts/CIS_2.7.1_fail.sh
- ee/cis/macos-14/test/profiles/not_always_working_2.10.1.mobileconfig
- ee/cis/macos-15/test/scripts/CIS_2.7.1_pass.sh
- ee/cis/macos-14/test/scripts/CIS_2.7.1_pass.sh
- ee/cis/macos-26/test/scripts/CIS_5.6.sh
- ee/cis/macos-15/test/scripts/CIS_5.1.7_fail.sh
- ee/cis/macos-14/test/scripts/not_always_working_CIS_2.10.1.sh
- ee/cis/macos-26/test/scripts/CIS_2.7.1_pass.sh
- ee/cis/macos-14/test/scripts/CIS_5.6.sh
- ee/cis/macos-26/test/scripts/CIS_3.4_pass.sh
- tools/cis/cis-test-runner.py
- ee/cis/macos-26/cis-policy-queries.yml
- ee/cis/macos-14/cis-policy-queries.yml
Related issue: Resolves ##45644
Bring the macOS CIS benchmark policies up to the current CIS releases:
Policy changes:
console user and move it to Level 1, per the updated CIS audit.
parse the day value anchored to
expire-after:so a mixed directivesuch as
7d OR 30dis correctly rejected.descendants from the world-writable Library check.
AuthenticationAuthority).
Automated) on macOS 26; remove the deprecated CoreStorage 5.3.2 check
on macOS 14/15.
(15 minutes) to match the CIS audit.
Limitations documented (no reliable fleetd data source):
internal/external indicator.
Checklist for submitter
If some of the following don't apply, delete the relevant line.
changes/,orbit/changes/oree/fleetd-chrome/changes.See Changes files for more information.
Testing
Summary by CodeRabbit
New Features
Bug Fixes
expire-afterdays ≥ 30.Tests