feat(audit): Implement standalone audit logging for SSL-only mode#6304
Conversation
PR Code Analyzer ❗AI-powered 'Code-Diff-Analyzer' found issues on commit f29b3aa.
The table above displays the top 10 most important findings. Pull Requests Author(s): Please update your Pull Request according to the report above. Repository Maintainer(s): You can Thanks. |
PR Reviewer Guide 🔍(Review updated until commit f29b3aa)Here are some key observations to aid the review process:
|
PR Code Suggestions ✨Latest suggestions up to f29b3aa Explore these optional code suggestions:
Previous suggestionsSuggestions up to commit f2125ed
Suggestions up to commit 3d2d12d
Suggestions up to commit 5216482
Suggestions up to commit c46a1e2
|
eb69b3e to
1d57f01
Compare
|
Persistent review updated to latest commit 1d57f01 |
c6443a2 to
7b7804d
Compare
|
Persistent review updated to latest commit 7b7804d |
7b7804d to
8f10b41
Compare
|
Persistent review updated to latest commit 8f10b41 |
Enable audit logging as an independent capability in SSL-only mode (no auth/RBAC required). Users running OpenSearch without full security can now have a complete audit trail for compliance (SOC2, HIPAA, PCI-DSS). Key features: - New REQUEST_AUDIT category for all REST/transport actions - Client certificate identity enrichment (mTLS CN/SAN as effective_user) - Request body logging with sensitive header exclusion - Index resolution (wildcards expanded to actual indices) - Bulk per-item event tracking (configurable granularity) - Ignore-users and ignore-requests filtering - Transport-layer interception (shard-level operations) - Document-level compliance tracking (COMPLIANCE_DOC_WRITE and COMPLIANCE_DOC_READ) via IndexingOperationListener and lightweight ComplianceReadIndexSearcherWrapper - All sinks supported (internal_opensearch, log4j, webhook, Kafka) - 17 dynamic settings configurable at runtime via PUT _cluster/settings without node restart New production files: - AuditActionFilter: lightweight ActionFilter for non-FGAC modes - AuditTransportInterceptor: captures inter-node transport traffic - ComplianceReadIndexSearcherWrapper: field-level read tracking - SecuritySettings: dynamic Setting constants for cluster settings New test files: - AuditActionFilterTest: 11 unit tests - AuditTransportInterceptorTest: 11 unit tests - StandaloneAuditLoggingTest: 21 integration tests (basic events) - StandaloneAuditFilterFeaturesTest: 25 integration tests - StandaloneAuditComplianceDocTest: 14 integration tests - StandaloneAuditDynamicFilterSettingsTest: 10 integration tests - StandaloneAuditDynamicComplianceSettingsTest: 7 integration tests - StandaloneAuditDynamicConfigTest: 2 integration tests - StandaloneAuditTransportInterceptTest: 2 integration tests - StandaloneAuditMtlsTest: 1 integration test - StandaloneAuditDisabledCategoryTest: 1 integration test - StandaloneAuditSinksTest: 3 integration tests - StandaloneAuditWebhookSinkTest: 1 integration test Total tests: 128 (43 unit + 85 integration) Signed-off-by: Muzzamil Jolaade <muzzajol@amazon.com>
8f10b41 to
8ee2707
Compare
|
Persistent review updated to latest commit 8ee2707 |
|
@Taiwo435 please look into the test failures. Also address the code hygiene failure by running |
Signed-off-by: Muzzamil Jolaade <muzzajol@amazon.com>
|
Persistent review updated to latest commit 104720d |
…e_log_diffs Pass indexService to ciol via setIs() inside the setReaderWrapper lambda so that write_log_diffs can retrieve original documents when logging index modifications in ssl-only audit mode. Signed-off-by: Muzzamil Jolaade <muzzajol@amazon.com>
|
Persistent review updated to latest commit 5011ebd |
- Self-loop guard: read audit index prefix from settings instead of hardcoded 'security-auditlog'. Extracts literal text from Joda date pattern or falls back to datastream name. - IndexNameExpressionResolver: inject from plugin instead of creating per-filter instance in AuditActionFilter. - Transport interceptor: accept live AuditConfig.Filter instead of frozen Settings. Dynamic updates via PUT _cluster/settings now take effect immediately on transport events. - logRequestAudit: also check disabledRestCategories to properly suppress REQUEST_AUDIT when disabled via rest-specific setting. - BulkRequest handling: iterate BulkRequest.requests() for REST-level bulk in addition to BulkShardRequest.items() for shard-level. Tests added: - Unit tests for getAuditIndexPrefix edge cases (Joda, plain, datastream) - Integration tests for transport interceptor dynamic settings - Integration test for disabled_rest_categories suppression - Integration tests for mixed bulk operations with correct request types - Integration test for multi-index bulk per-item events - Integration test verifying disabled_rest_categories does not suppress TRANSPORT_AUDIT - write_log_diffs regression tests (13 tests) for ssl-only mode Signed-off-by: Muzzamil Jolaade <muzzajol@amazon.com>
|
Persistent review updated to latest commit f55e52d |
Signed-off-by: Muzzamil Jolaade <muzzajol@amazon.com>
|
Persistent review updated to latest commit d077550 |
DarshitChanpura
left a comment
There was a problem hiding this comment.
Thanks @Taiwo435. This is a solid first PR towards getting Audit Logging to work with SSL-only mode.
I've left a few comments around potential bugs, perf improvement and nits.
|
Persistent review updated to latest commit 38a12e2 |
38a12e2 to
939edc4
Compare
|
Persistent review updated to latest commit 939edc4 |
…ntainability - Self-loop guard: extract isAuditIndex() helper, fail-open on null/empty prefix with WARN log at construction (prevents silent audit suppression) - Bulk path: short-circuit when REQUEST_AUDIT disabled (avoids per-item allocation), add try-catch for consistency, extract logBulkItem() helper to DRY the BulkShardRequest/BulkRequest loops - Header filtering: extract AuditHeaderUtils shared utility, unify behavior between AuditActionFilter and AuditTransportInterceptor (both now respect exclude_sensitive_headers setting via WildcardMatcher) - Transport interceptor: add hardcoded internal:* skip to prevent AssertionError during cluster state application, catch AssertionError for defense-in-depth - Filter order: named constant AUDIT_FILTER_ORDER = -200 - ComplianceConfig: WatchedReadState atomic holder for concurrency - SSL cert extraction: rate-limited WARN logging (was silent catch) - SecuritySettings: fix COMPLIANCE_READ/WRITE_IGNORE_USERS defaults - Tests: 10 new unit tests for self-loop guard + constructor WARN logging Signed-off-by: Muzzamil Jolaade <muzzajol@amazon.com>
939edc4 to
c46a1e2
Compare
|
Persistent review updated to latest commit c46a1e2 |
…cking ComplianceLeafReader was extending FilterLeafReader, which caused search operations requiring a CodecReader or SequentialStoredFieldsLeafReader to fail with IOException. Changed to extend SequentialStoredFieldsLeafReader and implemented doGetSequentialStoredFieldsReader() with a ComplianceStoredFieldsReader wrapper, matching the pattern used by DlsFlsFilterLeafReader in FGAC mode. Signed-off-by: Muzzamil Jolaade <muzzajol@amazon.com>
Add volatile to ignoredUrlParams/ignoredUrlParamsMatcher fields that were missing visibility guarantees across threads. Reorder writes in setIgnoredAuditUsers() and setIgnoredAuditRequests() to build set and matcher in local variables first, then publish the operational matcher last to ensure readers always see a self-consistent state. Signed-off-by: Muzzamil Jolaade <muzzajol@amazon.com>
|
Persistent review updated to latest commit 5216482 |
5216482 to
3d2d12d
Compare
|
Persistent review updated to latest commit 3d2d12d |
DarshitChanpura
left a comment
There was a problem hiding this comment.
Left a couple more comments.
- Add REQUEST_AUDIT disabled category short-circuit before both bulk and non-bulk paths in AuditActionFilter, avoiding wasted allocation and index-resolution when the category is off. Consistent with AuditTransportInterceptor behavior. - Add BulkRequest index collection for the non-bulk-resolved path (BulkRequest doesn't implement IndicesRequest). - Add per-item self-loop guard in logBulkItem() to skip audit-index writes within bulk operations. - Rewrite shouldIgnoreUsersAddedAtRuntime integration test to use mTLS client with known DN, verifying actual runtime suppression with assertExactlyScanAll(0, ...) instead of just statusCode(200). Signed-off-by: Muzzamil Jolaade <muzzajol@amazon.com>
|
Persistent review updated to latest commit f2125ed |
Add getDisabledRestCategories() check alongside getDisabledCategories() so REQUEST_AUDIT is short-circuited when disabled via either the unified or REST-specific setting. Consistent with AbstractAuditLog.checkRestFilter behavior in FGAC mode. Signed-off-by: Muzzamil Jolaade <muzzajol@amazon.com>
|
Persistent review updated to latest commit f29b3aa |
| Setting.Property.Sensitive | ||
| ); | ||
|
|
||
| public static final Setting<Boolean> AUDIT_ENABLED_SETTING = Setting.boolSetting( |
There was a problem hiding this comment.
@DarshitChanpura @Taiwo435 Please add Setting.Property.Sensitive in the list of settings properties for all of these. When Setting.Property.Sensitive is present, the setting is only toggle-able by a user mapped to one of the roles from plugins.security.restapi.roles_enabled: ["all_access","xyz_role"] (colloquially referred to as a "security admin"). Its crude, but does at least move towards some semblance of fine grained controls for cluster settings. All of the settings in here should only be toggle-able by a security admin of the cluster.
Even in SSL-Only mode we theoretically could enforce that a cluster admin is the user that presents the admin certificate. So even when in SSL-Only mode (i.e. no users and roles) we can still differentiate between a trusted and non-trusted caller by presence of that certificate.
| String principal = threadPool.getThreadContext().getTransient(ConfigConstants.OPENDISTRO_SECURITY_SSL_PRINCIPAL); | ||
| User user = threadPool.getThreadContext().getTransient(ConfigConstants.OPENDISTRO_SECURITY_USER); | ||
| String effectiveUser = user != null ? user.getName() : principal; | ||
| if (effectiveUser == null || !filter.isAuditDisabled(effectiveUser)) { |
There was a problem hiding this comment.
IMO !filter.isAuditDisabled doesn't read naturally. Prefer filter.isAuditEnabled()
| msg.addRequestType(request.getClass().getSimpleName()); | ||
|
|
||
| // Target indices | ||
| if (request instanceof IndicesRequest) { |
There was a problem hiding this comment.
@DarshitChanpura @Taiwo435 Can we use the IndexResolverReplacer here? Can we please make sure to test with both legacy and v4 privilege evaluation code paths.
| * Reuses FieldReadCallback directly with FieldMaskingRule.ALLOW_ALL | ||
| * (no field masking since no FLS is active). | ||
| */ | ||
| public class ComplianceReadIndexSearcherWrapper implements CheckedFunction<DirectoryReader, DirectoryReader, IOException> { |
There was a problem hiding this comment.
Is there any opportunity to make this more generic in case the same code can be re-used?
Enable audit logging as an independent capability in SSL-only mode (no auth/RBAC required). Users running OpenSearch without full security can now have a complete audit trail for compliance (SOC2, HIPAA, PCI-DSS).
Key features:
New production files:
New test files:
Total tests: 128 (43 unit + 85 integration)
Description
Category: New feature
Why these changes are required?
Users running OpenSearch in SSL-only mode (TLS without authentication/authorization) currently have zero audit trail. Compliance frameworks (SOC2, HIPAA, PCI-DSS, GDPR) require audit logs even without access
control. This change makes audit logging an independent capability that works without FGAC.
What is the old behavior before changes and new behavior after changes?
plugins.security.ssl_only: true), configuring an audit sink (plugins.security.audit.type: log4j) now enables full audit logging — capturing all REST/transport actions, requestbodies, client cert identity, compliance doc read/write tracking, with 17 dynamically configurable settings.
Issues Resolved
Resolves #6303
Testing
toggle, transport intercept
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.