Repository navigation
[MS Exchange Server] Added fields to ignore if empty. - #12417
Closed
matthewscherer wants to merge 0 commit into
Closed
matthewscherer wants to merge 0 commit into
matthewscherer wants to merge 0 commit into
Conversation
andrewkroh
added
bugfix
|
Pinging @elastic/sec-windows-platform (Team:Security-Windows Platform) |
Contributor
🚀 Benchmarks reportPackage
|
| Data stream | Previous EPS | New EPS | Diff (%) | Result |
|---|---|---|---|---|
messagetracking |
12195.12 | 10204.08 | -1991.04 (-16.33%) | 💔 |
To see the full report comment with /test benchmark fullreport
Contributor
|
changes lgtm but it requires manifest.yml version bump and the changelog.yml entry (eg https://github.com/elastic/integrations/pull/12381/files#diff-22d0d39b8e0908029d837bafe11c9ca489aee4b752de6546b3fc678ce7b370f1) |
Contributor
My recommended way to do this is to use |
andrewkroh
reviewed
Jan 31, 2025
Contributor
There was a problem hiding this comment.
This change needs an accompanying test case where a new log sample{s} with empty senderaddress/networkmessageid is added to a packages/microsoft_exchange_server/data_stream/messagetracking/_dev/test/pipeline/*.log file.
Here's a patch with those changes. The "expected" files were re-generated using elastic-package test pipeline -v -d messagetracking -g.
Details
diff --git a/packages/microsoft_exchange_server/data_stream/messagetracking/_dev/test/pipeline/test-messagetracking.log b/packages/microsoft_exchange_server/data_stream/messagetracking/_dev/test/pipeline/test-messagetracking.log
index 6a83935be5..8b2045d40c 100644
--- a/packages/microsoft_exchange_server/data_stream/messagetracking/_dev/test/pipeline/test-messagetracking.log
+++ b/packages/microsoft_exchange_server/data_stream/messagetracking/_dev/test/pipeline/test-messagetracking.log
@@ -2,3 +2,5 @@
2024-01-25T15:16:09.949Z,10.11.12.14,exchange-mail.my.domain.com,10.11.12.14,exchange-mail,08DC1DB12C345BE5;2024-01-25T15:16:09.544Z;0,exchange-mail\Default exchange-mail,SMTP,RECEIVE,70912345566403,<20240123200014.123F425E28@host01.my.domain.com>,1e6eb197-c6b4-1234-1b69-56dc1db88f50,mailuser@my.domain.com,,7229,1,,,vzdump backup status (host01.my.domain.com): backup successful,root@host01.my.domain.com,root@host01.my.domain.com,0cA: ,Incoming,,10.11.12.13,10.11.12.14,S:ProxyHop1=exchange-mail.my.domain.com(10.11.12.14);S:MessageValue=MediumHigh;S:Replication=Failed;S:FirstForestHop=exchange-mail.my.domain.com;S:FromEntity=Internet;S:ProxiedClientIPAddress=10.11.12.13;S:ProxiedClientHostname=host01.my.domain.com;S:DeliveryPriority=Normal;S:AccountForest=my.domain.com,Email,05503123-c5b9-46fe-1234-56dc1db88f8f,15.02.0330.005
2024-01-25T15:16:14.415Z,10.11.12.14,exchange-mail.my.domain.com,10.11.12.14,exchange-mail,08DC1DB12C345BE9;2024-01-25T15:16:12.885Z;0,exchange-mail\Default exchange-mail,SMTP,RECEIVE,70912345566407,<20240123200018.123C42553@pve-vhost01.my.domain.com>,c95b5dd1-f520-1234-e6dc-56dc1db8914d,mailuser@my.domain.com,,8251,1,,,vzdump backup status (pve-vhost01.my.domain.com): backup successful,root@pve-vhost01.my.domain.com,root@pve-vhost01.my.domain.com,0cA: ,Incoming,,10.11.12.15,10.11.12.14,S:ProxyHop1=exchange-mail.my.domain.com(10.11.12.14);S:MessageValue=MediumHigh;S:Replication=Failed;S:FirstForestHop=exchange-mail.my.domain.com;S:FromEntity=Internet;S:ProxiedClientIPAddress=10.11.12.15;S:ProxiedClientHostname=pve-vhost01.my.domain.com;S:DeliveryPriority=Normal;S:AccountForest=my.domain.com,Email,d6aef52d-0e05-1234-e29b-56dc1db89238,15.02.0330.005
2024-01-07T00:00:07.463Z,192.168.0.1,exchange,192.168.0.2,exchange.example.com,;250 2.0.0OK20240107001234.567E6224C8@monitor.example.com[Hostname=exchange.example.com];ClientSubmitTime:,Intra-Organization SMTP Send Connector,SMTP,SEND,29519319995411,20240107001234.567E6224C8@monitor.example.com,0b7099ea-cb95-1234-328e-08dc5f139ac8,uwe.musterman@example.com,250 2.1.5Recipient OK,38663,1,,,ein Titel,support@example.com,support@example.com,2024-01-07T00:00:05.535Z;LSRV=exchange.example.com:TOTAL-HUB=1.921|SMR=0.127(SMRDE=0.002|SMRC=0.125(SMRCL=0.105|X-SMRCR=0.125))|CAT=1.698(CATOS=0.018(CATSM=0.017(CATSM-Malware Agent=0.017))|CATRESL=0.004|CATORES=1.567(CATRS=1.566(CATRS-ScanMail Routing Agent=0.117|CATRS-Transport Rule Agent=0.002(X-ETREX=0.002)|CATRS-Index Routing Agent=1.444))|CATORT=0.108(CATRT=0.107(CATRT-Journal Agent=0.107)))|QDM=0.010|SMSC=0.006(X-SMSDR=0.011)|SMS=0.076(SMSMBXD=0.071),Originating,,,,S:E2ELatency=1.928;S:MsgRecipCount=1;S:IncludeInSla=True;S:Microsoft.Exchange.Transport.MailRecipient.RequiredTlsAuthLevel=Opportunistic;S:IsSmtpResponseFromExternalServer=False;S:DeliveryPriority=Normal;S:AccountForest=example.com,Email,a7ae9ef9-e10c-4111-19bf-08dc0f111bee,15.01.2507.035
+2025-01-14T09:29:05.327Z,216.160.83.56,HELLOWORLD,175.16.199.1,global-mail-onmicrosoft-com.mail.protection.outlook.com,";250 2.6.0 <1736846922.907742.2942@foo.bar.local> [InternalId=5450313518243, Hostname=AAAAAAAAAAAAA.bbbbbbbb.prod.outlook.com] 141301 bytes in 0.101, 1361.827 KB/sec Queued mail for delivery;ClientSubmitTime:",Outbound to Office 365 - cb1ce4e7-cbb6-46e8-8a4c-e1f2efbdd202,SMTP,SENDEXTERNAL,8774618205228,<1736846922.907742.2942@foo.bar.local>,cb1ce4e7-cbb6-46e8-8a4c-e1f2efbdd202,email@email.com,250 2.1.5 Recipient OK,136349,1,,,John Doe,,<>,2025-01-14T09:28:50.814Z;SRV=HELLOWORLD.foo.example.com:TOTAL-FE=0.074|SMR=0.072(SMRPI=0.002(SMRPI-FrontendProxyAgent=0.002))|SMS=0.002;SRV=HELLOWORLD.foo.example.com:TOTAL-HUB=14.439|SMR=0.279(SMRDE=0.174|SMRC=0.105(SMRCL=0.105|X-SMRCR=0.016))|CAT=0.046(CATOS=0.018(CATSM=0.018(CATSM-Malware Agent=0.017))|CATRESL=0.003|CATORES=0.021(CATRS=0.021(CATRS-Prioritization Agent=0.002|CATRS-Index Routing Agent=0.017))|CATORT=0.002(CATRT=0.002(CATRT-CodeTwo Exchange Rules Transport Agent=0.001)))|QDE=13.818|SMS=0.291,Incoming,,,,S:E2ELatency=14.513;S:ExternalSendLatency=0.402;S:ToEntity=Internet;S:FromEntity=Internet;S:MsgRecipCount=1;S:IncludeInSla=True;S:Microsoft.Exchange.Transport.MailRecipient.RequiredTlsAuthLevel=DomainValidation;S:Microsoft.Exchange.Transport.MailRecipient.EffectiveTlsAuthLevel=DomainValidation;S:IsSmtpResponseFromExternalServer=True;S:DeliveryPriority=Normal;S:OriginalFromAddress=<>;S:AccountForest=wgs.wuerth.com,Email,82f8feae-ef9b-41bf-8500-1e10946655ae,15.01.2507.039
+2025-01-14T09:28:39.334Z,2a02:cf40::0000:1234:5678:9abc,HELLOWORLD,,,"MDB:f4adaa08-ff49-4bca-ba70-68e9b80597b0, Mailbox:84dd42d9-8814-4758-a8bb-e8765d62ec90, Event:190620893, MessageClass:IPM.Note, CreationTime:2025-01-14T09:28:39.321Z, ClientType:MOMT, SubmissionAssistant:MailboxTransportSubmissionEmailAssistant",,STOREDRIVER,NOTIFYMAPI,,,,,,,,,,,email@email.com,,2025-01-14T09:28:39.321Z;LSRV=HELLOWORLD.foo.example.com:TOTAL-SUB=0.013|SA=0.013|MTSS-PEN=0.000,,,,,S:ItemEntryId=00-00-00-00-39-E7-E4-05-8C-C2-08-4C-AC-96-9A-D0-DE-C4-85-EE-07-00-D8-00-A6-A0-EC-B2-39-4D-BD-2F-31-4C-F7-4F-3F-4C-00-00-00-14-1A-99-00-00-46-7D-7B-CD-1B-EA-FC-40-9C-37-A9-7E-B0-DD-4C-16-00-08-65-53-76-46-00-00,,74364e3b-32b7-4108-9a2c-6dda84007459,15.01.2507.039
diff --git a/packages/microsoft_exchange_server/data_stream/messagetracking/_dev/test/pipeline/test-messagetracking.log-expected.json b/packages/microsoft_exchange_server/data_stream/messagetracking/_dev/test/pipeline/test-messagetracking.log-expected.json
index 1ee68f0e1d..83c61ba3a5 100644
--- a/packages/microsoft_exchange_server/data_stream/messagetracking/_dev/test/pipeline/test-messagetracking.log-expected.json
+++ b/packages/microsoft_exchange_server/data_stream/messagetracking/_dev/test/pipeline/test-messagetracking.log-expected.json
@@ -6,15 +6,14 @@
"direction": "Incoming",
"from": {
"address": [
- "root@host01.my.domain.com"
+ "root@host01.my.domain.com",
+ "MAILER-DAEMON@host01.my.domain.com"
]
},
"local_id": "2fd37dca-1234-5bfb-175d-08dc1db88f52",
"message_id": "<20240124222112.B4AE1234EF@host01.my.domain.com>",
"sender": {
- "address": [
- "MAILER-DAEMON@host01.my.domain.com"
- ]
+ "address": "MAILER-DAEMON@host01.my.domain.com"
},
"subject": "Undelivered Mail Returned to Sender",
"to": {
@@ -24,7 +23,7 @@
}
},
"event": {
- "ingested": "2024-09-04T11:53:25.104693004Z",
+ "ingested": "2025-01-31T22:06:04.860204507Z",
"original": "2024-01-25T15:16:09.843Z,,,,exchange-mail,No suitable shadow servers,,SMTP,HAREDIRECTFAIL,70971234566456,<20240124222112.B4AE1234EF@host01.my.domain.com>,2fd37dca-1234-5bfb-175d-08dc1db88f52,mailuser@my.domain.com,,15054,1,,,Undelivered Mail Returned to Sender,MAILER-DAEMON@host01.my.domain.com,root@host01.my.domain.com,,Incoming,,,,S:DeliveryPriority=Normal;S:OriginalFromAddress=root@host01.my.domain.com;S:AccountForest=my.domain.com,Email,dc69df25-1234-564c-41c4-08dc1db88f7f,15.02.0330.005"
},
"message": "2024-01-25T15:16:09.843Z,,,,exchange-mail,No suitable shadow servers,,SMTP,HAREDIRECTFAIL,70971234566456,<20240124222112.B4AE1234EF@host01.my.domain.com>,2fd37dca-1234-5bfb-175d-08dc1db88f52,mailuser@my.domain.com,,15054,1,,,Undelivered Mail Returned to Sender,MAILER-DAEMON@host01.my.domain.com,root@host01.my.domain.com,,Incoming,,,,S:DeliveryPriority=Normal;S:OriginalFromAddress=root@host01.my.domain.com;S:AccountForest=my.domain.com,Email,dc69df25-1234-564c-41c4-08dc1db88f7f,15.02.0330.005",
@@ -69,9 +68,7 @@
"local_id": "1e6eb197-c6b4-1234-1b69-56dc1db88f50",
"message_id": "<20240123200014.123F425E28@host01.my.domain.com>",
"sender": {
- "address": [
- "root@host01.my.domain.com"
- ]
+ "address": "root@host01.my.domain.com"
},
"subject": "vzdump backup status (host01.my.domain.com): backup successful",
"to": {
@@ -81,7 +78,7 @@
}
},
"event": {
- "ingested": "2024-09-04T11:53:25.104705446Z",
+ "ingested": "2025-01-31T22:06:04.86022234Z",
"original": "2024-01-25T15:16:09.949Z,10.11.12.14,exchange-mail.my.domain.com,10.11.12.14,exchange-mail,08DC1DB12C345BE5;2024-01-25T15:16:09.544Z;0,exchange-mail\\Default exchange-mail,SMTP,RECEIVE,70912345566403,<20240123200014.123F425E28@host01.my.domain.com>,1e6eb197-c6b4-1234-1b69-56dc1db88f50,mailuser@my.domain.com,,7229,1,,,vzdump backup status (host01.my.domain.com): backup successful,root@host01.my.domain.com,root@host01.my.domain.com,0cA: ,Incoming,,10.11.12.13,10.11.12.14,S:ProxyHop1=exchange-mail.my.domain.com(10.11.12.14);S:MessageValue=MediumHigh;S:Replication=Failed;S:FirstForestHop=exchange-mail.my.domain.com;S:FromEntity=Internet;S:ProxiedClientIPAddress=10.11.12.13;S:ProxiedClientHostname=host01.my.domain.com;S:DeliveryPriority=Normal;S:AccountForest=my.domain.com,Email,05503123-c5b9-46fe-1234-56dc1db88f8f,15.02.0330.005"
},
"message": "2024-01-25T15:16:09.949Z,10.11.12.14,exchange-mail.my.domain.com,10.11.12.14,exchange-mail,08DC1DB12C345BE5;2024-01-25T15:16:09.544Z;0,exchange-mail\\Default exchange-mail,SMTP,RECEIVE,70912345566403,<20240123200014.123F425E28@host01.my.domain.com>,1e6eb197-c6b4-1234-1b69-56dc1db88f50,mailuser@my.domain.com,,7229,1,,,vzdump backup status (host01.my.domain.com): backup successful,root@host01.my.domain.com,root@host01.my.domain.com,0cA: ,Incoming,,10.11.12.13,10.11.12.14,S:ProxyHop1=exchange-mail.my.domain.com(10.11.12.14);S:MessageValue=MediumHigh;S:Replication=Failed;S:FirstForestHop=exchange-mail.my.domain.com;S:FromEntity=Internet;S:ProxiedClientIPAddress=10.11.12.13;S:ProxiedClientHostname=host01.my.domain.com;S:DeliveryPriority=Normal;S:AccountForest=my.domain.com,Email,05503123-c5b9-46fe-1234-56dc1db88f8f,15.02.0330.005",
@@ -131,9 +128,7 @@
"local_id": "c95b5dd1-f520-1234-e6dc-56dc1db8914d",
"message_id": "<20240123200018.123C42553@pve-vhost01.my.domain.com>",
"sender": {
- "address": [
- "root@pve-vhost01.my.domain.com"
- ]
+ "address": "root@pve-vhost01.my.domain.com"
},
"subject": "vzdump backup status (pve-vhost01.my.domain.com): backup successful",
"to": {
@@ -143,7 +138,7 @@
}
},
"event": {
- "ingested": "2024-09-04T11:53:25.1047078Z",
+ "ingested": "2025-01-31T22:06:04.86022384Z",
"original": "2024-01-25T15:16:14.415Z,10.11.12.14,exchange-mail.my.domain.com,10.11.12.14,exchange-mail,08DC1DB12C345BE9;2024-01-25T15:16:12.885Z;0,exchange-mail\\Default exchange-mail,SMTP,RECEIVE,70912345566407,<20240123200018.123C42553@pve-vhost01.my.domain.com>,c95b5dd1-f520-1234-e6dc-56dc1db8914d,mailuser@my.domain.com,,8251,1,,,vzdump backup status (pve-vhost01.my.domain.com): backup successful,root@pve-vhost01.my.domain.com,root@pve-vhost01.my.domain.com,0cA: ,Incoming,,10.11.12.15,10.11.12.14,S:ProxyHop1=exchange-mail.my.domain.com(10.11.12.14);S:MessageValue=MediumHigh;S:Replication=Failed;S:FirstForestHop=exchange-mail.my.domain.com;S:FromEntity=Internet;S:ProxiedClientIPAddress=10.11.12.15;S:ProxiedClientHostname=pve-vhost01.my.domain.com;S:DeliveryPriority=Normal;S:AccountForest=my.domain.com,Email,d6aef52d-0e05-1234-e29b-56dc1db89238,15.02.0330.005"
},
"message": "2024-01-25T15:16:14.415Z,10.11.12.14,exchange-mail.my.domain.com,10.11.12.14,exchange-mail,08DC1DB12C345BE9;2024-01-25T15:16:12.885Z;0,exchange-mail\\Default exchange-mail,SMTP,RECEIVE,70912345566407,<20240123200018.123C42553@pve-vhost01.my.domain.com>,c95b5dd1-f520-1234-e6dc-56dc1db8914d,mailuser@my.domain.com,,8251,1,,,vzdump backup status (pve-vhost01.my.domain.com): backup successful,root@pve-vhost01.my.domain.com,root@pve-vhost01.my.domain.com,0cA: ,Incoming,,10.11.12.15,10.11.12.14,S:ProxyHop1=exchange-mail.my.domain.com(10.11.12.14);S:MessageValue=MediumHigh;S:Replication=Failed;S:FirstForestHop=exchange-mail.my.domain.com;S:FromEntity=Internet;S:ProxiedClientIPAddress=10.11.12.15;S:ProxiedClientHostname=pve-vhost01.my.domain.com;S:DeliveryPriority=Normal;S:AccountForest=my.domain.com,Email,d6aef52d-0e05-1234-e29b-56dc1db89238,15.02.0330.005",
@@ -193,9 +188,7 @@
"local_id": "0b7099ea-cb95-1234-328e-08dc5f139ac8",
"message_id": "20240107001234.567E6224C8@monitor.example.com",
"sender": {
- "address": [
- "support@example.com"
- ]
+ "address": "support@example.com"
},
"subject": "ein Titel",
"to": {
@@ -205,7 +198,7 @@
}
},
"event": {
- "ingested": "2024-09-04T11:53:25.104709787Z",
+ "ingested": "2025-01-31T22:06:04.860224882Z",
"original": "2024-01-07T00:00:07.463Z,192.168.0.1,exchange,192.168.0.2,exchange.example.com,;250 2.0.0OK20240107001234.567E6224C8@monitor.example.com[Hostname=exchange.example.com];ClientSubmitTime:,Intra-Organization SMTP Send Connector,SMTP,SEND,29519319995411,20240107001234.567E6224C8@monitor.example.com,0b7099ea-cb95-1234-328e-08dc5f139ac8,uwe.musterman@example.com,250 2.1.5Recipient OK,38663,1,,,ein Titel,support@example.com,support@example.com,2024-01-07T00:00:05.535Z;LSRV=exchange.example.com:TOTAL-HUB=1.921|SMR=0.127(SMRDE=0.002|SMRC=0.125(SMRCL=0.105|X-SMRCR=0.125))|CAT=1.698(CATOS=0.018(CATSM=0.017(CATSM-Malware Agent=0.017))|CATRESL=0.004|CATORES=1.567(CATRS=1.566(CATRS-ScanMail Routing Agent=0.117|CATRS-Transport Rule Agent=0.002(X-ETREX=0.002)|CATRS-Index Routing Agent=1.444))|CATORT=0.108(CATRT=0.107(CATRT-Journal Agent=0.107)))|QDM=0.010|SMSC=0.006(X-SMSDR=0.011)|SMS=0.076(SMSMBXD=0.071),Originating,,,,S:E2ELatency=1.928;S:MsgRecipCount=1;S:IncludeInSla=True;S:Microsoft.Exchange.Transport.MailRecipient.RequiredTlsAuthLevel=Opportunistic;S:IsSmtpResponseFromExternalServer=False;S:DeliveryPriority=Normal;S:AccountForest=example.com,Email,a7ae9ef9-e10c-4111-19bf-08dc0f111bee,15.01.2507.035"
},
"message": "2024-01-07T00:00:07.463Z,192.168.0.1,exchange,192.168.0.2,exchange.example.com,;250 2.0.0OK20240107001234.567E6224C8@monitor.example.com[Hostname=exchange.example.com];ClientSubmitTime:,Intra-Organization SMTP Send Connector,SMTP,SEND,29519319995411,20240107001234.567E6224C8@monitor.example.com,0b7099ea-cb95-1234-328e-08dc5f139ac8,uwe.musterman@example.com,250 2.1.5Recipient OK,38663,1,,,ein Titel,support@example.com,support@example.com,2024-01-07T00:00:05.535Z;LSRV=exchange.example.com:TOTAL-HUB=1.921|SMR=0.127(SMRDE=0.002|SMRC=0.125(SMRCL=0.105|X-SMRCR=0.125))|CAT=1.698(CATOS=0.018(CATSM=0.017(CATSM-Malware Agent=0.017))|CATRESL=0.004|CATORES=1.567(CATRS=1.566(CATRS-ScanMail Routing Agent=0.117|CATRS-Transport Rule Agent=0.002(X-ETREX=0.002)|CATRS-Index Routing Agent=1.444))|CATORT=0.108(CATRT=0.107(CATRT-Journal Agent=0.107)))|QDM=0.010|SMSC=0.006(X-SMSDR=0.011)|SMS=0.076(SMSMBXD=0.071),Originating,,,,S:E2ELatency=1.928;S:MsgRecipCount=1;S:IncludeInSla=True;S:Microsoft.Exchange.Transport.MailRecipient.RequiredTlsAuthLevel=Opportunistic;S:IsSmtpResponseFromExternalServer=False;S:DeliveryPriority=Normal;S:AccountForest=example.com,Email,a7ae9ef9-e10c-4111-19bf-08dc0f111bee,15.01.2507.035",
@@ -237,6 +230,93 @@
"tags": [
"preserve_original_event"
]
+ },
+ {
+ "@timestamp": "2025-01-14T09:29:05.327Z",
+ "client": {
+ "domain": "HELLOWORLD",
+ "ip": "216.160.83.56"
+ },
+ "email": {
+ "direction": "Incoming",
+ "local_id": "cb1ce4e7-cbb6-46e8-8a4c-e1f2efbdd202",
+ "message_id": "<1736846922.907742.2942@foo.bar.local>",
+ "subject": "John Doe",
+ "to": {
+ "address": [
+ "email@email.com"
+ ]
+ }
+ },
+ "event": {
+ "ingested": "2025-01-31T22:06:04.860225882Z",
+ "original": "2025-01-14T09:29:05.327Z,216.160.83.56,HELLOWORLD,175.16.199.1,global-mail-onmicrosoft-com.mail.protection.outlook.com,\";250 2.6.0 <1736846922.907742.2942@foo.bar.local> [InternalId=5450313518243, Hostname=AAAAAAAAAAAAA.bbbbbbbb.prod.outlook.com] 141301 bytes in 0.101, 1361.827 KB/sec Queued mail for delivery;ClientSubmitTime:\",Outbound to Office 365 - cb1ce4e7-cbb6-46e8-8a4c-e1f2efbdd202,SMTP,SENDEXTERNAL,8774618205228,<1736846922.907742.2942@foo.bar.local>,cb1ce4e7-cbb6-46e8-8a4c-e1f2efbdd202,email@email.com,250 2.1.5 Recipient OK,136349,1,,,John Doe,,<>,2025-01-14T09:28:50.814Z;SRV=HELLOWORLD.foo.example.com:TOTAL-FE=0.074|SMR=0.072(SMRPI=0.002(SMRPI-FrontendProxyAgent=0.002))|SMS=0.002;SRV=HELLOWORLD.foo.example.com:TOTAL-HUB=14.439|SMR=0.279(SMRDE=0.174|SMRC=0.105(SMRCL=0.105|X-SMRCR=0.016))|CAT=0.046(CATOS=0.018(CATSM=0.018(CATSM-Malware Agent=0.017))|CATRESL=0.003|CATORES=0.021(CATRS=0.021(CATRS-Prioritization Agent=0.002|CATRS-Index Routing Agent=0.017))|CATORT=0.002(CATRT=0.002(CATRT-CodeTwo Exchange Rules Transport Agent=0.001)))|QDE=13.818|SMS=0.291,Incoming,,,,S:E2ELatency=14.513;S:ExternalSendLatency=0.402;S:ToEntity=Internet;S:FromEntity=Internet;S:MsgRecipCount=1;S:IncludeInSla=True;S:Microsoft.Exchange.Transport.MailRecipient.RequiredTlsAuthLevel=DomainValidation;S:Microsoft.Exchange.Transport.MailRecipient.EffectiveTlsAuthLevel=DomainValidation;S:IsSmtpResponseFromExternalServer=True;S:DeliveryPriority=Normal;S:OriginalFromAddress=<>;S:AccountForest=wgs.wuerth.com,Email,82f8feae-ef9b-41bf-8500-1e10946655ae,15.01.2507.039"
+ },
+ "message": "2025-01-14T09:29:05.327Z,216.160.83.56,HELLOWORLD,175.16.199.1,global-mail-onmicrosoft-com.mail.protection.outlook.com,\";250 2.6.0 <1736846922.907742.2942@foo.bar.local> [InternalId=5450313518243, Hostname=AAAAAAAAAAAAA.bbbbbbbb.prod.outlook.com] 141301 bytes in 0.101, 1361.827 KB/sec Queued mail for delivery;ClientSubmitTime:\",Outbound to Office 365 - cb1ce4e7-cbb6-46e8-8a4c-e1f2efbdd202,SMTP,SENDEXTERNAL,8774618205228,<1736846922.907742.2942@foo.bar.local>,cb1ce4e7-cbb6-46e8-8a4c-e1f2efbdd202,email@email.com,250 2.1.5 Recipient OK,136349,1,,,John Doe,,<>,2025-01-14T09:28:50.814Z;SRV=HELLOWORLD.foo.example.com:TOTAL-FE=0.074|SMR=0.072(SMRPI=0.002(SMRPI-FrontendProxyAgent=0.002))|SMS=0.002;SRV=HELLOWORLD.foo.example.com:TOTAL-HUB=14.439|SMR=0.279(SMRDE=0.174|SMRC=0.105(SMRCL=0.105|X-SMRCR=0.016))|CAT=0.046(CATOS=0.018(CATSM=0.018(CATSM-Malware Agent=0.017))|CATRESL=0.003|CATORES=0.021(CATRS=0.021(CATRS-Prioritization Agent=0.002|CATRS-Index Routing Agent=0.017))|CATORT=0.002(CATRT=0.002(CATRT-CodeTwo Exchange Rules Transport Agent=0.001)))|QDE=13.818|SMS=0.291,Incoming,,,,S:E2ELatency=14.513;S:ExternalSendLatency=0.402;S:ToEntity=Internet;S:FromEntity=Internet;S:MsgRecipCount=1;S:IncludeInSla=True;S:Microsoft.Exchange.Transport.MailRecipient.RequiredTlsAuthLevel=DomainValidation;S:Microsoft.Exchange.Transport.MailRecipient.EffectiveTlsAuthLevel=DomainValidation;S:IsSmtpResponseFromExternalServer=True;S:DeliveryPriority=Normal;S:OriginalFromAddress=<>;S:AccountForest=wgs.wuerth.com,Email,82f8feae-ef9b-41bf-8500-1e10946655ae,15.01.2507.039",
+ "microsoft": {
+ "exchange": {
+ "connectorid": "Outbound to Office 365 - cb1ce4e7-cbb6-46e8-8a4c-e1f2efbdd202",
+ "customdata": "S:E2ELatency=14.513;S:ExternalSendLatency=0.402;S:ToEntity=Internet;S:FromEntity=Internet;S:MsgRecipCount=1;S:IncludeInSla=True;S:Microsoft.Exchange.Transport.MailRecipient.RequiredTlsAuthLevel=DomainValidation;S:Microsoft.Exchange.Transport.MailRecipient.EffectiveTlsAuthLevel=DomainValidation;S:IsSmtpResponseFromExternalServer=True;S:DeliveryPriority=Normal;S:OriginalFromAddress=<>;S:AccountForest=wgs.wuerth.com",
+ "eventid": "SENDEXTERNAL",
+ "internalmessageid": "8774618205228",
+ "logid": "82f8feae-ef9b-41bf-8500-1e10946655ae",
+ "messageinfo": "2025-01-14T09:28:50.814Z;SRV=HELLOWORLD.foo.example.com:TOTAL-FE=0.074|SMR=0.072(SMRPI=0.002(SMRPI-FrontendProxyAgent=0.002))|SMS=0.002;SRV=HELLOWORLD.foo.example.com:TOTAL-HUB=14.439|SMR=0.279(SMRDE=0.174|SMRC=0.105(SMRCL=0.105|X-SMRCR=0.016))|CAT=0.046(CATOS=0.018(CATSM=0.018(CATSM-Malware Agent=0.017))|CATRESL=0.003|CATORES=0.021(CATRS=0.021(CATRS-Prioritization Agent=0.002|CATRS-Index Routing Agent=0.017))|CATORT=0.002(CATRT=0.002(CATRT-CodeTwo Exchange Rules Transport Agent=0.001)))|QDE=13.818|SMS=0.291",
+ "networkmessageid": "cb1ce4e7-cbb6-46e8-8a4c-e1f2efbdd202",
+ "recipientcount": 1,
+ "recipientstatus": "250 2.1.5 Recipient OK",
+ "returnpath": "<>",
+ "schemaversion": "15.01.2507.039",
+ "source": "SMTP",
+ "sourcecontext": ";250 2.6.0 <1736846922.907742.2942@foo.bar.local> [InternalId=5450313518243, Hostname=AAAAAAAAAAAAA.bbbbbbbb.prod.outlook.com] 141301 bytes in 0.101, 1361.827 KB/sec Queued mail for delivery;ClientSubmitTime:",
+ "transporttraffictype": "Email"
+ }
+ },
+ "network": {
+ "bytes": 136349
+ },
+ "server": {
+ "domain": "global-mail-onmicrosoft-com.mail.protection.outlook.com",
+ "ip": "175.16.199.1"
+ },
+ "tags": [
+ "preserve_original_event"
+ ]
+ },
+ {
+ "@timestamp": "2025-01-14T09:28:39.334Z",
+ "client": {
+ "domain": "HELLOWORLD",
+ "ip": "2a02:cf40::0000:1234:5678:9abc"
+ },
+ "email": {
+ "from": {
+ "address": [
+ "email@email.com"
+ ]
+ },
+ "sender": {
+ "address": "email@email.com"
+ }
+ },
+ "event": {
+ "ingested": "2025-01-31T22:06:04.86022684Z",
+ "original": "2025-01-14T09:28:39.334Z,2a02:cf40::0000:1234:5678:9abc,HELLOWORLD,,,\"MDB:f4adaa08-ff49-4bca-ba70-68e9b80597b0, Mailbox:84dd42d9-8814-4758-a8bb-e8765d62ec90, Event:190620893, MessageClass:IPM.Note, CreationTime:2025-01-14T09:28:39.321Z, ClientType:MOMT, SubmissionAssistant:MailboxTransportSubmissionEmailAssistant\",,STOREDRIVER,NOTIFYMAPI,,,,,,,,,,,email@email.com,,2025-01-14T09:28:39.321Z;LSRV=HELLOWORLD.foo.example.com:TOTAL-SUB=0.013|SA=0.013|MTSS-PEN=0.000,,,,,S:ItemEntryId=00-00-00-00-39-E7-E4-05-8C-C2-08-4C-AC-96-9A-D0-DE-C4-85-EE-07-00-D8-00-A6-A0-EC-B2-39-4D-BD-2F-31-4C-F7-4F-3F-4C-00-00-00-14-1A-99-00-00-46-7D-7B-CD-1B-EA-FC-40-9C-37-A9-7E-B0-DD-4C-16-00-08-65-53-76-46-00-00,,74364e3b-32b7-4108-9a2c-6dda84007459,15.01.2507.039"
+ },
+ "message": "2025-01-14T09:28:39.334Z,2a02:cf40::0000:1234:5678:9abc,HELLOWORLD,,,\"MDB:f4adaa08-ff49-4bca-ba70-68e9b80597b0, Mailbox:84dd42d9-8814-4758-a8bb-e8765d62ec90, Event:190620893, MessageClass:IPM.Note, CreationTime:2025-01-14T09:28:39.321Z, ClientType:MOMT, SubmissionAssistant:MailboxTransportSubmissionEmailAssistant\",,STOREDRIVER,NOTIFYMAPI,,,,,,,,,,,email@email.com,,2025-01-14T09:28:39.321Z;LSRV=HELLOWORLD.foo.example.com:TOTAL-SUB=0.013|SA=0.013|MTSS-PEN=0.000,,,,,S:ItemEntryId=00-00-00-00-39-E7-E4-05-8C-C2-08-4C-AC-96-9A-D0-DE-C4-85-EE-07-00-D8-00-A6-A0-EC-B2-39-4D-BD-2F-31-4C-F7-4F-3F-4C-00-00-00-14-1A-99-00-00-46-7D-7B-CD-1B-EA-FC-40-9C-37-A9-7E-B0-DD-4C-16-00-08-65-53-76-46-00-00,,74364e3b-32b7-4108-9a2c-6dda84007459,15.01.2507.039",
+ "microsoft": {
+ "exchange": {
+ "customdata": "S:ItemEntryId=00-00-00-00-39-E7-E4-05-8C-C2-08-4C-AC-96-9A-D0-DE-C4-85-EE-07-00-D8-00-A6-A0-EC-B2-39-4D-BD-2F-31-4C-F7-4F-3F-4C-00-00-00-14-1A-99-00-00-46-7D-7B-CD-1B-EA-FC-40-9C-37-A9-7E-B0-DD-4C-16-00-08-65-53-76-46-00-00",
+ "eventid": "NOTIFYMAPI",
+ "logid": "74364e3b-32b7-4108-9a2c-6dda84007459",
+ "messageinfo": "2025-01-14T09:28:39.321Z;LSRV=HELLOWORLD.foo.example.com:TOTAL-SUB=0.013|SA=0.013|MTSS-PEN=0.000",
+ "schemaversion": "15.01.2507.039",
+ "source": "STOREDRIVER",
+ "sourcecontext": "MDB:f4adaa08-ff49-4bca-ba70-68e9b80597b0, Mailbox:84dd42d9-8814-4758-a8bb-e8765d62ec90, Event:190620893, MessageClass:IPM.Note, CreationTime:2025-01-14T09:28:39.321Z, ClientType:MOMT, SubmissionAssistant:MailboxTransportSubmissionEmailAssistant"
+ }
+ },
+ "tags": [
+ "preserve_original_event"
+ ]
}
]
-}
\ No newline at end of file
+}
diff --git a/packages/microsoft_exchange_server/data_stream/messagetracking/elasticsearch/ingest_pipeline/default.yml b/packages/microsoft_exchange_server/data_stream/messagetracking/elasticsearch/ingest_pipeline/default.yml
index 1b9e5c7d38..efcc35b3bb 100644
--- a/packages/microsoft_exchange_server/data_stream/messagetracking/elasticsearch/ingest_pipeline/default.yml
+++ b/packages/microsoft_exchange_server/data_stream/messagetracking/elasticsearch/ingest_pipeline/default.yml
@@ -52,10 +52,10 @@ processors:
field: email.local_id
copy_from: microsoft.exchange.networkmessageid
ignore_empty_value: true
-- append:
+- set:
field: email.sender.address
- value: "{{{microsoft.exchange.senderaddress}}}"
- ignore_failure: true
+ copy_from: microsoft.exchange.senderaddress
+ ignore_empty_value: true
- dissect:
field: microsoft.exchange.customdata
pattern: "%{}S:OriginalFromAddress=%{_tmp.email.from.address};"
@@ -63,13 +63,12 @@ processors:
- append:
field: email.from.address
value: "{{{_tmp.email.from.address}}}"
- if: ctx._tmp?.email?.from?.address != null && ctx?._tmp?.email?.from?.address != "<>"
+ if: ctx._tmp?.email?.from?.address != null && ctx._tmp.email.from.address != "<>"
ignore_failure: true
- append:
field: email.from.address
value: "{{{microsoft.exchange.senderaddress}}}"
- if: ctx.email?.from?.address == null
- ignore_failure: true
+ if: ctx.microsoft?.exchange?.senderaddress != null && ctx.microsoft.exchange.senderaddress != ""
- remove:
field: microsoft.exchange.senderaddress
ignore_missing: true
@@ -84,7 +83,7 @@ processors:
ignore_failure: true
ignore_missing: true
- set:
- field: event.ingested
+ field: event.ingested
copy_from: _ingest.timestamp
ignore_failure: true
- remove:
Comment on lines
55
to
57
Contributor
There was a problem hiding this comment.
According to ECS, email.sender.address should be a scalar. So I think we should have
- set:
field: email.sender.address
copy_from: microsoft.exchange.senderaddress
ignore_empty_value: trueThis also handles the case where senderaddress is empty.
Contributor
There was a problem hiding this comment.
The above append processor needs to handle the case where senderaddress is emtpy. The current if condition looks wrong.
- append:
field: email.from.address
value: "{{{microsoft.exchange.senderaddress}}}"
if: ctx.microsoft?.exchange?.senderaddress != null && ctx.microsoft.exchange.senderaddress != ""
andrewkroh
changed the title
qcorporation
force-pushed
the
allow_empty_fields_in_the_exchange_integration
branch
from
February 3, 2025 22:34
cbae7cb to
c30036b
Compare
andrewkroh
added
Integration:1password
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
qcorporation
force-pushed
the
allow_empty_fields_in_the_exchange_integration
branch
from
February 4, 2025 18:13
c30036b to
6e46c11
Compare
qcorporation
force-pushed
the
main
branch
2 times, most recently
from
February 5, 2025 22:00
eda4138 to
f728ca7
Compare
💔 Build Failed
Failed CI StepsHistory
|
|
2 of 5 tasks
Contributor
Author
|
This PR was replaced with this one. #12846 |
matthewscherer
deleted the
allow_empty_fields_in_the_exchange_integration
branch
February 20, 2025 20:30
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed commit message
A customer came across a couple log lines with missing networkmessageid and senderaddress fields. This in turn rejected the logs messages. Allowing these two fields to be empty or otherwise missing will allow them to pass the ingest function.
Checklist
changelog.ymlfile.Author's Checklist
How to test this PR locally
Related issues