Repository navigation
[o365] Field file.size being mapped as keyword leading to mapping conflict and rules failing #21137
Description
Hello,
On the o365.audit dataset the field file.size is now being wrongly mapped as a keyword field, this field is populated from one of those fields in the ingest pipeline:
- set:
tag: set_file_size_723563fe
field: file.size
copy_from: o365audit.FileSizeBytes
ignore_empty_value: true
- set:
tag: set_file_size_c6196fe8
field: file.size
copy_from: o365audit.FileSize
ignore_empty_value: true
override: false
Since there is no explicit mapping for file.size, it will be dynamically mapped depending on the value from the field.
Somewhere between May and June the field o365audit.FileSizeBytes started arriving as a keyword instead as a long, which lead to a mapping conflict in kibana on all backing indices created after that change, which can break dashboards and security rules.
On an event from May I have this:
But on an event on June I have this:
So it changed from a long to a keyword, since I have a backing index with file.size mapped as long, this leads to mapping conflicts and break security rules, one example of a rule breaking is the native rule: M365 OneDrive/SharePoint Excessive File Downloads
This can be fixed with a convert processor to convert the file.size or using a explicit mapping for it.
Activity
infra-vault-gh-plugin-prod commentedon Sep 8, 2026
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)
[-][M365] Field file.size being mapped as keyword leading to mapping conflict and rules failing[/-][+][o365] Field file.size being mapped as keyword leading to mapping conflict and rules failing[/+]efd6 commentedon Sep 8, 2026
This was caused by #19245. Since
file.sizeis an ECS-defined field, with typelong, dynamically mapping it to akeywordis a bug, so this should be fixed. It would be a breaking change since users who have ingested data since #19245 will have the field mapped as akeyword(and so see the same issue that you are seeing, but from the other direction).I've just gone through my notes; I have one that specifically refers to #19245 and this behaviour from 2026-06-24. This is the final conclusion in that note:
The assessment of the change:
chrisberkhout commentedon Sep 9, 2026
#19245 letting a string value come out of the pipeline for
file.sizewould not be a problem if thefile.sizefield was mapped to its ECS type.I previously assumed the ECS dynamic mappings shipped with Elasticsearch would map ECS fields correctly, but the only do that for some ECS fields.
The easiest way for #19245 to have avoided the problem, or to fix it now, is to add an explicit ECS mapping for
file.size.Perhaps the type change in the
_sourcedocument could be considered breaking, but I don't think we have clear rules about that.leandrojmp commentedon Sep 9, 2026
hello @chrisberkhout
There is an old issue to discuss how to deal with those kind of breaking changes and make it clear for the users, it is this one: #13861
But it seems to be stalled.
A mapping type change may always introduce a breaking change in Security Rules and Visualizations/Dashboards and the workaround to fix this (reindex, remove the data, runtime field) may not be possible.
[o365] Convert stringified numbers back to numbers in the audit pipel…