[brijesh-elastic]

Proposed commit message

o365: fix pipeline failures on list events with a named ListBaseType

SharePoint and OneDrive list events (RecordType 36, such as ListViewed,
ListUpdated, ListColumnCreated and ListColumnUpdated) report
ListBaseType as a name such as "DocumentLibrary". Since 4.0.0 (#21875),
ListBaseType has been in the script_normalize_numbers field list, and
these events failed with:

  Character D is neither a decimal digit number, decimal point, nor "e"
  notation exponential mark.

The script treats any string that contains an "e" or "E" as a number in
scientific notation and passes it to
new BigDecimal(value).toBigIntegerExact(). The names contain an "e", so
BigDecimal threw, and the whole document was indexed as
event.kind: pipeline_error with only the raw o365audit.* fields.

Microsoft documents ListBaseType as a string enumeration, so remove it
from the list. Also catch NumberFormatException and ArithmeticException
in normalize() and return the value unchanged, the same way
script_convert_numbers handles values it cannot parse, so that a
non-integer string in another listed field no longer fails the whole
document. Numbers from the o365audit input, and numeric strings in
scientific notation from the CEL input, are normalized as before.

Add a SharePoint ListViewed event with a ListBaseType of
"DocumentLibrary" to the pipeline tests, and an event with a
non-integer UserType ("1.5e0") and a named LogonType ("Delegate") so
both catch branches are covered.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

• Clone integrations repo.
• Install elastic package locally.
• Start elastic stack using elastic-package.
• Move to integrations/packages/o365 directory.
• Run the following command to run tests.

elastic-package test pipeline --data-streams audit

The RecordType 36 ListViewed event in test-sharepoint-events.json
fails on main in script_normalize_numbers. With the change it is
indexed with event.action: ListViewed and
o365.audit.ListBaseType: DocumentLibrary. The new event in
test-numbery-json-events.json keeps UserType: "1.5e0" and
LogonType: "Delegate" unchanged instead of failing.

Related issues

• Relates [o365] script_normalize_numbers fails on SharePoint/OneDrive list events (RecordType 36) when ListBaseType is "DocumentLibrary" (4.0.1) #22053

[infra-vault-gh-plugin-prod]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[github-actions]

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

---

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

[Copilot]

🟢 Approval recommended

The localized fix preserves successful conversions and includes regression coverage, with only a nonblocking changelog-link correction remaining.

1 open issue

• Fix 4.0.2 changelog link to reference this pull request · New

What changed in this PR

This PR fixes Office 365 audit pipeline failures when SharePoint or OneDrive events contain a non-numeric ListBaseType, addressing #22053.

Changes:

• Preserve strings that cannot be normalized as integers while retaining successful numeric conversions.
• Add regression coverage for ListBaseType: DocumentLibrary.
• Bump the package to 4.0.2 and document the fix.

File	Description
packages/​o365/​manifest.yml	Bumps the package version to 4.0.2.
packages/​o365/​data_stream/​audit/​elasticsearch/​ingest_pipeline/​default.yml	Handles number-format and arithmetic exceptions during normalization.
packages/​o365/​data_stream/​audit/​_dev/​test/​pipeline/​test-sharepoint-events.json-expected.json	Expects the non-numeric value to survive successful processing.
packages/​o365/​data_stream/​audit/​_dev/​test/​pipeline/​test-sharepoint-events.json	Adds DocumentLibrary to a regression fixture.
packages/​o365/​changelog.yml	Adds the 4.0.2 bugfix release note.

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

The narrowly scoped fix preserves successful numeric conversions, includes a regression fixture, and has no identified blocking issues.

No open issues 🎉

1 resolved since last review

• Fix 4.0.2 changelog link to reference this pull request

_{Review effort: Balanced}

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[marc-gr]

Verified: with the PR, elastic-package test pipeline --data-streams audit
passes (54 cases, ES 8.19.21). With the pipeline reverted to main, only
test-sharepoint-events.json fails, with the error from #22053. Nothing
blocking.

Requested changes

1. Remove ListBaseType from script_normalize_numbers params.fields;
   keep the try/catch.
   Microsoft documents it as a string: the Management Activity API schema
   has ListBaseType | Edm.String, and MS-SITEDATS defines it as an enum of
   names (UnspecifiedBaseType, GenericList, DocumentLibrary, Unused,
   DiscussionBoard, Survey, Issue). As efd6 said on #22053, it should not
   have been added to the list in #21875. The PR's reason for keeping it
   ("a numeric value from the o365audit input still becomes an integer
   string, as it did before 3.8.2") does not hold: before 3.8.2 the pipeline
   ran convert type: string on it (convert-listbasetype-to-string), which
   turns a double 1.0 into "1.0", not "1". The field is mapped keyword, so
   a numeric value still indexes fine without the normalisation.

2. Add a test for the ArithmeticException branch.
   Removing that catch leaves every pipeline test passing. The branch is
   reachable: on main, "1.5e0" and "1.5E-3" fail with "Rounding
   necessary" and "1e2147483648" with "Underflow"; with the PR they are
   kept unchanged. A value like "1.5e0" in a normalised field (for example
   UserType) in test-numbery-json-events.json, next to the #21875 number
   cases, would cover it.

3. Use a real list event in the test.
   The new ListBaseType is added to a PageViewed event (RecordType 4,
   ItemType: Page), which does not carry that field in real data. Please
   add a separate RecordType 36 ListViewed (or ListUpdated) event so the
   test shows the case from #22053. It goes through the same processors, so
   the code path is unchanged.

Suggestions

4. Commit message, input attribution. "Numeric values, including those
   in scientific notation from the legacy o365audit input, are normalized as
   before" names the wrong input. o365audit values arrive as Number and
   take the longValue() branch; strings in scientific notation come from
   the CEL input (decode_json_string_numbers()). Suggested: "Numbers from
   the o365audit input, and numeric strings in scientific notation from the
   CEL input, are normalized as before."

5. Changelog: say when it started. Suggested:

   Fix ingest pipeline failures, introduced in 4.0.0, on SharePoint and
   OneDrive list events whose ListBaseType is a name such as
   DocumentLibrary. These events were indexed as pipeline_error
   without any ECS fields.

6. Pipeline description. "Values that are not integers ... are left
   unchanged" is only true for strings; a number 1.5 still becomes "1"
   through longValue(). With item 1 the ListBaseType example also goes.
   Suggested: "Strings that are not integers are left unchanged."

Suggested proposed commit message (if items 1 to 4 are taken)

o365: fix pipeline failures on list events with a named ListBaseType

SharePoint and OneDrive list events (RecordType 36, such as ListViewed,
ListUpdated, ListColumnCreated and ListColumnUpdated) report
ListBaseType as a name such as "DocumentLibrary". Since 4.0.0 (#21875),
ListBaseType has been in the script_normalize_numbers field list, and
these events failed with:

  Character D is neither a decimal digit number, decimal point, nor "e"
  notation exponential mark.

The script treats any string that contains an "e" or "E" as a number in
scientific notation and passes it to
new BigDecimal(value).toBigIntegerExact(). The names contain an "e", so
BigDecimal threw, and the whole document was indexed as
event.kind: pipeline_error with only the raw o365audit.* fields.

Microsoft documents ListBaseType as a string enumeration, so remove it
from the list. Also catch NumberFormatException and ArithmeticException
in normalize() and return the value unchanged, the same way
script_convert_numbers handles values it cannot parse, so that a
non-integer string in another listed field no longer fails the whole
document. Numbers from the o365audit input, and numeric strings in
scientific notation from the CEL input, are normalized as before.

Add a SharePoint list event with a ListBaseType of "DocumentLibrary",
and a non-integer value in scientific notation, to the pipeline tests.

How to test (for the PR body)

cd packages/o365
elastic-package test pipeline --data-streams audit

All cases pass. The RecordType 36 event fails on main in
script_normalize_numbers and, with the change, is indexed with
event.action: ListViewed and o365.audit.ListBaseType: DocumentLibrary.
No system or script test reaches this path (no mock record has
ListBaseType or RecordType 36), so system tests add nothing here.

Checked and fine

• Version 4.0.2 in manifest and changelog, type: bugfix, link correct.
• No mapping change: o365.audit.ListBaseType is already keyword
  (fields/fields.yml:416), README matches, no dashboard uses it.
• No backport: o365 has no backport branches, and the bug exists only in
  4.0.0 and 4.0.1.
• No overlapping open PR (only #21804, which touches script tests only).
• Closes #22053 is right.

Not checked: Elasticsearch 9.x, the legacy o365audit input end to end,
real Microsoft RecordType 36 payloads.

[Copilot]

🟡 Changes recommended

Removing ListBaseType from params.fields bypasses normalization and regresses numeric compatibility for legacy values such as 1.0 and "1e0".

1 open issue

• Restore ListBaseType field normalization · New

_{Review effort: Balanced}

[vera-review-bot]

🟢 Reviewed the latest commits 1897f67 — nothing new beyond already posted comments.

A new commit triggers another review — at most once every 15 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot - v0.4.3 | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

[brijesh-elastic]

Thanks @marc-gr for the detailed review and for running the tests.

1. Removed ListBaseType from params.fields and kept the try/catch. You're
   right about the pre-3.8.2 behaviour: convert type: string gave "1.0",
   not "1", so keeping it didn't preserve anything.
2. Added an event to test-numbery-json-events.json with UserType "1.5e0"
   (ArithmeticException branch). With ListBaseType gone from the list,
   nothing reached the NumberFormatException branch either, so the same
   event also has LogonType "Delegate".
3. Reverted the PageViewed change and added a separate RecordType 36
   ListViewed event with ListBaseType "DocumentLibrary".

Also took the commit message, changelog and pipeline description
suggestions, and updated the PR description.

[elastic-vault-github-plugin-prod]

✅ All changelog entries have the correct PR link.

[infra-vault-gh-plugin-prod]

💚 Build Succeeded

• Buildkite Build
• Commit: 1897f67

History

• 💚 Build #52168 succeeded 184de44

cc @brijesh-elastic

[mergify]

This pull request does not currently match the merge queue conditions, so it cannot be queued from here. The box comes back if it matches again.

[elastic-vault-github-plugin-prod]

Package o365 - 4.0.2 containing this change is available at https://epr.elastic.co/package/o365/4.0.2/