[ilyannn]

Note: drafted with 🤖 Claude Code/Claude Opus 5.5 under my supervision.

What goes wrong today

Five field-mapping bugs in the log data stream. All are silent: no error, no tag, and the existing pipeline tests passed with them in place.

email.cc.address is never set. On email filter logs with cc=, the append that fills email.cc.address is gated on ctx.fortinet?.cc?.address. The kv script writes every key under fortinet.firewall.*, and nothing writes fortinet.cc.address, so the condition is never true and the CC recipients are not indexed.

The email sender is also indexed as a TLS certificate issuer. utm.yml appends sender= to email.sender.address, then an unguarded rename moves the same field to tls.server.issuer. Every email filter event with sender= gets an email address in tls.server.issuer, where TLS dashboards and rules read it.

email.to.address is a string, and the append after it never runs. utm.yml renames recipient= to email.to.address and then appends the same, by then missing, field. ECS defines email.to.address as an array. elastic-package test pipeline rejects the string (expected array, found "bob@example.org" (string)), but no test line had a recipient= until this change.

RFC 5424 format logs get no @timestamp from the log. A FortiGate with set format rfc5424 sends no date=/time= fields. The header timestamp is matched but not captured, and @timestamp is only set from date + time, so these events keep the time they were read rather than the time they were logged. The package's own test line (test-fortinet-7-4.log, line 75) has no @timestamp in its expected output.

FortiExtender signal statistics are not searchable. On "Cellular Signal Statistics" (logid 0111046409), rsrp, rsrq, signalstrength, sinr and temperature carry units (rsrp="-89 dBm"), but fields.yml maps them as integer and nothing converts them. On logs-* data streams (index.mapping.ignore_malformed: true) the values land in _ignored: kept in _source, but no query or aggregation can use them. Without ignore_malformed the document is rejected.

One line, two documents

A new email filter test line, built from the FortiOS log reference sample 0508020503 with documentation addresses:

date=2019-03-28 time=10:57:42 logid="0508020503" type="utm" subtype="emailfilter" eventtype="smtp" level="information" vd="vdom1" eventtime=1553795862 policyid=1 sessionid=1234567 srcip=192.0.2.11 srcport=49556 srcintf="port21" srcintfrole="undefined" dstip=198.51.100.55 dstport=25 dstintf="port23" dstintfrole="undefined" proto=6 service="SMTP" profile="sniffer-profile" action="log-only" from="jdoe@example.com" to="bob@example.org" cc="carol@example.org" sender="jdoe@example.com" recipient="bob@example.org" direction="outgoing" msg="general email log" subject="quarterly report" size="1048" attachment="no"

+ "email.cc.address": ["carol@example.org"],
- "email.to.address": "bob@example.org",
+ "email.to.address": ["bob@example.org"],
  "email.sender.address": ["jdoe@example.com"],
- "tls.server.issuer": "jdoe@example.com",

The RFC 5424 test line (test-fortinet-7-4.log, line 75):

<165>1 2022-11-14T06:54:03Z use2-dmz-fw02 - - - - eventtime=1668408842876531561 tz="-0800" logid="0317013312" type="utm" ...

- "@timestamp": <the time the event was read>,
+ "@timestamp": "2022-11-14T06:54:03.000Z",

The FortiExtender test line (test-fortinet-6-2.log line 27, and the same line in test-fortinet-7-4.log line 29):

... action="Cellular Signal Statistics" ... sinr="7.0 dB" rsrp="-89 dBm" rsrq="-16 dB" signalstrength="92 dBm" rssi="-54" temperature="40 C" ...

- "fortinet.firewall.rsrp": "-89 dBm",
+ "fortinet.firewall.rsrp": -89,
- "fortinet.firewall.rsrq": "-16 dB",
+ "fortinet.firewall.rsrq": -16,
- "fortinet.firewall.signalstrength": "92 dBm",
+ "fortinet.firewall.signalstrength": 92,
- "fortinet.firewall.sinr": "7.0 dB",
+ "fortinet.firewall.sinr": 7.0,
- "fortinet.firewall.temperature": "40 C",
+ "fortinet.firewall.temperature": 40,

How Log Shape found it

Log Shape, an internal tool I'm building (not public yet), runs a line through the package's own pipeline on Elasticsearch and shows which characters of the line each field came from and which processor moved them. Its coverage audit reports conditions that are never true (or never false) over the package's test lines and grok alternatives that never win, and indexes the resulting documents the way Fleet would.

It started with coverage: over the 214 log test lines on main, the audit flagged four of the five bugs.

• append_email_cc_address_4d235e7f: "the condition needs fortinet.cc.address, which no processor, script or agent template writes and no test config seeds; it was false on all 214 evaluations".
• utm.yml's append_email_sender_address_37ff254c and append_email_to_address_78c2982a: condition never true, because no test line carried sender= or recipient=. On a line it wrote by adding sender=, the audit reported "tls.server.issuer holds an email address (alice@example.com): that field never carries one".
• On test-fortinet-7-4.log line 75: "the line carries a timestamp, but @timestamp is the ingest time".
• "fortinet.firewall.rsrp = -89 dBm does not fit its mapping on 2 test line(s): kept in _source but indexed as _ignored, so no query can find it", and the same for rsrq (-16 dB), signalstrength (92 dBm), sinr (7.0 dB) and temperature (40 C).

The fifth, the string email.to.address, showed up when the new email filter test line ran. The quotes below are from a run on main's pipeline (b762a87). Grades: measured means Elasticsearch reported where the value sits in the line; located, Log Shape found it by searching its input field; traced, a processor path the run confirmed.

• The sender lands in tls.server.issuer. On the new email line, Log Shape placed jdoe@example.com (characters 467–483, the sender= value, measured) in email.sender.address, traced through append_email_sender_address_37ff254c, and in tls.server.issuer, traced through rename_fortinet_firewall_sender_to_tls_server_issuer_53552963.
• The recipient is renamed before the append can run. Log Shape placed the recipient= value bob@example.org (measured) in email.to.address, traced through rename_fortinet_firewall_recipient_to_email_to_address_e1ed6c6b. The next processor, append_email_to_address_78c2982a, was skipped.
• The cc address never reaches email.cc.address. The cc= value carol@example.org (measured) reached no field but fortinet.firewall.cc, and append_email_cc_address_4d235e7f was skipped. On this branch, the same run traces carol@example.org to email.cc.address through append_email_cc_address_4d235e7f, and bob@example.org to email.to.address through append_email_to_address_78c2982a.
• The RFC 5424 header timestamp feeds no field. In the header of line 75, Log Shape placed 165 (located) in log.syslog.priority and use2-dmz-fw02 (measured) in observer.name. It placed nothing on 2022-11-14T06:54:03Z (characters 7–27), which grok_event_original_995e7c66 matches as %{TIMESTAMP_ISO8601} without a field name. Both date processors that set @timestamp from _temp.time were skipped. On this branch, Log Shape places 2022-11-14T06:54:03Z (characters 7–27, measured) in @timestamp, traced through date__temp_syslog5424_ts_to_timestamp.
• The FortiExtender values reach the index with their units. On test-fortinet-6-2.log line 27, Log Shape placed -89 dBm (characters 429–436, measured) in fortinet.firewall.rsrp, traced, with no processor between the kv script and the index, against a declared integer. The same holds for sinr (7.0 dB), rsrq (-16 dB), signalstrength (92 dBm) and temperature (40 C).
• The review: three existing lines changed, as intended, and nothing regressed. Log Shape's base-versus-head review of this branch against main ran all 215 log test lines (214 existing, 1 added by this change). Failures: 0 → 0. Of the 214 existing lines, 211 are unchanged. The three that changed are the ones above: line 75 of test-fortinet-7-4.log gains @timestamp: 2022-11-14T06:54:03.000Z, and on the two FortiExtender lines the five fields become numbers. The header grok's winning patterns are the same on both sides. Nothing regressed.

The change

In default.yml / utm.yml, the five field-mapping fixes. Representative hunks:

-        - "^(?:%{SYSLOG5424PRI}%{NONNEGINT} )+(?:%{TIMESTAMP_ISO8601}|-) +…
+        - "^(?:%{SYSLOG5424PRI}%{NONNEGINT} )+(?:%{TIMESTAMP_ISO8601:_temp.syslog5424_ts}|-) +…
…
-      if: ctx.fortinet?.cc?.address != null
+      if: ctx.fortinet?.firewall?.cc != null
…
-  - rename: { field: fortinet.firewall.recipient, target_field: email.to.address }
   - append: { field: email.to.address, value: "{{{fortinet.firewall.recipient}}}", … }
+  - remove: { field: fortinet.firewall.recipient, … }
…
-  - rename: { field: fortinet.firewall.sender, target_field: tls.server.issuer }
+  - remove: { field: fortinet.firewall.sender, … }   # already copied to email.sender.address

The rest of the patch (tests, changelog, …) is on the Files tab.

Proposed commit message

[fortinet_fortigate] Fix email fields, RFC 5424 @timestamp and FortiExtender signal values

- email.cc.address was gated on ctx.fortinet?.cc?.address, which
  nothing writes; gate it on fortinet.firewall.cc, the field it reads.
- fortinet.firewall.sender was renamed to tls.server.issuer after being
  appended to email.sender.address; remove it instead.
- fortinet.firewall.recipient was renamed to email.to.address as a
  string, leaving the append after it unreachable; drop the rename so
  the append runs, then remove the raw field.
- RFC 5424 format logs carry no date/time fields and the header
  timestamp was not captured, so @timestamp was not set from the log;
  capture it and use it when date/time are absent.
- FortiExtender rsrp, rsrq, signalstrength, sinr and temperature carry
  units ("-89 dBm") but are mapped as numbers; strip the unit and
  convert, and map sinr as float.

What does this PR do?

Fixes the five bugs above in the log data stream (default.yml, utm.yml, fields.yml). They were found with the coverage audit and confirmed by running lines through the pipeline; the email.to.address one was exposed by the new email filter test line.

• CC: the condition now tests ctx.fortinet?.firewall?.cc, the field the value already reads.
• Sender: the rename to tls.server.issuer is replaced by a remove of fortinet.firewall.sender, which keeps the raw field's previous end state (it was not kept, and it is not declared in fields.yml). from= still goes to email.from.address, and email.sender.address is correct per ECS (the address responsible for transmitting the message). TLS issuers already come from scertissuer/issuer/ccertissuer, and I found no FortiOS log type that uses sender for a certificate issuer.
• Recipient: the rename is dropped so the existing append runs, followed by a remove of the raw field as before.
• RFC 5424: the header grok captures the timestamp into _temp.syslog5424_ts (removed with _temp at the end of the pipeline), and a new date processor (ISO8601) sets @timestamp from it only when _temp.time (from date/time) is absent. Logs that have date/time are unchanged.
• FortiExtender: a small script strips a trailing unit (letters and spaces) from the five fields and parses the number. A value that still does not parse is left unchanged, as before. sinr is reported with a decimal (7.0 dB), so its mapping changes from integer to float.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file (1.36.12).
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices. (no dashboards changed)

How to test this PR locally

cd packages/fortinet_fortigate
elastic-package test pipeline

The email filter line under "One line, two documents" is added to test-fortinet-7-4.log. Its expected document has email.cc.address: ["carol@example.org"], email.sender.address: ["jdoe@example.com"], email.to.address: ["bob@example.org"] and no tls object.

Changes to existing expected documents (checked field by field; nothing else moved):

file	doc	field	before	after
test-fortinet-7-4.log-expected.json	line 75 (RFC 5424)	@timestamp	absent	2022-11-14T06:54:03.000Z
test-fortinet-6-2.log-expected.json line 27, test-fortinet-7-4.log-expected.json line 29	FortiExtender	fortinet.firewall.rsrp	"-89 dBm"	-89
		fortinet.firewall.rsrq	"-16 dB"	-16
		fortinet.firewall.signalstrength	"92 dBm"	92
		fortinet.firewall.sinr	"7.0 dB"	7.0
		fortinet.firewall.temperature	"40 C"	40

The RFC 5424 @timestamp agrees with event.start from eventtime (2022-11-14T06:54:02Z) to within the second.

The base-versus-head review above agrees: on the added line +email.cc.address, -tls.server.issuer and email.to.address as an array; on existing lines, only the changes in this table. elastic-package check passes (with a numeric PR link in the changelog).

Related issues

None filed; found with a pipeline coverage audit.

Data flow and coverage

How the data flows (before → after)

Each diagram follows one of this change's own test lines through the pipeline on main and on this branch: the processors that touched the values the fix is about, each value as it stood after that step, and what was indexed. Red: where main goes wrong; green: the fixed outcome; heavy blue border: a processor this change adds or edits.

fortinet_fortigate/log · test-fortinet-7-4.log#82

flowchart LR
  L["…carol@example.org#quot; sender=#quot;jdoe@example.com#quot; recipient=#quot;bob@example.org…"]
  subgraph sa["After: this change"]
    direction LR
    a0_0["script script_2073fe5f: fortinet.firewall.recipient = #quot;bob@example.org#quot;"]
    a0_2["remove remove_fortinet_firewall_recipient: fortinet.firewall.recipient"]
    a0_3["indexed email.to.address = #91;#quot;bob@example.org#quot;#93;"]
    a0_0 --> a0_2 --> a0_3
    a1_0["script script_2073fe5f: fortinet.firewall.cc = #quot;carol@example.org#quot;"]
    a1_1["append append_email_cc_address_4d235e7f: email.cc.address = #quot;carol@example.org#quot;"]
    a1_2["indexed email.cc.address = #91;#quot;carol@example.org#quot;#93;"]
    a1_0 --> a1_1 --> a1_2
  end
  subgraph sb["Before: main"]
    direction LR
    b0_0["script script_2073fe5f: fortinet.firewall.recipient = #quot;bob@example.org#quot;"]
    b0_1["rename rename_fortinet_firewall_recipient_to_email_to_address_e1ed6c6b: email.to.address = #quot;bob@example.org#quot;"]
    b0_2["indexed email.to.address = #quot;bob@example.org#quot;"]
    b0_0 --> b0_1 --> b0_2
    b1_0["script script_2073fe5f: fortinet.firewall.cc = #quot;carol@example.org#quot;"]
    b1_1["append append_email_cc_address_4d235e7f: email.cc.address"]
    b1_2["email.cc.address: not indexed"]
    b1_0 --> b1_1 --> b1_2
  end
  L --> b0_0
  L --> b1_0
  L --> a0_0
  L --> a1_0
  classDef bad fill:#fde2e2,stroke:#c62828,color:#7f1d1d
  classDef good fill:#e3f6e5,stroke:#2e7d32,color:#14532d
  classDef fix stroke:#1565c0,stroke-width:3px
  class b0_2,b1_2 bad
  class a0_3,a1_2 good
  class b0_1,b1_1,a0_2,a1_1 fix

Loading

Test coverage of the change

The changed pipeline region, with how the package's pipeline tests exercise each line on main → on this branch (# coverage: lines; counts are test lines).

packages/fortinet_fortigate/data_stream/log/elasticsearch/ingest_pipeline/default.yml

      if: ctx._temp?.time != null && ctx.event?.timezone == null
  # coverage: new — ran 1×
  - date:
      tag: date__temp_syslog5424_ts_to_timestamp
# …
        - ISO8601
      # coverage: new — if true 1× / false 214×; operand arms taken 4/4
      if: ctx._temp?.time == null && ctx._temp?.syslog5424_ts != null
  - gsub:
# …
      value: "{{{fortinet.firewall.cc}}}"
      # coverage: if true never / false 214× → if true 1× / false 214×
      if: ctx.fortinet?.firewall?.cc != null
  - set:
# …
      ignore_missing: true
  # coverage: new — ran 215×
  - script:
      tag: script_fortiextender_signal_units
# …
      lang: painless
      # coverage: new — if true 215× / false never
      if: ctx.fortinet?.firewall instanceof Map
      params:
# …
      source: |
        # coverage: new — statement 215×
        def fw = ctx.fortinet.firewall;
        # coverage: new — loop arms taken 1/2; statement 215×
        for (def name : params.fields) {
          # coverage: new — statement 1075×
          def v = fw[name];
          # coverage: new — if arms taken 2/2; statement 1075×
          if (!(v instanceof String)) {
            # coverage: new — statement 1065×
            continue;
          }
          # coverage: new — statement 10×
          String s = v.trim();
          # coverage: new — statement 10×
          int end = s.length();
          # coverage: new — loop arms taken 1/2; operand arms taken 7/8; statement 10×
          while (end > 0 && (Character.isLetter(s.charAt(end - 1)) || s.charAt(end - 1) == (char)' ')) {
            # coverage: new — statement 32×
            end--;
# … and 26 more changed lines (see the core diff (Files tab))

packages/fortinet_fortigate/data_stream/log/elasticsearch/ingest_pipeline/utm.yml

      if: ctx.fortinet?.firewall?.recipient != null
  # coverage: new — ran 72×
  - remove:
      tag: remove_fortinet_firewall_recipient
      field: fortinet.firewall.recipient
      ignore_missing: true
  - rename:
# …
      if: ctx.tls?.client?.issuer != null
  # coverage: ran 71× → ran 72×
  - remove:
      tag: remove_fortinet_firewall_sender
      description: The email sender was already copied to email.sender.address; it is not a TLS certificate issuer.
      field: fortinet.firewall.sender

[github-actions]

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

---

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

[elastic-vault-github-plugin-prod]

✅ All changelog entries have the correct PR link.

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[infra-vault-gh-plugin-prod]

💚 Build Succeeded

• Buildkite Build
• Commit: d923cfc

[Copilot]

🟢 Approval recommended

No active review issues remain for the FortiGate email, timestamp, and cellular-statistics mapping fixes.

No open issues 🎉

What changed in this PR

This PR fixes five field-mapping issues in the Fortinet FortiGate log data stream, correcting email fields, event timestamps, and numeric cellular statistics.

Changes:

• Corrects CC and recipient address arrays and prevents email senders from populating TLS issuer fields.
• Uses RFC 5424 header timestamps when log date/time fields are absent and converts unit-bearing FortiExtender statistics to numbers.
• Updates regression fixtures, documents the sinr float mapping, and bumps the package to 1.36.12.

File	Description
packages/​fortinet_fortigate/​manifest.yml	Bumps the package version to 1.36.12.
packages/​fortinet_fortigate/​docs/​README.md	Documents sinr as a float.
packages/​fortinet_fortigate/​data_stream/​log/​fields/​fields.yml	Changes the sinr mapping to float.
packages/​fortinet_fortigate/​data_stream/​log/​elasticsearch/​ingest_pipeline/​utm.yml	Fixes recipient arrays and removes the incorrect sender-to-TLS mapping.
packages/​fortinet_fortigate/​data_stream/​log/​elasticsearch/​ingest_pipeline/​default.yml	Fixes CC mapping, captures RFC 5424 timestamps, and converts cellular statistics.
packages/​fortinet_fortigate/​data_stream/​log/​_dev/​test/​pipeline/​test-fortinet-7-4.log-expected.json	Updates expected email fields, timestamps, and numeric statistics.
packages/​fortinet_fortigate/​data_stream/​log/​_dev/​test/​pipeline/​test-fortinet-7-4.log	Adds an email-filter regression fixture.
packages/​fortinet_fortigate/​data_stream/​log/​_dev/​test/​pipeline/​test-fortinet-6-2.log-expected.json	Expects numeric FortiExtender statistics.
packages/​fortinet_fortigate/​changelog.yml	Records the email, timestamp, and cellular-statistics fixes.

_{Review effort: Balanced}