[shmsr]

What does this PR do?

Adds a semantic rule, SVR00014, that checks the structure of Alerting v2 rule
templates in kibana/alerting_rule_template/. A template is Alerting v2 when it sets
"engine": "v2".

For v2 templates, the rule checks:

• rule.kind, rule.metadata (name, description, tags, routing_tags, builder),
  rule.schedule, rule.time_field, and rule.query (base and breach.segment), with
  Kibana's length and count limits.
• The alert lifecycle. Alert rules must set recovery and no_data. Signal rules must not
  set recovery, no_data, state_transition, or metadata.routing_tags.
• state_transition counts and timeframes, grouping.fields (1 to 16), and artifacts
  (1 to 100).

Blank checks use JavaScript's trim() whitespace, string lengths count UTF-16 code units,
and artifact data size is JSON.stringify(data).length.

Which templates are checked:

• Classic templates are skipped by this rule. A template is classic when engine is missing,
  null, not a string, "", or v1. Existing Kibana object checks, including ID and
  reference checks, still apply.
• Unsupported string engine values, such as V2 or v3, produce errors without a code
  on every format_version, so they cannot be excluded in validation.yml.
• A file that is not valid JSON is not reported here, because the Kibana object checks
  already report it.

What the rule does not check:

• ES|QL syntax. That needs Kibana's ES|QL parser, so queries still need testing against a stack.
• Unknown fields. Kibana rejects them, but this rule allows them so that a field added in a
  later Kibana does not fail an older package-spec.

The checks follow Kibana's
alerting-v2-schemas.

Which format versions

flowchart TD
  F["kibana/alerting_rule_template/*.json"] --> J{"Valid JSON?"}
  J -->|no| P["Existing Kibana object checks report the error<br/>cannot be excluded"]
  J -->|yes| E{"attributes.engine"}
  E -->|"missing, null, non-string, empty, v1"| C["Skipped by the v2 validator<br/>existing Kibana object checks still apply"]
  E -->|"unsupported string, such as V2 or v3"| U["Unsupported engine<br/>error without a code on every format_version<br/>cannot be excluded"]
  E -->|v2| V{"format_version >= 3.7.0?"}
  V -->|no| M["Requires format_version 3.7.0 or later<br/>error without a code<br/>cannot be excluded"]
  V -->|yes| S["SVR00014: v2 structural validation<br/>can be excluded in validation.yml"]

Loading

The checks run for integration and content packages in every validation mode.
Only v2 structural errors from format_version 3.7.0 or later receive SVR00014 and
can be excluded in validation.yml. Unsupported-engine errors have no code on any
format version. V2 templates below 3.7.0 also produce errors without a code. Neither
can be excluded. Malformed JSON is reported by the existing Kibana object checks
with errors that cannot be excluded.

Kibana 9.6.0 is the first version that installs v2 templates and the first that supports
format_version 3.7.0. The registry only offers a package to a Kibana that supports its
format_version, so tying v2 templates to 3.7.0 keeps them away from Kibana 9.4 and 9.5.
Those versions fail the install with
[attributes.name]: expected value of type [string] but got [undefined]:

flowchart LR
  P36["format_version 3.6.0<br/>with a v2 template"] --> REG[Package registry]
  P37["format_version 3.7.0<br/>with a v2 template"] --> REG
  REG -->|"spec.max 3.6: offers only 3.6.0"| K95["Kibana 9.4 / 9.5"]
  REG -->|"spec.max 3.7: offers both"| K96["Kibana 9.6+"]
  K95 --> FAIL["Install fails:<br/>attributes.name expected string"]
  K96 --> OK[Installs]

Loading

Why is it important?

Kibana does not validate the rule body of a v2 template when Fleet installs a package. It
only validates it when a user creates a rule from the template. Without this check, a broken
template ships and installs cleanly, and the first person to see the error is the user.

Companion to elastic/elastic-package#4054,
which removes the equivalent local validator from elastic-package in favor of this
central check.

Validation

• go test -count=1 ./code/go/... and make check passed.
• Regression tests confirm that excluding SVR00014 removes v2 structural errors but
  keeps unsupported-engine and malformed-JSON errors. These cover legacy, source,
  and build modes, plus zip packages.
• A format-version regression test confirms that the minimum-format errors cannot
  be excluded either.

Checklist

• I have added test packages to test/packages that prove my change is effective. good_alert_rule_templates_v2 (valid alert, signal, and classic templates) and bad_alert_rule_templates_v2 (one invalid template per failure), both at format_version: 3.7.0. A validator test sets good_alert_rule_templates_v2 to 3.6.0 and expects one error per v2 template.
• I have added an entry in spec/changelog.yml, under 3.7.1-next.

Related issues

• Relates elastic/elastic-package#4054

[teresaromero]

Thanks!!. Details are in the inline comments; the main points:

• Rollout: the rule has no since/warnOn, so it is a hard error for every format_version, including published packages, on a patch release. Is that intended? Could we gate it or start as a warning? Once CI is green, please comment test integrations to see the effect on real packages.
• SVR00014 is also used by #1250; whichever merges second needs to renumber.
• Tests: add a good and a consolidated bad v2 fixture package, registered in Test_ValidateFromPath.
• Code: report ReadFile errors separately from validation errors, and replace templateProblems with helpers returning error combined via errors.Join, with parsing and the engine skip in their own helper.
• Kibana reference: please link Kibana's alerting-v2-schemas with permalinks, and confirm whether accepting unknown fields is intended, since Kibana's schema is .strict().
• Smaller: unknown engine handling and docs, plus a few nits.

I only compared the limits with Kibana's v2 schema, not every rule.

We can trigger a test integrations comment so we see the impact of this changes on the current integrations.

[shmsr]

test integrations

[elastic-vault-github-plugin-prod]

Created or updated PR in integrations repository to test this version. Check elastic/integrations#22140

[shmsr]

test integrations

[elastic-vault-github-plugin-prod]

Created or updated PR in integrations repository to test this version. Check elastic/integrations#22140

[mrodm]

test integrations

Created or updated PR in integrations repository to test this version. Check elastic/integrations#22140

@shmsr it looks like that there are conflicts in this integrations PR, and it could not be validated the changes in that CI pipeline.

The easiest way to solve this, is to close that PR and delete the branch, and then post again here the test integrations comment.

[shmsr]

test integrations

[elastic-vault-github-plugin-prod]

Created or updated PR in integrations repository to test this version. Check elastic/integrations#22217

[teresaromero]

Thanks for this work, i think this looks good from the ecosystem perspective! 🚀

integration is still running but by this point i think this is fully compatible.

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[mrodm]

I'd set spec 3.7.0 , because 9.6.0-SNAPSHOT is the kibana supporting spec 3.7.

Moreover, I would remove the warning before 3.7 and make and error because as we found in the elastic-package PR those package cannot be installed in < 9.6 kibana versions.

Suggested change

	{fn: warnOn(semantic.ValidateAlertRuleTemplates), types: []string{"integration", "content"}, until: semver.MustParse("3.8.0")},
	{fn: semantic.ValidateAlertRuleTemplates, types: []string{"integration", "content"}, since: semver.MustParse("3.8.0")},
	{fn: semantic.ValidateAlertRuleTemplates, types: []string{"integration", "content"}, since: semver.MustParse("3.7.0")},

[shmsr]

Done in 081d451 and 2812f13. SVR00014 is now an error from format_version 3.7.0, with no warning period, and the changelog entry is under 3.7.1-next.

I also added a second rule for packages below 3.7.0. Any engine: v2 template, or any engine other than "" or v1, is an error there. It has no code, so validation.yml can't hide it. Without it, a 3.6.0 package could still ship a v2 template, and the registry would offer it to Kibana 9.4 and 9.5, which fail to install it:

flowchart LR
  P36["format_version 3.6.0<br/>with a v2 template"] --> REG[Package registry]
  P37["format_version 3.7.0<br/>with a v2 template"] --> REG
  REG -->|"spec.max 3.6: offers only 3.6.0"| K95["Kibana 9.4 / 9.5"]
  REG -->|"spec.max 3.7: offers both"| K96["Kibana 9.6+"]
  K95 --> FAIL["Install fails:<br/>attributes.name expected string"]
  K96 --> OK[Installs]

Loading

Both v2 test packages now use format_version: 3.7.0 and kibana.version: ^9.6.0. I've also updated the PR description, which still described the rule as ungated.

After discussing offline with Mario i think we need still some changes. I was not aware of the compatbility issue between versions. Check his comments pls

[shmsr]

test integrations

[elastic-vault-github-plugin-prod]

Created or updated PR in integrations repository to test this version. Check elastic/integrations#22217

[shmsr]

The test integrations run on 2812f13 (elastic/integrations#22217; (my bad that I forgot to check the format version here when I reviewed the PR)) found one failing package, gcp_cloudrun_otel.

It got five engine: v2 templates in elastic/integrations#22065, which merged yesterday, but it's still on format_version: "3.6.2". So the new rule flags each template:

Alerting v2 rule templates (engine v2) require format_version 3.7.0 or later

So I was wrong earlier when I said no published package ships v2 templates. That was true when I wrote it, but not anymore. The earlier run only passed because ab2114e still treated this as a warning.

The templates themselves look good. I tried the package locally with this branch: it fails at 3.6.2, and at 3.7.0 it builds and passes SVR00014.

I'd keep the rule as it is. This is the case @mrodm brought up, and the package already requires Kibana ^9.6.0, so moving it to 3.7.0 doesn't change who can install it. The published 0.4.0 can stay as it is. The next version just needs format_version: "3.7.0", and that already works with the elastic-package integrations uses today. It has to land before integrations picks up a package-spec release with this rule; so I can take this up? Does that sound ok?

@teresaromero @mrodm what do you think? or would you rather wait to release until the package is updated?

[mrodm]

LGTM the changes to use spec 3.7.0 for alert rule template v2. Thanks for these changes.

In this last review, just wondering about some errors that could be skipped and I'm not sure if they should be part of those. Could you check it ?

I'd keep the rule as it is. This is the case @mrodm brought up, and the package already requires Kibana ^9.6.0, so moving it to 3.7.0 doesn't change who can install it. The published 0.4.0 can stay as it is. The next version just needs format_version: "3.7.0", and that already works with the elastic-package integrations uses today. It has to land before integrations picks up a package-spec release with this rule; so I can take this up? Does that sound ok?

@teresaromero @mrodm what do you think? or would you rather wait to release until the package is updated?

Until the next elastic-package release containing this package-spec changes (in the next package-spec release), this gcp_cloudrun_otel package could not be updated to use format_version: 3.7.0. I think that package and if any other package (but I hope no other packages 😅 🤞 ) should be updated once elastic-package version is updated in the integrations repository. Until that, CI would fail when the package tries to update the format_version with an error of invalid spec or similar.

[infra-vault-gh-plugin-prod]

💚 Build Succeeded

• Buildkite Build
• Commit: f349d05

History

• 💚 Build #2511 succeeded 89b0acd
• 💚 Build #2507 succeeded 2812f13
• 💚 Build #2505 succeeded ab2114e
• 💚 Build #2501 succeeded bcb64f6
• 💚 Build #2453 succeeded 2214c16
• 💚 Build #2452 succeeded 74a8e3f

cc @shmsr

[teresaromero]

🚀 thanks for the latest changes!!

[mrodm]

Thanks @shmsr !