Conversation
The `index_pivot` transform reads from `.monitoring-es-*` and `metrics-elasticsearch.stack_monitoring.index*`. On deployments that extend the monitoring ILM policy beyond the default retention, those indices eventually age into the frozen tier, where they are backed by partially mounted searchable snapshots. The transform then stalls: it keeps running without erroring but stops advancing its checkpoint. Add a `must_not` clause on the `_tier` metadata field so the transform only reads data that is still on local storage. Users who want to backfill older monitoring data can still do so with a manual copy of the transform that drops this clause. Bumps `fleet_transform_version` to 0.4.0 so Fleet reinstalls the transform. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
✅ Elastic Docs Style Checker (Vale)No issues found on modified lines! The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale. |
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every other transform in this repository that filters on `_tier` excludes both `data_cold` and `data_frozen`. The cold tier is also backed by searchable snapshots, and this transform never legitimately needs data that old: it builds 10-minute buckets with a 60s sync delay, so anything that has aged out of hot/warm was already processed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
3kt
changed the title
Contributor
🚀 Benchmarks reportTo see the full report comment with |
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
The remaining documentation update is a minor, non-blocking nit.
Pull request overview
Updates Elasticsearch’s index_pivot transform to exclude cold and frozen tiers, preventing checkpoint stalls.
Changes:
- Excludes
data_coldanddata_frozendocuments. - Bumps transform version to
0.4.0. - Bumps package version to
1.22.1and adds a bugfix changelog entry.
File summaries
| File | Description |
|---|---|
packages/elasticsearch/manifest.yml |
Bumps package version. |
packages/elasticsearch/elasticsearch/transform/index_pivot/transform.yml |
Adds tier exclusions and updates the transform version. |
packages/elasticsearch/changelog.yml |
Documents the fix. |
Review details
Suppressed comments (1)
packages/elasticsearch/elasticsearch/transform/index_pivot/transform.yml:44
- This changes the transform's historical coverage, but the user-facing documentation still says the job "will process all compatible historical data" (
_dev/build/docs/README.md:201, rendered indocs/README.md:2932). With thismust_not, cold/frozen monitoring documents are intentionally skipped, so update the documentation (and regenerate the rendered README) to describe the hot/warm-only behavior and the manual backfill option.
must_not:
- terms:
_tier: ["data_cold", "data_frozen"]
- Files reviewed: 3/3 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Update the public documentation to describe the new retention boundary and manual backfill path.
Get a fresh assessment by requesting another Copilot review.
Review details
- Files reviewed: 3/3 changed files
- Comments generated: 1
- Review effort level: Lite
Comment on lines
+42
to
+44
| must_not: | ||
| - terms: | ||
| _tier: ["data_cold", "data_frozen"] |
Contributor
|
✅ All changelog entries have the correct PR link. |
💚 Build Succeeded
History
|
Contributor
|
Tick the box to add this pull request to the merge queue (same as
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The
index_pivottransform reads from.monitoring-es-*,metrics-elasticsearch.stack_monitoring.index*andmetricbeat-*. On deployments that extend the monitoring ILM policy (e.g. hot for 7 days, then frozen until deletion at 90 days), those indices age onto searchable snapshots and the transform gets stuck: it does not error, it just stops advancing its checkpoint, and stays stuck indefinitely.It never needs that data anyway: 10-minute buckets with a 60s sync delay, so anything aged out of hot/warm was already processed.
Change
Same clause every other
_tier-filtering transform in this repo uses (30 of them across 24 packages, e.g.wiz,crowdstrike,beaconing,cloud_security_posture,aws).fleet_transform_version0.3.0->0.4.0so Fleet reinstalls the transform; package patch bump to1.22.1.Users who want the older data in
monitoring-indicescan clone the transform, drop this clause and run it as a batch transform.Testing
Self-monitored 9.4.1 stack (Elasticsearch, Kibana, Fleet Server, Elastic Agent) via
elastic-package, this build installed through Fleet, integration pointed at the cluster itself. Workload: 6 indices + 3 data streams, ~50k docs and continuous search traffic over 25 minutes.logs-elasticsearch.index_pivot-default-0.4.0, with the_tierclause present in the deployed query.health: green, 6650 docs processed, 156 indexed, 0 search/index failures.monitoring-indicesfilled for all demo indices and data stream backing indices, with populated deltas,tier,creation_dateandage; system-index exclusions unchanged.0.3.0) and new queries return an identical hit count against the same live monitoring data, so the clause is a no-op without cold/frozen data._validate/queryreturnsvalid: true; counter-test on the tier the test index actually lives on drops hits to 0, confirming the filter is applied.elastic-package lintpasses.