Repository navigation
Add Facility for deploying ElasticSearch Transform #23
Description
Background
And the ability to deploy a defined ElasticSearch transform to be be deployed when a package is applied or upgraded.
Acceptance Criteria
- As a user I should be able to define an ElasticSearch transform as part of a package.
- The ElasticSearch transform should be started after it is added to the search database.
- As a user I should be able to update the attributes of the transform including the name possibly. This should not result in two Transform running.
- As a user we should be able to delete a transform through an update to the package.
- As a user I should be able to view statistics and information about a transform after deployment using the ElasticSearch API or Kibana if available.
Sample Transform Creation Statements Captured From Kibana Devtools
PUT _transform/endpoint_host_metadata_transform
{
"source": {
"index": "metrics-endpoint.metadata-default"
},
"dest": {
"index": "metrics-endpoint.metadata_current-default"
},
"pivot": {
"group_by": {
"agent.id": {
"terms": {
"field": "agent.id"
}
}
},
"aggregations": {
"HostDetails": {
"scripted_metric": {
"init_script": "state.timestamp_latest = 0L; state.last_doc=''",
"map_script": "def current_date = doc['@timestamp'].getValue().toInstant().toEpochMilli(); if (current_date > state.timestamp_latest) {state.timestamp_latest = current_date;state.last_doc = new HashMap(params['_source']);}",
"combine_script": "return state",
"reduce_script": "def last_doc = '';def timestamp_latest = 0L; for (s in states) {if (s.timestamp_latest > (timestamp_latest)) {timestamp_latest = s.timestamp_latest; last_doc = s.last_doc;}} return last_doc"
}
}
}
},
"description": "collapse and update the latest document for each host",
"frequency": "1m",
"sync": {
"time": {
"field": "event.created",
"delay": "60s"
}
}
}
POST _transform/endpoint_host_metadata_transform/_start
DELETE _transform/endpoint_host_metadata_transform
Activity
ycombinator commentedon Aug 13, 2020
@ruflin could use your input here.
This item impacts the package spec. We could enhance the spec to allow transforms to be optionally defined as
<packageRoot>/dataset/<dataset>/elasticsearch/transform/<transform>.jsonfiles. So your sample transform above would be defined inendpoint/dataset/metadata_current/elasticsearch/transform/endpoint_host_metadata_transform.json. The contents of the JSON file would be the same JSON that you'd send to thePUT _transformElasticsearch API. The only part I'm not 100% sure about is whether the JSON should hardcode the values of thesource.indexanddest.indexproperties or if those need to be dynamically generated by Kibana before loading the transform into Elasticsearch?These items impact the actual implementation, which would happen in Kibana. Once we agree on the spec changes (discussion in the previous paragraph), I'll make a PR to update the spec in this repo here and then we can file an issue in Kibana for the implementation items.
ruflin commentedon Aug 14, 2020
It seems the transform depends on one dataset to create a second one. The index
metrics-endpoint.metadata-defaultwould be part of the existingmetadatadataset. The transform part would be its own dataset and most follow the indexing strategy. Withmetrics-endpoint.metadata_current-defaultthis seems to be the case, so the dataset would be calledmetadata_current. This dataset also contains the mapping for the data after the transformation.So what
<packageRoot>/dataset/<dataset>/elasticsearch/transform/<transform>.jsonmakes sense to me. @ycombinator Should we support json and yaml? ;-)But what about the namespace part? Will the transform be applied to all namespaces? Can the namespace somehow be passed in as a param on
start?@nnamdifrankie Could you open a second issue in Kibana around the part installing / removing / updating the transforms as this not directly applies to the spec.
nnamdifrankie commentedon Aug 14, 2020
@ruflin What repository for the kibana ticket. And to answer your question about the metadata_current we have already defined it and it is managed by ingest. https://github.com/elastic/package-storage/pull/325/files#diff-fd155145fe68664d8a5cbd8b7727ac5f
ruflin commentedon Aug 17, 2020
@nnamdifrankie https://github.com/elastic/kibana
Could you also share your thoughts on how you would see it working with the namespaces?
nnamdifrankie commentedon Aug 17, 2020
This is a good I will answer and ask some question on how the indices are created.
Answer:
We currently write to the default namespace and read from all namespaces
-*, with our index pattern and we do not specify any namespaces. So to begin we will be successful if we target the default namespace.Questions:
metrics-endpoint.metadata_currentwill this create an indexmetrics-endpoint.metadata_current-defaultruflin commentedon Aug 18, 2020
Related Kibana issue can be found here: elastic/kibana#75153
ruflin commentedon Aug 18, 2020
Indices are not created by default, there is only the template. The index (data stream) is created when the first document is ingested.
polyfractal commentedon Aug 18, 2020
Side note: it looks like the scripted_metric agg is trying to capture the "most recent" document for each host? If that's true, I think it can be done with a TopMetrics agg instead
We're trying to discourage the use of scripted-metric... it can be very dangerous, and the top-metrics agg should be faster too. :)
nnamdifrankie commentedon Aug 31, 2020
@polyfractal
I know we spoke offline on slack at the time of your comment. But I wanted to comment here for posterity. We are not trying to gather specific metrics but we are trying to retrieve the latest document related to the pivot property, and replace the last document in the destination. Hence we are using this approach, but if there are other better approaches please comment. Thanks.
polyfractal commentedon Aug 31, 2020
If you need the latest document, perhaps TopHits agg instead?
mtojek commentedon Nov 19, 2021
It's a bit old issue. Feel free to reopen it if it's still valid.
ruflin commentedon Nov 19, 2021
This actually got implemented. We are now in the progress in figuring out how we can improve it.
mtojek commentedon Mar 7, 2022
Reopening this issue as we noticed that transforms and index templates haven't been formalized with package-spec, which mean they're "illegal".
@pzl @nnamdifrankie Could you please add missing definitions to spec?
joshdover commentedon Mar 8, 2022
For context, back in elastic/kibana#75153 support was added to Kibana to install transforms and the required associated index template for a transform's destination index. Endpoint is currently the only package to be using these today, however no associated package-spec changes were ever merged for transforms or their associated index templates
mtojek commentedon Mar 8, 2022
ping @kevinlog
CohenIdo commentedon Mar 9, 2022
Hey @mtojek, another point that is related to this point is when we perform
elastic-package checkafter adding the mentioned assets, we receive the following linting error:Another point I would like to raise after conversation with @joshdover is:
Adding assets to create the cloud security posture solution led us to discover that in some cases we would like to create a dependency between assets.
As an example, I created two Transform assets. (
transform1andtransform2)The source index of
transform2is the destination index oftransform1.When the assets were initialized, we got an error saying the source index of
transform2didn't exist.As a workaround, I created the destination index of
transform1before initializing the assets.As a user, I'd like to be able to tell the integration that
transform2depends ontransform1.cc: @kfirpeled
kfirpeled commentedon Mar 22, 2022
Hey all, I'd like to raise several points to consider here, regarding
latesttransform and the transform destination convention.In this discussion @joshdover brought to our attention that our Transform destination index is not following the data stream naming scheme and the importance of following it to the user.
Since
latestTransform can't use a datastream as destination, and that I had the impression the naming convention is reserved to datastreams only.I wonder if we should introduce a hint, in the naming convention, that the current index is not a datastream or that this is a latest Transform kind of index. Or maybe that is me, misunderstanding how it works.
The second question is, when installing package's Transforms, how to make them aware to kibana namespaces. How to create a transform that the source of it is a datastream? which is kibana namespace aware. And how to create a datastream that the destination of it is kibana namespace aware?
kevinlog commentedon Mar 22, 2022
@kfirpeled
The issue has some relevant discussion about transforms and space awareness. We're currently discussing options with the ML team on how to solve this. There's nothing concrete yet, but you can follow along. @joshdover latest comment here has some ideas.
joshdover commentedon Jul 12, 2022
closed by #307