Allow integration authors to deprecate integrations and data streams

Summary

Integration authors want to deprecate packages or data streams as they are no longer relevant, or when they are superseded by new options.

Fleet and integrations should support deprecated packages and data streams, and provide a clear and consistent UX around deprecated configurations in integrations that steers users to migrate to new and improved functionality.

Implementation plan

See sub-issues for implementation plan

Original description

With 8.x.x we may want to deprecate some package or data streams and create new alternatives. We need to discuss many things around, including publishing and installation rules of deprecated policies and "deprecated" captions. content

There is a similar thread started here: #192 , but I guess we need a more formal design (nothing adhoc).

It'd be nice to have some metadata for the deprecation, so maybe the UIs can provide a better experience to users. Without explicit metadata it may be difficult for Kibana to show something meaningful to users unless they explicitly check the titles or descriptions package by package.

If explicit metadata exists, Kibana may warn the user about needed actions as soon as it discovers that an installed package is deprecated.

For example having something like this in the manifest could indicate the user what to do, even providing links to the new packages or even eventually providing automatic migrations.

deprecation:
  reason: replaced
  comment: "The package has been split into separate packages: Microsoft DHCP and Microsoft Defender Endpoint."
  replacedBy:
    - microsoft_dhcp
    - microsoft_defender_endpoint

If done at the data stream level, and assuming a compatible configuration, it might be easier to provide an automated migration path. With the following information at some point kibana could know how to automatically upgrade from the old microsoft.dhcp datastream to the new microsoft_dhcp.log:

deprecation:
  reason: replaced
  comment: "This data stream has been moved to a new package: Microsoft DHCP"
  replacedBy:
    package: microsoft_dhcp
    data_stream: log

For packages (or data streams) that are just discontinued:

deprecation:
  reason: discontinued
  comment: "This package has been discontinued, it won't receive further updates and may stop working in the future."

I like this approach. It's consistent with the "skip" structure for system tests: https://github.com/elastic/integrations/blob/153945daf72d6ef0f6fce53346daf0e4ef9b1753/packages/aws/data_stream/ec2_metrics/_dev/test/system/test-default-config.yml#L1

Could we also add a new reason like "Going to be deprecated soon" if it is linked to a product that will soon be EOL for example?

@jlind23

Could we also add a new reason like "Going to be deprecated soon" if it is linked to a product that will soon be EOL for example?

Umm, thinking about this, I think that we should consider a package or data stream marked as deprecated as "Going to be unsupported and/or unmaintained soon because of reason". The reason should help the user (and the UI or other tooling) to know what to expect and/or what to do.
Adding another level of "going to" may be not neccesary, it would also mean that we need to communicate two things: when the package is going to be deprecated and when the package is actually deprecated, it can be confusing.

If the deprecation is caused because the monitored product is going to be EOL, we can use the discontinued reason and use the comment to explain the situation:

deprecation:
  reason: discontinued
  comment: "_The product_ will be EOL in _some date_ and this package won't be further maintained"

There may be the case where a package is deprecated because is going to be reimplemented, but the replacement doesn't exist yet, or there is no automatic migration path, in that case I think that it would be ok to don't declare a replacement, just:

deprecation:
  reason: replaced
  comment: "This data stream is going to be reimplemented"

This could be understood as: "the data stream is going to be removed, a new implementation will be provided but it is not ready yet". A message like this could be literally shown to a new user as a warning when configuring a policy with this data stream.
Later on a new version could update this information when the replacement is more clear.

@jlind23 would these definitions cover your use case?

@jsoriano

@jsoriano perfectly indeed! Another question that we should ask to @akshay-saraswat:

Shall we let inside the registry packages that are over deprecation date?
Work on some kind of clean up policy. wdyt?

Shall we let inside the registry packages that are over deprecation date?
Work on some kind of clean up policy. wdyt?

I would leave this for a future discussion. In general I think that it is not nice to remove published versions (apart of removing harmful or illegal content), but we may consider a clean up policy for practical reasons.

One core feature related to this needed in Fleet is that even if a package is removed from the registry, Kibana will still keep working assuming the package was installed before. The package cannot be upgraded anymore or reinstalled, but things keep working. Ideally Fleet would know from which registry it was installed and can indicate, that the package does not exists anymore in the remote registry.

As a follow up, we likely have to expand the deprecation discussion to more granular parts which could be deprecated:

integration policy
data stream
field
?

Each likely needs its own story on how it can be deprecated. Ideally we provide framework and guideline for each. Especially around fields discussions are already happening in various places.

As an integration policy, do you mean a policy template?

In terms of spec changes, it should be relatively easy.

Changes in data streams and packages can be quick wins, as they both contain manifests. We can augment these files with a special "deprecation" structure.
Theoretically, we can introduce a similar flag to "field" (like we do in Beats).

I don't have any preference regarding Kibana/Fleet behavior. I guess we could redirect users of one package to another? or just suggest it to them?

@mtojek

@mtojek Yes, policy templates. Should we discuss each of these topics in a separate issue as some might need work in Fleet, others don't and we can close one by one.

@mtojek

@mtojek Yes, policy templates. Should we discuss each of these topics in a separate issue as some might need work in Fleet, others don't and we can close one by one.

@ruflin what would be the use case of deprecating policy templates without deprecating the data_stream that contains it?

Hi there, our team may have a use case for deprecated inputs.

The packages for the stack components (elasticsearch/kibana/logstash) currently have two inputs, one for metrics collection via a metricbeat module and one for logs. The metricbeat collection will eventually be superseded by a new approach that uses different metrics schema, assets and collection method, while the logs collection remains the same. When the new data streams are ready the package would allow a transition period where both metrics collection are available, with the metricbeat one marked as deprecated.
Once the deprecation period is done and we ship a new version that removes the input, it'd be nice to surface that in the UI when trying to upgrade to that version (maybe with a delta of the current/next version inputs, or an additional metadata flag) so that users can take actions before committing to a breaking change.

Does that sound like a valid use case ?

@nimarezainia

@nimarezainia @criamico are we still on track to get this completed by the end of the CY quarter (End of march) as orginally planned?

cc @kpollich

@jlind23

@jlind23 the work is on track, only #956 is taking longer as the integration team is currently helping me with real case scenarios and testing. That one could take longer to be finished, but the all the other issue are close to be completed.

@criamico

@criamico Should we pull someone from the docs team to start looking into https://github.com/elastic/ingest-docs/issues/1879?

Should we pull someone from the docs team to start looking into https://github.com/elastic/ingest-docs/issues/1879?

Yes I'll write in the channel and ask for assistance on it.

@vishaangelova

Closing this as done knowing that @vishaangelova started working on https://github.com/elastic/ingest-docs/issues/1879

Navigation Menu

Repository navigation

Allow integration authors to deprecate integrations and data streams #227

Summary

Implementation plan

20 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Allow integration authors to deprecate integrations and data streams #227

Description

Summary

Implementation plan

Activity

mtojek commented on Oct 13, 2021

jsoriano commented on Oct 19, 2021

mtojek commented on Oct 19, 2021

jlind23 commented on Oct 19, 2021

jsoriano commented on Oct 19, 2021

jlind23 commented on Oct 20, 2021

jsoriano commented on Oct 20, 2021

ruflin commented on Oct 20, 2021

ruflin commented on Nov 25, 2021

mtojek commented on Nov 25, 2021

ruflin commented on Nov 25, 2021

jsoriano commented on Nov 25, 2021

klacabane commented on Sep 8, 2022

20 remaining items

jlind23 commented on Mar 16, 2026

criamico commented on Mar 16, 2026

jlind23 commented on Mar 18, 2026

criamico commented on Mar 18, 2026

jlind23 commented on Mar 31, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions