Skip to content

[Bug]: Beats setup tasks (filebeat/metricbeat) inconsistent with auditbeat #508

Open

Description

@afeefghannam89

The three Beats run their setup task (loads index template, ILM, ingest pipelines, dashboards into Elasticsearch) inconsistently, and each has gaps. Auditbeat is the closest to correct and should be the reference for the other two.

Problems

  • metricbeat uses bare metricbeat setup (= index-management + dashboards + ML, but no pipelines), with a marker misleadingly named pipelines_created.
  • filebeat is not gated on the Elasticsearch output (fails on the default Logstash output); passes --pipelines --dashboards but no --index-management; modules live in modules.d/, so setup --pipelines finds none (needs --modules); marker is per-module.
  • auditbeat uses a run-once-forever marker, so modules/filesets added later never get their pipelines loaded. The marker is per host while the setup runs once, so a run limited to a host without the marker (--limit) runs the setup again.

Target

  • Gate all three on beats_<beat>_output == "elasticsearch" and run_once.
  • Always use explicit flags, never bare setup, so the task is self-documenting: setup --index-management --pipelines, plus --dashboards only when Kibana is configured. --index-management is safe (setup.template.overwrite: false).
  • Rename metricbeat's pipelines_created marker to an accurate name (e.g. setup_done).
  • Replace the run-once-forever marker with re-run on change: register the module/config task and run setup when: <register> is changed. With run_once, when only checks the first host, so the condition has to look at the registered result of all hosts.
  • filebeat: add --modules (or adopt the inline data-driven design) so module pipelines actually load.
  • Drop "(experimental)" from the beats_filebeat_modules description.

Background

With the Elasticsearch output the index template and pipelines auto-load on first run anyway; with the Logstash output nothing auto-loads, so setup is only really needed there — yet the tasks gate on the ES output and skip exactly that case.

The module handling is redesigned data-driven in #510 / #505 / #516 (inline, one list = desired state), which simplifies this further (inline → setup --pipelines sees the modules directly, no --modules). This issue is the fix for the current architecture.

Activity

  1. added this to the 1.0.0 milestone on Jul 23, 2026
  2. changed the title [-][Bug]: Filebeat ingest-pipeline setup runs without ES and repeats per module[/-] [+][Bug]: Beats setup tasks (filebeat/metricbeat) inconsistent with auditbeat[/+] on Jul 27, 2026
  3. afeefghannam89 commented on Sep 16, 2026

    @afeefghannam89
    MemberAuthor

    The gate proposed here, beats_<beat>_output == "elasticsearch", is the same comparison #552 needs. #552 writes it as beats_<beat>_output.type | default(beats_<beat>_output) so that #522 does not have to touch it. Worth using the same form here, or #522 inherits two places instead of one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugExisting behaviour is wrong or brokencomponent:beatsroles/beats - shared, or several beats at once

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions