Repository navigation
Open
Labels
bugExisting behaviour is wrong or brokenExisting behaviour is wrong or brokencomponent:beatsroles/beats - shared, or several beats at onceroles/beats - shared, or several beats at once
Milestone
Description
The three Beats run their setup task (loads index template, ILM, ingest pipelines, dashboards into Elasticsearch) inconsistently, and each has gaps. Auditbeat is the closest to correct and should be the reference for the other two.
Problems
- metricbeat uses bare
metricbeat setup(= index-management + dashboards + ML, but no pipelines), with a marker misleadingly namedpipelines_created. - filebeat is not gated on the Elasticsearch output (fails on the default Logstash output); passes
--pipelines --dashboardsbut no--index-management; modules live inmodules.d/, sosetup --pipelinesfinds none (needs--modules); marker is per-module. - auditbeat uses a run-once-forever marker, so modules/filesets added later never get their pipelines loaded. The marker is per host while the setup runs once, so a run limited to a host without the marker (
--limit) runs the setup again.
Target
- Gate all three on
beats_<beat>_output == "elasticsearch"andrun_once. - Always use explicit flags, never bare
setup, so the task is self-documenting:setup --index-management --pipelines, plus--dashboardsonly when Kibana is configured.--index-managementis safe (setup.template.overwrite: false). - Rename metricbeat's
pipelines_createdmarker to an accurate name (e.g.setup_done). - Replace the run-once-forever marker with re-run on change:
registerthe module/config task and runsetupwhen: <register> is changed. Withrun_once,whenonly checks the first host, so the condition has to look at the registered result of all hosts. - filebeat: add
--modules(or adopt the inline data-driven design) so module pipelines actually load. - Drop "(experimental)" from the
beats_filebeat_modulesdescription.
Background
With the Elasticsearch output the index template and pipelines auto-load on first run anyway; with the Logstash output nothing auto-loads, so setup is only really needed there — yet the tasks gate on the ES output and skip exactly that case.
The module handling is redesigned data-driven in #510 / #505 / #516 (inline, one list = desired state), which simplifies this further (inline → setup --pipelines sees the modules directly, no --modules). This issue is the fix for the current architecture.
Metadata
Metadata
Assignees
Labels
bugExisting behaviour is wrong or brokenExisting behaviour is wrong or brokencomponent:beatsroles/beats - shared, or several beats at onceroles/beats - shared, or several beats at once
Activity
[-][Bug]: Filebeat ingest-pipeline setup runs without ES and repeats per module[/-][+][Bug]: Beats setup tasks (filebeat/metricbeat) inconsistent with auditbeat[/+]afeefghannam89 commentedon Sep 16, 2026
The gate proposed here,
beats_<beat>_output == "elasticsearch", is the same comparison #552 needs. #552 writes it asbeats_<beat>_output.type | default(beats_<beat>_output)so that #522 does not have to touch it. Worth using the same form here, or #522 inherits two places instead of one.