Skip to content

Support nested assets in custom Logstash pipelines - #1207

Open
sylvesterkaczmarek wants to merge 1 commit into
cisagov:contributor-prsfrom
sylvesterkaczmarek:fix/363-stage-nested-pipeline-assets
Open

sylvesterkaczmarek wants to merge 1 commit into
cisagov:contributor-prsfrom
sylvesterkaczmarek:fix/363-stage-nested-pipeline-assets

Conversation

@sylvesterkaczmarek

@sylvesterkaczmarek sylvesterkaczmarek commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Maintenance update on 6 November 2026

Refreshed onto contributor-prs at 7046399b after the integration branch history changed. This PR now contains its 5-file topic change in one commit.

python -m pytest -q shared/tests/test_logstash_pipeline_assets.py: 11 passed, using real Bash and rsync. Shell syntax and ShellCheck pass. Git whitespace checks pass.

No Logstash image build or live deployment was run.

Earlier validation counts below record the original branch state. ChatGPT assisted with this refresh, corrections and validation.


Summary

Addresses the custom-filter and site-specific lookup use case in #363 using Malcolm's existing pipeline bind mount.

  • Replace the flat copy operation, which fails on nested lookup directories, with a tested recursive staging helper.
  • Preserve bundled filters and allow intentional same-name overrides. Keep the source mount unchanged and normalize runtime permissions for startup transformations.
  • Load only top-level *.conf files as pipeline configuration, while keeping nested dictionaries and Ruby assets available.
  • Treat missing and empty mounts as no-ops, reject overlapping directories and invalid pipeline names, skip source symlinks, and propagate staging failures.
  • Document a translate-filter example, read-only Compose bind mount, and restart/removal behavior.

Validation

  • python3 -m unittest -q scripts.installer.tests.unit.test_logstash_pipeline_assets: 11 tests passed using the real shell helper and filesystem operations.
  • Both shell scripts pass bash -n; the test file passes Ruff and Black using the documented repository options.
  • git diff --check passed.
  • Three-way file merge check with the existing documentation change in Add repeatable enrichment storage and processing benchmarks #1206 passed.

No live Logstash pipeline or full Docker image build was run. This is scoped support for custom pipeline assets; the broader plugin-package management work remains open. Based on upstream main at 7cfac5b.

Integration refresh (15 October 2026)

  • Rebases the changes onto upstream contributor-prs at a1da19b0, not the older main snapshot. Local dry-run integration of all 21 reviewed open PRs completed without merge conflicts.
  • Focused tests passed: python -m pytest -q shared/tests/test_logstash_pipeline_assets.py.
  • Combined local Python integration suite: 776 passed, 6 skipped, 157 subtests passed; three Ruby suites: 24 tests, 97 assertions, all passed. Ten modified shell scripts passed ShellCheck (error severity), the NetFlow/TLS Docker Compose overlays parsed, and changed local Markdown links resolved.
  • This is offline/repository-level validation, not a live Malcolm, Zeek, Kubernetes, Azure or AWS deployment test, nor a complete Docker image build.

@sylvesterkaczmarek
sylvesterkaczmarek force-pushed the fix/363-stage-nested-pipeline-assets branch from ddb8445 to 72decf3 Compare October 15, 2026 20:58
@sylvesterkaczmarek
sylvesterkaczmarek changed the base branch from main to contributor-prs October 15, 2026 20:58
Refresh the focused change onto contributor-prs at 7046399.

`python -m pytest -q shared/tests/test_logstash_pipeline_assets.py`: **11 passed**, using real Bash and rsync. Shell syntax and ShellCheck pass.

No Logstash image build or live deployment was run.
@sylvesterkaczmarek
sylvesterkaczmarek force-pushed the fix/363-stage-nested-pipeline-assets branch from 72decf3 to 550c4cd Compare November 6, 2026 07:47
@mmguero mmguero added this to Malcolm Nov 9, 2026
@mmguero mmguero added this to the z.staging milestone Nov 9, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

2 participants