Skip to content

Preserve saved installer configuration during bulk environment import - #1222

Closed
sylvesterkaczmarek wants to merge 114 commits into
cisagov:contributor-prsfrom
sylvesterkaczmarek:fix/949-preserve-imported-configuration-20261010
Closed

sylvesterkaczmarek wants to merge 114 commits into
cisagov:contributor-prsfrom
sylvesterkaczmarek:fix/949-preserve-imported-configuration-20261010

Conversation

@sylvesterkaczmarek

@sylvesterkaczmarek sylvesterkaczmarek commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Addresses #949.

Changes

  • Suspend automatic value-dependency updates while existing .env settings are being imported, preventing a field loaded later in the process from resetting one loaded earlier.
  • Recalculate dependent values when the import finishes, protecting explicitly imported settings and valid inferred values such as a remote NetBox URL.
  • Continue updating visibility during loading, and restore normal dependency reactions for subsequent user-initiated changes.
  • Add regression tests for a saved Hedgehog live-capture configuration where liveArkime=false must stay disabled, and for dependency updates after loading.

Validation

  • 14 focused unit tests across environment loading, precedence, dependencies and sentinels: passed.
  • The complete installer unit suite on the patch reports 61 tests, 4 failures and 6 errors. The same four failures and six errors also occur on the unchanged upstream checkout (59 tests). No additional failing cases.
  • Python compileall on modified modules and tests: passed.
  • git diff --check: passed.

A live Hedgehog deployment was not available for integration testing.

Integration refresh (15 October 2026)

  • Rebases the changes onto upstream contributor-prs at a1da19b0, not the older main snapshot. Local dry-run integration of all 21 reviewed open PRs completed without merge conflicts.
  • Focused tests passed: python -m pytest -q shared/tests/test_env_file_import_existing.py.
  • Combined local Python integration suite: 776 passed, 6 skipped, 157 subtests passed; three Ruby suites: 24 tests, 97 assertions, all passed. Ten modified shell scripts passed ShellCheck (error severity), the NetFlow/TLS Docker Compose overlays parsed, and changed local Markdown links resolved.
  • This is offline/repository-level validation, not a live Malcolm, Zeek, Kubernetes, Azure or AWS deployment test, nor a complete Docker image build.

sylvesterkaczmarek and others added 30 commits October 6, 2026 13:09
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
…tel-timezone-timestamps-20261006

Preserve threat-intelligence timestamps across host timezones
…t-pattern-boolean-semantics-20261006

Reject STIX conjunctions and partial non-equality patterns
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
…fest-cutoff-times-20261006

Apply UTC cutoffs and the supplied fallback time to MISP manifests
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
…p-without-health-wait-20261006

Keep PCAP duplicate checks enabled when startup health waiting is disabled
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
…worker-cancellation-20261006

Cancel waiting scan workers without releasing unowned capacity
…ention-overlapping-patterns-20261006

Count overlapping index selections once during retention pruning
…-timezone-invariance-20261006

Preserve UTC query instants across API host timezones
…ral-node-matching-20261006

Match literal sensor names when checking PCAP duplicates
…ock-inode-lifetime-20261006

Keep cleanup runners coordinated through a stable lock file
…single-record-registry-20261006

Accept single-record Filebeat registries during cleanup
…-candidate-byte-precision-20261007

Preserve fractional index sizes when choosing retention deletions
…positive-upload-chunks-20261007

Reject nonpositive file-upload chunk sizes before reading data
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
3.1.59 is affected by GHSA-239g-whfq-7xj9, GHSA-g5vv-9gxw-82hx and
GHSA-rw58-wc66-99g4; 3.1.62 also clears GHSA-59cr-6r3x-644w, the last
open GitPython advisory in OSV.
sylvesterkaczmarek and others added 7 commits October 15, 2026 21:13
Retain the original per-file exception containment, record candidate files, discovered names and PLUGINS changes, and format with Black. Add logging and unexpected-parser-error regression coverage.

Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
…x-plugin-discovery-20261015

Make NetBox plugin discovery deterministic and scoped
@sylvesterkaczmarek
sylvesterkaczmarek force-pushed the fix/949-preserve-imported-configuration-20261010 branch from 55959a8 to bb58e5d Compare October 15, 2026 20:58
@sylvesterkaczmarek
sylvesterkaczmarek changed the base branch from main to contributor-prs October 15, 2026 20:58
@mmguero mmguero self-assigned this Oct 20, 2026
@mmguero mmguero added the install.py Relating to the install.py configuration script label Oct 20, 2026
@mmguero mmguero added this to the v26.10.0 milestone Oct 20, 2026
@mmguero mmguero added this to Malcolm Oct 20, 2026
@mmguero mmguero moved this to Review in Malcolm Oct 20, 2026
@mmguero

mmguero commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator

Thanks for this, @sylvesterkaczmarek. I spent some time testing it, and it does what it says. Your Hedgehog regression test fails on the current code and passes with the patch, and a round-trip through the .env files now preserves the capture settings in most of the cases that used to come back wrong.

While digging in, I found that the same underlying problem shows up in a few places outside load_from_env_files. Importing with --import-malcolm-config-file still re-enables liveArkime, and the compose-file load that install.py runs right after the env load can still reset settings on Hedgehog. Some user edits made after a load can do it too, since loaded values are still treated as unmodified defaults. Fixing that properly means changing how every load path tracks where a value came from.

That change reaches into parts of the installer with a lot of history behind them, so I'm going to take it on myself for the next release (v26.11.0). You don't need to rework this PR to cover the broader approach. I'll keep it open for now as a reference, and I expect some from this will carry over into the final fix.

Thanks again.

@mmguero mmguero modified the milestones: v26.10.0, v26.11.0 Oct 23, 2026
@sylvesterkaczmarek

Copy link
Copy Markdown
Contributor Author

Thanks for testing the Hedgehog round trip and tracing the remaining load paths. Agreed that tracking which settings were explicitly supplied needs to work consistently across .env, imported config and compose loading, not just in this one importer. I'll leave this PR as a reference for the broader v26.11.0 fix rather than expand it into overlapping changes. The existing regression should be useful for checking the final implementation.

mmguero added a commit to mmguero-dev/Malcolm that referenced this pull request Nov 10, 2026
Value rules now declare a kind (derived, cascade, invariant) and are paused
while configuration loads, then refreshed once. Loaded values are protected
from cascades like values set in the session. Sources have fixed precedence:
command line, settings file, its raw .env backup, existing .env files,
compose file, ISO preseed. Saving is staged and all-or-nothing, and invalid
loaded values block the save.

Also fixes: blank Zeek intel cron lost on export/import, suggested memory
sizes not loading, generated .env files missing a final newline, and the
broken get_env_var_by_item_key wrapper.

ref: cisagov#949
     cisagov#1110
     cisagov#1222

also, update installer tests that expected behavior the code doesn't have

The Traefik tests exposed OpenSearch while open ports was still "No", which
turns exposure back off; they now choose "Customize" first. The storage
directory tests expected blank directories to be flagged when default
locations are off, but a blank directory uses its default location, so they
now test that, and a new compose integration test checks the fallback.

Signed-off-by: Seth Grover <seth.d.grover@gmail.com>
@mmguero

mmguero commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator

closing in lieu of #1300

@mmguero mmguero closed this Nov 10, 2026
@github-project-automation github-project-automation Bot moved this from Review to Done in Malcolm Nov 10, 2026
@mmguero mmguero moved this from Done to Declined in Malcolm Nov 10, 2026
@mmguero mmguero removed this from the v26.11.0 milestone Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

install.py Relating to the install.py configuration script

Projects

Status: Declined

Development

Successfully merging this pull request may close these issues.

4 participants