Skip to content

Add opt-in shared MaxMind GeoIP databases for Arkime and Logstash - #1215

Open
sylvesterkaczmarek wants to merge 1 commit into
cisagov:contributor-prsfrom
sylvesterkaczmarek:fix/394-share-geoip-databases
Open

sylvesterkaczmarek wants to merge 1 commit into
cisagov:contributor-prsfrom
sylvesterkaczmarek:fix/394-share-geoip-databases

Conversation

@sylvesterkaczmarek

@sylvesterkaczmarek sylvesterkaczmarek commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Maintenance update on 6 November 2026

Refreshed onto contributor-prs at 7046399b after the integration branch history changed. This PR now contains its 8-file topic change in one commit.

python -m pytest -q shared/tests/test_shared_geoip_databases.py: 10 passed, 4 subtests passed. Shell checks, Python checks and Compose mount/permission validation pass. Git whitespace checks pass.

No live MaxMind download or Arkime/Logstash deployment was run.

Earlier validation counts below record the original branch state. ChatGPT assisted with this refresh, corrections and validation.


Summary

Implements an opt-in way to standardize the MaxMind GeoIP database files used by Arkime and Logstash, addressing #394.

  • Add one Docker named volume (geoip-shared) shared between arkime, arkime-live (publishers) and logstash (read-only consumer).
  • Publish Arkime's GeoLite2-City.mmdb, GeoLite2-ASN.mmdb, and GeoLite2-Country.mmdb into that volume on its existing GeoIP update path, using temporary files and atomic renames.
  • Introduce LOGSTASH_GEOIP_SHARED_DB=false as an opt-in setting. When enabled, Logstash validates that both required MMDBs are present, then adds explicit database paths to its runtime GeoIP filter configurations (including Beats, Zeek and enrichment pipelines).
  • Preserve all existing GeoIP filter IDs and custom database options.
  • Keep Logstash's existing built-in GeoIP sources unchanged by default, including on hosts without MaxMind credentials. Enabling the feature before databases are populated produces a clear startup error.
  • Document setup, the required Arkime-first startup, and coordinated refresh/restarts.

Verification

  • 10 self-contained regression tests passed, covering all 13 existing GeoIP filters, correct City/ASN selection, idempotence, custom configurations, missing files, malformed filters, atomic publication and the three Compose mounts.
  • Docker Compose configuration validation passed (docker compose config --no-interpolate --no-env-resolution --quiet).
  • Ruff, Black, Bash/POSIX-shell syntax, Python compilation and Git diff checks passed.

No live Logstash+Arkime integration deployment or MaxMind credential-based download was tested; a deployment test is recommended prior to enabling the opt-in feature in production.

Integration refresh (15 October 2026)

  • Rebases the changes onto upstream contributor-prs at a1da19b0, not the older main snapshot. Local dry-run integration of all 21 reviewed open PRs completed without merge conflicts.
  • Focused tests passed: python -m pytest -q shared/tests/test_shared_geoip_databases.py.
  • Combined local Python integration suite: 776 passed, 6 skipped, 157 subtests passed; three Ruby suites: 24 tests, 97 assertions, all passed. Ten modified shell scripts passed ShellCheck (error severity), the NetFlow/TLS Docker Compose overlays parsed, and changed local Markdown links resolved.
  • This is offline/repository-level validation, not a live Malcolm, Zeek, Kubernetes, Azure or AWS deployment test, nor a complete Docker image build.

@sylvesterkaczmarek
sylvesterkaczmarek force-pushed the fix/394-share-geoip-databases branch from 5343f27 to a2b1623 Compare October 15, 2026 20:58
@sylvesterkaczmarek
sylvesterkaczmarek changed the base branch from main to contributor-prs October 15, 2026 20:58
Refresh the focused change onto contributor-prs at 7046399.

`python -m pytest -q shared/tests/test_shared_geoip_databases.py`: **10 passed, 4 subtests passed**. Shell checks, Python checks and Compose mount/permission validation pass.

No live MaxMind download or Arkime/Logstash deployment was run.
@sylvesterkaczmarek
sylvesterkaczmarek force-pushed the fix/394-share-geoip-databases branch from a2b1623 to 41fcf8d Compare November 6, 2026 07:48
@mmguero mmguero added this to Malcolm Nov 9, 2026
@mmguero mmguero added this to the z.staging milestone Nov 9, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

2 participants