Repository navigation
Malcolm v26.10.0 - #1275
Merged
Merged
Malcolm v26.10.0#1275
Conversation
…s (3.9.0), and urllib3 (2.8.0) Signed-off-by: Seth Grover <seth.d.grover@gmail.com>
Document lookups previously offered no control over result ordering or paging, and omitted the total hit count. Add optional sorting and offset paging to /mapi/document, along with total.value and total.relation. Callers can request exact counts through track_total_hits while existing requests retain the backend’s default counting behavior. Both /mapi/document and /mapi/agg discarded shard status and timeout information, allowing incomplete searches to look like ordinary empty or short results. Return the full shard status and timed_out flag alongside available results. Preserve HTTP 200 for partial responses so clients can use the data and assess its completeness. Validate the new arguments and return HTTP 400 for malformed input. Reject aggregation names that collide with response metadata. Add debug summaries with result counts and backend search status, log shard failure details when present, and remove duplicate aggregation argument logging. Update the API documentation and examples, including outdated aggregation keys that used values instead of field names. Validated against local OpenSearch with 138 passing checks and no failures in each of two runs, covering sorting on @timestamp and firstPacket. Checks exercised legacy requests, paging and count behavior, plus invalid inputs. Another 32 isolated scenarios checked response compatibility and metadata handling with simulated partial failures, timeouts, and missing metadata. Python syntax and all 13 JSON documentation examples passed validation. A Bash validation script was used locally to verify the results (see attachment to the bugs listed below). Live fault injection, result-window rejection, and Elasticsearch verification remain outstanding. Defer search_after because offset paging satisfies the current request. Reliable cursor paging needs a suitable unique tie-breaker across document types, and snapshot-consistent paging would also require point-in-time support. Existing backend result-window limits still apply. Fixes #1130 Fixes #1131 Signed-off-by: Seth Grover <seth.d.grover@gmail.com>
Create the debconf template in a private mktemp directory and clean it up on exit or interruption. Abort if directory creation fails. Remove unused malcolm.answer writes. Refs: GHSA-wmrp-jr97-8xf4 https://chatgpt.com/c/6ac404ec-1ca8-832b-a722-3f612549dd32 Signed-off-by: Seth Grover <seth.d.grover@gmail.com>
ARKIME_WISE_CONFIG_PIN_CODE no longer ships with a default value. The example env file leaves it empty, and the Arkime entrypoint generates a random PIN at startup when none is set. Existing installs still using WISE2019 are migrated to an empty value through env-var-actions.yml. auth_setup now prompts for the WISE PIN and accepts --auth-arkime-wise-pin in non-interactive mode. A blank answer leaves the current value unchanged. --auth-arkime-password no longer defaults to "Malcolm", and the Arkime secret values are only written when provided. Fix two bugs in checkEnvFilesAndValues found while adding the migration: value maps in copied_environment_variables dropped empty-string results because of a truthiness check, and the examples-directory fallback looked for .env files instead of .env.example. Remove the explicit 0644 mode from that UpdateEnvFiles call so it no longer loosens permissions on secret env files. Update malcolm-config.md and authsetup.md to match. Closes #1133 Signed-off-by: Seth Grover <seth.d.grover@gmail.com>
The MaxMind account ID, license key, and alternate download URL were passed to the Arkime image build as build arguments. Because the ARGs were declared in the final stage ahead of every RUN instruction, their values were recorded in the history of each subsequent layer and published with the image. GITHUB_TOKEN was also passed as a build argument, and scripts/build.sh exposed all of these values on the docker compose command line. These values are now provided as BuildKit secrets and mounted as environment variables only for the RUN instructions that need them, so they no longer appear in image history or process arguments. - arkime.Dockerfile: remove the MaxMind ARGs and mount them as secrets for the GeoIP download step - nginx.Dockerfile: remove the GITHUB_TOKEN ARG from the docbuild stage and mount it as JEKYLL_GITHUB_TOKEN. The previous export ran in a subshell and never reached Jekyll, so the token is now passed to the docs build for the first time - docker-build-push-scan.yml: move the credentials from build-args to the build-push-action secrets input - docker-compose-dev.yml: add build secrets for arkime, arkime-live, and nginx-proxy, sourced from environment variables - scripts/build.sh: export the credentials for the compose secrets instead of passing --build-arg values, and collapse the duplicated build invocation into one - malcolm-iso and hedgehog-raspi build_via_vagrant.sh: write environment.chroot with umask 077 and register the cleanup trap before writing it Also fix three existing bugs in scripts/build.sh: a doubled "=" in the CONFIG_ENV_DIR assignment, missing -f tests when reading MaxMind values from arkime-secret.env, and a bad substitution in the <runtime>-compose fallback. Signed-off-by: Seth Grover <seth.d.grover@gmail.com>
) control.py and the appliance's agg-init.sh both set nginx/htpasswd and htadmin/metadata to 0644, leaving the bcrypt password hashes readable by any local user who can reach the Malcolm directory. The nginx and htadmin worker processes run as PUID/PGID, which matches the files' owner, so world-read access is not needed. Set both files to 0640. control.py applies this in start(), so existing installs are corrected the next time Malcolm starts. Signed-off-by: Seth Grover <seth.d.grover@gmail.com>
Several API endpoints made outbound HTTP requests to other Malcolm services without a timeout. A dependency that accepted a connection and then stopped responding held the API request open until gunicorn killed the worker, and with gunicorn's default single sync worker, that blocked the entire API, including the /mapi/ping healthcheck. - Add (connect, read) timeouts to every outbound requests call, configurable through MALCOLM_API_HTTP_CONNECT_TIMEOUT_SEC (default 5) and MALCOLM_API_HTTP_READ_TIMEOUT_SEC (default 30) - Bound each /mapi/ready check with MALCOLM_API_READY_TIMEOUT_SEC (default 5) and run the checks concurrently, so the endpoint's total time is that of its slowest check rather than the sum of all of them - Add socket_timeout to the Valkey connections used by /mapi/ready and the keyspace endpoint - Add an optional timeout argument to malcolm_utils.check_socket, with the default unchanged at 10 seconds - Run gunicorn with --workers 2 --threads 4 in both compose files and the Kubernetes manifest, so one slow request can't block the API or its healthcheck Closes #1135 Signed-off-by: Seth Grover <seth.d.grover@gmail.com>
…2-r23g) When NGINX_AUTH_MODE is keycloak_remote and KEYCLOAK_SSL_VERIFY is not true, nginx does not verify the remote Keycloak's TLS certificate, so a host able to intercept that traffic could impersonate the identity provider. Verification stays disabled by default to accommodate self-signed certificates, but the risk is now visible. - docker_entrypoint.sh: log a warning at startup in keycloak_remote mode when verification is disabled, explaining the risk and how to enable it. The check uses the same exact "true" match as nginx_auth_helpers.lua - authsetup.md: document the verification setting in the remote Keycloak walkthrough, recommend enabling it, and explain how to trust internal CAs through nginx/ca-trust/ Signed-off-by: Seth Grover <seth.d.grover@gmail.com>
…v3-2383) The Beats input on port 5044 used ssl_client_authentication "optional", so with BEATS_SSL enabled a forwarder that presented no client certificate could still connect and submit events over TLS. Presented certificates were verified, but omitting one bypassed authentication. Set it to "required" so only forwarders holding a certificate issued by Malcolm's CA can send events. All supported producers already present the client certificate generated by auth_setup, including Malcolm's internal Filebeat, Hedgehog Linux sensors, and the documented external forwarder configurations. When BEATS_SSL is false, logstash-start.sh still strips the ssl_ settings from the input, so the plaintext configuration is unaffected. Also set ssl.client_authentication "required" explicitly on the Filebeat TCP input (port 5045). It already required client certificates through libbeat's default when certificate_authorities is set; this documents the behavior and keeps it from silently changing if that setting is removed. Update the BEATS_SSL descriptions in malcolm-config.md and third-party-logs.md to state that client certificate authentication is required. Closes #1136 Signed-off-by: Seth Grover <seth.d.grover@gmail.com>
watch-pcap-uploads-folder.py runs as root and called os.path.isfile() and os.chown() on paths in the upload directory, both of which follow symlinks. Because the upload service's SFTP server allows creating symlinks, a symlink placed in the upload directory could cause the watcher to change the ownership of its target, and the gap between the check and the chown allowed a regular file to be swapped for a symlink. - Inspect each upload with lstat() and remove symlinks and other non-regular entries instead of processing them - Open uploads with O_NOFOLLOW and perform the ownership change, file type detection, and size check through that descriptor (fchown, magic.from_descriptor, fstat) - Before moving or decompressing an upload, confirm the path still refers to the same file, and remove it if it changed - Open compressed uploads through the same O_NOFOLLOW descriptor, and set ownership and mode on the decompression temp file through its descriptor rather than by path Closes #1137 Signed-off-by: Seth Grover <seth.d.grover@gmail.com>
The hh-extracted-files location proxies to a host taken from the request path, which Malcolm can't validate (see GHSA-j89q-3q7m-8vr6). Requests were forwarded with the client's Authorization header and cookies intact, so a crafted link could cause a user's Malcolm credentials (the basic auth header, Keycloak session cookie, or the bearer token set by the Lua auth code) to be sent to an arbitrary host on port 8006. Clear the Authorization and Cookie headers in that location, as the Arkime and upload locations already do for Authorization. Hedgehog's extracted-file server authorizes Malcolm by its ACL and does not rely on either header. Signed-off-by: Seth Grover <seth.d.grover@gmail.com>
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
…XII 2.1, drop MISP context values STIX patterns (#1148) Builds on #1144. Patterns are normalized to disjunctive normal form and each clause is judged on its own, so no emitted item can match traffic the pattern wouldn't: - OR alternatives and IN members emit one item each; OR branches that can't be expressed are dropped and the rest kept - an AND within one observation that includes a file or certificate hash is reduced to the hash(es), noted in meta.desc - network-traffic address-type tests are dropped in favor of the address; src_ref/dst_ref values map to Intel::ADDR - other conjunctions, relations between observations, and qualifiers are rejected, with the reason in the warning - object paths are normalized before the type map lookup, so quoted and unquoted hash names resolve the same TAXII 2.1 and STIX parsing (#1149) - TAXII 2.1 envelopes are parsed (previously every page failed) - since is sent to the server as added_after - collections reporting can_read: false are skipped - indicators are parsed one at a time, so one malformed object no longer discards the whole bundle; STIX 2.0 bundle spec_version is passed to its objects - empty indicator names no longer produce a leading ". " in meta.desc MISP context values (#1150) - MISP objects with a file, certificate, or key hash emit only their hashes, noted in meta.desc - filename|hash composites emit only the hash, with the file name in the description - attributes with to_ids: false are skipped by default; restore the previous behavior per feed with require_to_ids: False in misp.yaml, or globally with --misp-require-to-ids false BEHAVIOR CHANGE: with the to_ids default, about 4.5% of MISP items from public feeds are no longer emitted (mostly URLhaus payload host IPs). Tests: test_stix_point_patterns.py extended; new test_misp_hash_reduction.py and test_stix_bundle_parsing.py (103 tests total). Docs updated in docs/zeek-intel.md. Fixes #1148 Fixes #1149 Fixes #1150 Signed-off-by: Seth Grover <seth.d.grover@gmail.com>
When importing a settings file, _load_configuration_section reported every configuration item without is_modified set as "missing from settings file." Items whose imported value matched their default, and items exported with the empty-value sentinel, were counted even though they were present in the file, so a fresh export re-imported by the same version reported a large number of missing items. Count only keys absent from the file's configuration section, matching how _load_installation_section already reports missing installation items. Signed-off-by: Seth Grover <seth.d.grover@gmail.com>
When importing a configuration file with --import-malcolm-config-file, install.py restored each entry in its "raw" section with base64_decode_files_to_dir, which joined the entry's key onto the config directory without verifying the result. A key containing ".." components, or an absolute path, wrote the decoded contents outside the config directory, anywhere writable by the account running the installer. - Skip entries whose path is absolute, empty, or not a string, or which resolve outside the destination directory through ".." components or symlinks, logging a warning for each - Add an optional filename pattern to base64_decode_files_to_dir, and restrict imported files to *.env, matching what configuration export produces - Return the number of files written, and report it in the installer's restore message - Verify that "raw" is a dictionary before processing it, and skip non-string Base64 values instead of raising Reported by Sithum Shihara (@zaara2004). Closes #1147 Signed-off-by: Seth Grover <seth.d.grover@gmail.com>
The suricataDisableSids DependencySpec was copied from suricataRuleUpdate and kept its ValueRule with default_value=False. Toggling autoSuricata or liveSuricata during import tried to apply False to a list-of-strings item, which the validator rejects, so each import logged "Failed to set default" twice. Drop the ValueRule and keep only the visibility rule. The item's own default_value=[] covers fresh installs, and leaving out an empty-list rule avoids wiping imported SIDs whenever a Suricata toggle fires. Signed-off-by: Seth Grover <seth.d.grover@gmail.com>
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
…ion 4.x) addresses #1269 Signed-off-by: Seth Grover <seth.d.grover@gmail.com>
…6.10.0_merge_cisagov
Contributor
|
Thank you for the kind acknowledgement and for all the work the Malcolm team puts into this project; it's a real pleasure to contribute. |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Malcolm v26.10.0 updates Zeek to 9.0.1 LTS, bringing new log types (including multicast participants and the NTP control and private modes) and a corresponding Multicast dashboard; substantially expands threat intelligence enrichment, mapping Zeek intel, Suricata rule metadata, and capa results into normalized
threat.*fields with MITRE ATT&CK tactic, technique, and subtechnique context; normalizes Zeek OT protocol fields to their Arkime equivalents for eight protocols so filters and drill-downs carry across Dashboards and Arkime; adds optional Community ID computation in Logstash for Zeek logs that arrive without one; and extends the Malcolm API's document lookup with sorting, paging, total hit counts, and backend search status, so a partial or timed-out search can no longer look like a normal empty result. Uploaded PCAP processing received two significant correctness fixes: Zeek now waits for its intelligence files to finish loading before processing packets, and Suricata's unix-socket instances are warmed up and reloaded so alerts carry rule metadata. A large supply-chain hardening effort pins and verifies nearly every external artifact consumed during the build. OpenSearch and OpenSearch Dashboards, Arkime, Keycloak, Logstash, Filebeat, NetBox, and other components have been updated. Many other bug fixes and general improvements are also included.This release also fixes nine security issues, the most significant being stored cross-site scripting through the file-upload container, which allowed a user with upload access to run arbitrary JavaScript in another Malcolm user's session, and credential disclosure through the Hedgehog extracted-files proxy, which forwarded the caller's headers to a destination the caller names.
The Malcolm team would like to thank @sylvesterkaczmarek, @pavanchow, @Darshan060224, @sercanokur, @nagameTW, @katsugtgz, and @zaara2004 for their contributions to this release.
If you are upgrading from an existing Malcolm installation, run
./scripts/statusfor Malcolm to migrate some settings prior to running./scripts/configure,./scripts/start, or other Malcolm control scripts.v26.09.0...v26.10.0
✨ Features and enhancements
multicast_participants.logis parsed, mapped, and surfaced in a new Multicast dashboard, andntp_control.logandntp_private.logare parsed and added to the index template and Arkime field definitions #1119zeek.*fields are preserved; the new top-level fields are added alongside them, with Modbus function and exception names converted to their numeric codes to match Arkime's integer typing. The OT dashboards and the OpenSearch composable template were updated to match #993, #1257, #986threat.*fields across every source that provides it: Zeek intel kill chain phases becomethreat.tactic.*, MISP MITRE galaxy tags incif_tagsbecomethreat.technique.*andthreat.technique.subtechnique.*, Mandiant and Google Threat Intelligence actors becomethreat.group.name, and Suricata's Emerging Threatsmitre_*metadata and capa's subtechnique results are mapped the same way. Tactic and technique names come from generated ATT&CK dictionaries so a given ID always resolves to one name, with reference URLs added and ATT&CK for ICS distinguished from Enterprise. The Threat Intelligence dashboard, Arkime field definitions, and the WISE field list were updated accordingly #695meta.threat_score), and STIX 2.1confidencevalues. Mandiant's M-Score is treated as confidence rather than severity, and absent values are left unset rather than invented #1185, #1190, #1201, #1244ioc_typesandinclude_autogenerated_tagsparameters for Google Threat Intelligence collections, so a deployment can load only the indicator types it needs and leave out machine-generated tags; both reduce the memory a large collection costs in ZeekLOGSTASH_ZEEK_COMPUTE_COMMUNITY_ID(enabled by default). An ID Zeek supplied is never overwritten; for logs with noprotocolumn, the transport is inferred by log type only where the analyzer runs over a single transport #400sort,offset, andtrack_total_hitsarguments and atotalhit count with its exact-or-lower-bound relation, and return the backend'sshardsandtimed_outstatus from both document lookup and field aggregations, so a client can tell a partial or timed-out search from a genuinely empty one. Argument validation errors now return HTTP 400 with anerrorstring, andMALCOLM_API_DEBUG=truelogs a compact per-request search summary #1130, #1131schemereader is the default and handles it alongside Zeek and Suricata. Some link types, including IEEE 802.11, are still not fully decoded by Arkime #407kiosk.serviceand removes the browser autostart entry; the question never appears on Malcolm installs #1264NETBOX_DEVICETYPE_IMPORT_VENDORSto limit the first import to selected manufacturers. Note that limiting vendors also limits which manufacturers can be matched during MAC OUI enrichment #1196ZEEK_PORTS_…environment variables: BACnet, C12.22, EtherNet/IP, GE-SRTP, Genisys, HART-IP, Modbus, Omron FINS, OPC UA Binary, Profinet IO-CM, ROC+, S7comm, and Synchrophasor. TCP and UDP entries are applied to whichever transports each analyzer supports, so a site running one of these protocols on a nonstandard port no longer needs a custom Zeek script to parse it. The three variables that already existed are renamed into this pattern (ZEEK_SYNCHROPHASOR_PORTS,ZEEK_GENISYS_PORTS, andZEEK_ENIP_PORTSbecomeZEEK_PORTS_SYNCHROPHASOR,ZEEK_PORTS_GENISYS, andZEEK_PORTS_ENIP), and existing values are migrated on upgrade #1272, #1273🛡️ Security Remediation & Hardening
file-uploadcontainer, where uploaded HTML and SVG files could be read back on the Malcolm origin as active content: the responses carried a content-derivedContent-Type,Content-Disposition: inlineon the FilePond PHP routes, noX-Content-Type-Options: nosniff, and a CSP with noscript-srcordefault-src, so an inline script ran with the session of whoever opened the URL. Uploaded content was reachable through several static paths (including two that bypassed the location guarding the upload directory) and through FilePond'srestoreandloadroutes. Triage also found that the chunked-uploadpatchroute accepted an unvalidated transfer ID, allowing files to be written into any directory writable by the web server user while bypassing the extension deny-list. Since Malcolm's upload page never reads uploads back, the read side is removed entirely: the upload container's nginx returns 404 by default and serves only the upload page, its static assets, and the two PHP entrypoints; those entrypoints accept only POST and DELETE; therestore,load, andpatchhandlers are stripped from the vendored PHP at build time with the build failing if any remain; and PHP responses now carrynosniffanddefault-src 'none'; sandboxGHSA-qv8w-m37v-6c7jhh-extracted-filesproxy, which built its upstream from a caller-supplied path segment and forwarded the caller'sAuthorizationheader, sessionCookie, and theX-Forwarded-User,X-Forwarded-Groups, andX-Forwarded-Rolesheaders written by Malcolm's own auth layer to that destination. All five headers are now cleared before the request is proxied, in bothnginx.confandnginx_readonly.confGHSA-325g-72fr-vj2xWISE2019), which was published in both the example configuration and the documentation.ARKIME_WISE_CONFIG_PIN_CODEnow ships empty, the Arkime container generates a random PIN when none is configured, existing installs still holding the known value are migrated to empty on upgrade, and./scripts/auth_setupprompts for it (with--auth-arkime-wise-pinfor noninteractive setup). The shipped default for the Arkime viewer password secret was removed in the same pass GHSA-w2jh-h3pq-273j #1133scripts/build.sh, and the ISO and Raspberry Pi build scripts writeenvironment.chrootwith restrictive permissions and register cleanup before the write. While reviewing this, the MaxMind build arguments were found to have been declared in the Arkime image's final stage, recording their values in the layer history of published images; those credentials have been rotated, and the new images were verified to contain no credential values GHSA-3293-9mf2-qf68 #1134BEATS_SSL=trueand accepted a forwarder that presented none. Forwarders using the client certificate generated by./scripts/auth_setup, including Hedgehog Linux sensors and Malcolm's internal Filebeat, are unaffected GHSA-f2gq-m4v3-2383 #1136lstat(), with symlinks and other non-regular entries removed rather than processed, opened once withO_NOFOLLOW, and handled entirely through that file descriptor GHSA-98jw-5wq7-6qjv #1137/mapi/pinghealthcheck. Timeouts are configurable throughMALCOLM_API_HTTP_CONNECT_TIMEOUT_SECandMALCOLM_API_HTTP_READ_TIMEOUT_SEC, each/mapi/readycheck is bounded byMALCOLM_API_READY_TIMEOUT_SECand the checks now run concurrently, Valkey connections set a socket timeout, and the API runs with multiple gunicorn workers and threads under both Docker Compose and Kubernetes GHSA-x9f2-h2rj-vp63 #1135nginx/htpasswdandhtadmin/metadatafrom 0644 to 0640, so web authentication hashes are no longer readable by other local users when the parent directories permit traversal. Becausescripts/control.pyapplies this at startup, existing installations are corrected the next time Malcolm starts GHSA-rxpm-f8cm-2m5m/tmpas root. The debconf template is now written to a privatemktemp -ddirectory with cleanup on exit and on signal, and the unused answer-file writes were removed GHSA-wmrp-jr97-8xf4 #1132/mapi/eventendpoint in a newevent.submitterfield, set after the submitted body is merged so a caller cannot supply it, and namespace caller-supplied document IDs as<date>-alert.<id>. Arkime session IDs and Zeek UIDs never contain a., so a caller can no longer choose an ID that collides with, and overwrites, a record written by Malcolm's own pipeline--import-malcolm-config-fileto the configuration directory: entries whose path is absolute, empty, non-string, or resolves outside the destination (through..components or symlinks) are skipped with a warning, imported files are limited to*.envto match what configuration export produces, and the number of files actually restored is reported. Thanks to @zaara2004 for reporting this via email #1147/mapi/opensearch/, which matched nothing in the RBAC path table and therefore fell through to the default allow, letting any authenticated user reach cluster settings, snapshots, index deletion, and mapping changes. Access is now split into a reader tier holding a strict allowlist of method-independent query endpoints (_search,_msearch,_count,_field_caps,_validate/query), available to the same roles that can already read data through Dashboards, and an admin-only tier for everything else including scroll and point-in-time context management #1096latesttags, branch URLs, "latest release" lookups) and without verifying them. Release binaries (yq, supercronic, tini, evtx, the Arkime package and ja4plus plugin, pip for NetBox) now install through a sharedinstall-verified-binary.shhelper that checks each download against a SHA-256 digest recorded inshared/checksums/binaries.sha256; source archives (ECS, freq, htadmin, filepond-server-php, openresty, the NetBox device-type library and importer, the nginx LDAP auth module) are pinned to commits and verified before extraction; the nginx landing page's third-party assets and htadmin's web assets are checksummed; theepel-releasepackage is verified against a vendored EPEL 9 signing key; the images Malcolm controls are pinned by digest; Zeek packages are resolved to commits recorded in a lock file that the build installs from; CI dependencies and every GitHub Action are pinned; and downloads made by scripts outside the Dockerfiles are verified. A few sources intentionally continue to track upstream, because staying current is the point of fetching them at build time: reference data such as the IANA address registry, Wireshark's manufacturer database, and YARA rules, plus the Raspberry Pi kernel branch and upstream base images. Users who need hardened, verified builds today can continue to use Malcolm's Iron Bank images #1138lua-resty-openidcfrom 1.7.5 to 1.9.3, which terminates Keycloak authentication for the entire reverse proxy and was roughly two years behind. The upgrade forceslua-resty-sessionto 4.2.0, whose flattened option schema silently ignores unknown keys, so the session options were rewritten to 4.x flat keys rather than carrying forward a 3.x-shaped table that would have dropped theSecureflag from the session cookie without logging anything.accept_unsupported_alg = falsewas also set, since the default is permissive and would otherwise accept an unverified payload for algorithms Keycloak advertises butlua-resty-jwtcannot verify. A session secret is now set explicitly throughNGINX_SESSION_SECRETand generated and persisted when the operator hasn't supplied one, because 4.x otherwise derives a per-worker secret that invalidates every live session on container restart #1269NGINX_AUTH_MODE=keycloak_remoteandKEYCLOAK_SSL_VERIFYis nottrue, and the authentication documentation explains the setting, recommends enabling it, and describes usingnginx/ca-trust/for certificates not issued by a publicly trusted CA GHSA-5wxm-fqf2-r23g🐛 Bug fixes
Intel::read_fileshave finished loading, so packets at the start of a PCAP (or all of a small one) were evaluated against an empty or partially loaded intel store, producingconn.loganddns.logentries normally but nointel.loghits. This got worse as intel feeds grew. Packet processing is now suspended until every existing intel file finishes its initial read, and resumed immediately if a file fails outright so the PCAP is still processed rather than left suspended. Live capture is unaffected #1260pcap-filerun, so rules parsed at startup never carried it: across 106 uploaded-PCAPeve.jsonfiles, none of 33,303 alerts hadalert.metadata, including 5,346 alerts from Emerging Threats rules that all carry it. Each instance is now warmed up with a minimal one-packet PCAP and given a blocking rule reload on first connection, once per instance start. As a result, the existing Logstash mappings from[suricata][alert][metadata]populate for uploaded PCAP for the first time, including ET'smitre_*metadata intothreat.tactic.*andthreat.technique.*andcvemetadata intovulnerability.id, normalized to the standardCVE-YYYY-NNNNform from ET's underscored, prefix-less, lowercase, and multiple-ID-per-value variants.affected_productandattack_targetare mapped tovulnerability.categoryfor rules that also carrycvemetadata, with the raw values kept on every alert undersuricata.alert.metadata.*. Separately, scheduled rule updates reloaded only the live instance's socket, so the PCAP-processing instances kept the ruleset they started with until the container restarted; everysuricata-command*.socketis now reloaded. Reported upstream to OISF as #9252 #1262dashboardscontainer exiting withSocket timeouta few minutes after startup, most reliably whiledashboards-helperis creating index patterns and importing saved objects. This is an upstream OpenSearch Dashboards bug (opensearch-project/OpenSearch-Dashboards#12791) in which the legacy OpenSearch client leaks a connection after a request has already succeeded; the leaked connection times out two minutes later and nothing handles the error, so the server exits. Two small Node.js preload scripts are loaded throughNODE_OPTIONS: one prevents the connection leak, and one catches a socket timeout if it still happens and logs it rather than letting Dashboards crash. Both are temporary and should be removed once the upstream fix ships in a release Malcolm moves to #1221watch_common.pycould not tell a reader's open from a writer's, and every file processed bypcap_watcher.pygets read immediately afterward by the watcher's own libmagic calls and then by Arkime, Zeek, and Suricata. A processed file went back into the tracking deck with a history that could never qualify for processing, was logged as⏿ checkingevery few seconds until deleted, and stopped the deck being ordered oldest-first so files queued behind it were delayed. A file moved intoprocessed/under a name already seen, or opened by anything during its quiet period, was never processed at all and logged nothing. Read-only closes now retract one recorded open rather than being treated as activity, and each open is recorded separately so overlapping readers and a writer still holding the file are counted independently #1265sincewas parsed but never sent to the server, so every run downloaded the entire collection; and collections reported as unreadable were requested anyway. Compound STIX patterns were also handled incorrectly in both directions: originally anANDwithin one observation became independent Zeek entries, so a file name matched any file with that name, and the first fix for that rejected such patterns outright, discarding hashes and addresses that match safely on their own. Patterns are now normalized to disjunctive normal form and each clause judged separately, with anANDcontaining a file or certificate hash reduced to the hash, andORbranches that can't be expressed dropped while the others are kept. Escaped apostrophes and backslashes in STIX string literals are decoded rather than left escaped. On the MISP side, the attributes of an object describing one thing are reduced to its hashes rather than each becoming an independently matching item, compositefilename|hashattributes contribute only the hash, and attributes their authors markedto_ids: falseare skipped by default #1143, #1148, #1149, #1150, #1158, #1144, #1161mktime(value.timetuple()), which interprets their clock fields as local time and drops fractional seconds, someta.firstseen,meta.lastseen, and the corresponding CIF fields could disagree with the original indicator by hours. MISP manifest filtering had the same problem plus an undefined fallback that caused events without atimestampto be skipped entirely, including when no cutoff was requested #1139, #1141, #1140/fieldscatalogue, which previously stopped at the parent entry and therefore omitted fields such asfilescan.rules.scanner,suricata.alert.severity, anddestination.device.idthat hadn't yet appeared in indexed documents #1152, #1175opensearch_status.sh -wcould report that logs exist when an index's document count was absent, null, or zero #1151, #1170, #1232FileMovedEventonly retired the tracked source path when source and destination shared a parent directory, leaving stale names in the queue; and PCAPNG uploads acquired an unintendedpcapngmetadata tag because the extension was missing from the format-token exclusions #1145, #1153, #1146, #1238, #1240(group, name), either of which could report a failed process as healthy; cancelling queued scan workers raised unowned-token errors and left queued coroutines unstarted and unclosed; cancelling a Strelka upload could block the event loop on a pending future; a nonpositive configured chunk size could drop file content or produce endless empty chunks; normalizing an already-normalized Redis alias raisedAssertionErrorand prevented subscriber handlers from being constructed; and finite task groups never returned, because their permanent signal listener sat in the same group as the finite work #1159, #1163, #1164, #1169, #1171, #1176, #1177AttributeErrorthat aborted log cleanup before it could process eligible logs; a losing contender for the cleanup lock unlinked the winner's locked inode, letting a third runner create a new inode at the same path and enter cleanup concurrently; an uploaded Zeek archive whose basename matched an earlier one silently replaced it;safe-extract.pyleft partially populated output behind when extraction failed for any reason other than an archive bomb, which the caller could mistake for completed input; andpackage_zeek_logs.shsplitfindoutput on whitespace, mishandling paths containing spaces, tabs, or newlines #1157, #1165, #1239, #1241, #1233rgsearch when no package locations were known, which could fall back to scanning the working directory, and registering plugins into a function-localPLUGINSassignment rather than the application's global setting, leaving the plugin unregistered. Discovery results are now sorted and deduplicated so the generated order is stable across runs #1247ssl.logalongside TLS; the transport is now derived from the version. MQTT logs were likewise labeled UDP rather than TCP #1119include:keys at the same level, which is deprecated as of Suricata 8, by emitting a single array as the last key in the fileprune_files.shusing strict comparisons against the configured disk-usage percentage and path-size thresholds, so usage exactly at a limit never triggered cleanup or stopped it early, and failing on prune paths containing spaces #1231joinon a list and therefore failed on every timestamped JSON and plain-text line #1243scripts/build.shdefects found while moving credentials to BuildKit secrets: a doubled=in theCONFIG_ENV_DIRassignment, reading MaxMind values fromarkime-secret.envwithout testing that it exists, and a bad substitution in the<runtime>-composefallback. Incontrol.py, the fallback to the examples directory looked for.envrather than.env.example, and a mapped value of empty string was treated as absent during environment variable migration #1134, #1133✅ Component version updates
files.logno longer hasftime, and inintel.logthe extendedfile_descriptioncolumn is nowfile_descwhileassociatedandkill_chain_phasesare mapped intothreat.group.nameandthreat.tactic.*rather than kept aszeek.intel.*fieldsmulticast_participants.log,ntp_control.log, andntp_private.loglua-resty-openidcto v1.9.3, withlua-resty-sessionto v4.2.0 and newlua-resty-opensslv1.8.0 andlua-ffi-zlibv0.6 dependencieslogstash-output-opensearchplugin pinned to v2.1.1python-rapidjsonto v1.25🧹 Code and project maintenance
shared/tests/. A sharedconftest.pynow puts each component's source directory on the path, so a test can import what it needs by name and a singlepytest -q shared/testsrun collects everything. Test coverage grew substantially over the course of this release alongside the fixes above.📄 Configuration changes for Malcolm (in environment variables in
./config/). The Malcolm control script (e.g.,./scripts/status,./scripts/start) automatically handles creation and migration of variables according to./config/env-var-actions.yml.LOGSTASH_ZEEK_COMPUTE_COMMUNITY_IDinlogstash.envcontrols whether Community ID is computed in Logstash for Zeek logs that arrive without one (defaulttrue)NGINX_SESSION_SECRETinkeycloak.envsets the OIDC session secret used by the Keycloak authentication modes. It is generated and persisted automatically when left blank, though a value set here is the supported way to keep logins alive across an nginx restartNETBOX_DEVICETYPE_IMPORT_VENDORSandNETBOX_DEVICETYPE_IMPORT_FORCEinnetbox.envlimit the device-type library import to selected manufacturer slugs and force a re-import of a library revision that was already importedARKIME_WISE_CONFIG_PIN_CODEinarkime-secret.envnow ships empty; installations still holding the former shipped default are migrated to empty on upgrade, which causes the Arkime container to generate a random PIN at startupMALCOLM_API_HTTP_CONNECT_TIMEOUT_SEC(default 5),MALCOLM_API_HTTP_READ_TIMEOUT_SEC(default 30), andMALCOLM_API_READY_TIMEOUT_SEC(default 5) bound the API's outbound requests and each individual readiness checkSURICATA_SOCKET_WARMUP_TIMEOUT_SEC(default 60) andSURICATA_SOCKET_RULE_RELOAD_TIMEOUT_SEC(default 900) bound the warm-up and rule reload performed once per Suricata unix-socket instance startSENSITIVE_COUNTRY_CODESinlookup-common.envaddsKR, following the U.S. Department of Energy Sensitive Country ListZEEK_PORTS_BACNET,ZEEK_PORTS_C1222,ZEEK_PORTS_ENIP,ZEEK_PORTS_GE_SRTP,ZEEK_PORTS_GENISYS,ZEEK_PORTS_HART_IP,ZEEK_PORTS_MODBUS,ZEEK_PORTS_OMRON_FINS,ZEEK_PORTS_OPCUA_BINARY,ZEEK_PORTS_PROFINET_IO_CM,ZEEK_PORTS_ROC_PLUS,ZEEK_PORTS_S7COMM,ZEEK_PORTS_SYNCHROPHASORinzeek.envused to specify non-default ports to register certain Zeek analyzers formatted as a comma-separated list of Zeek ports (e.g.,12345/tcpor4041/tcp,4042/udp)ZEEK_SYNCHROPHASOR_PORTS,ZEEK_GENISYS_PORTS, andZEEK_ENIP_PORTShave been renamed toZEEK_PORTS_SYNCHROPHASOR,ZEEK_PORTS_GENISYS, andZEEK_PORTS_ENIP; any values set are carried over automatically⚰️ Breaking changes and removed or deprecated functionality
BEATS_SSL=true, rather than accepting a forwarder that presents no certificate. Forwarders configured with the client certificate generated by./scripts/auth_setup(infilebeat/certs/), including Hedgehog Linux sensors and Malcolm's internal Filebeat, are unaffected. Custom forwarders that connectED without a client certificate will need one.to_idsflag isfalseare now skipped by default, since their authors use that flag to mark values that shouldn't be used for detection. Measured against six public MISP feeds over two months, this removed about 4.5% of items (9,630 of 211,705), mostly reassignable payload host addresses. To restore the previous behavior, setrequire_to_ids: Falsefor a feed inmisp.yaml.ARKIME_WISE_CONFIG_PIN_CODEand the Arkime viewer password secret no longer ship with default values. Deployments that relied on the documented WISE PIN will need to set one explicitly through./scripts/auth_setup, or read the randomly generated value from the Arkime container.