Skip to content

Malcolm v26.10.0 - #1275

Merged
mmguero merged 218 commits into
mainfrom
v26.10.0_merge_cisagov
Oct 29, 2026
Merged

mmguero merged 218 commits into
mainfrom
v26.10.0_merge_cisagov

Conversation

@mmguero

@mmguero mmguero commented Oct 29, 2026

Copy link
Copy Markdown
Collaborator

Malcolm v26.10.0 updates Zeek to 9.0.1 LTS, bringing new log types (including multicast participants and the NTP control and private modes) and a corresponding Multicast dashboard; substantially expands threat intelligence enrichment, mapping Zeek intel, Suricata rule metadata, and capa results into normalized threat.* fields with MITRE ATT&CK tactic, technique, and subtechnique context; normalizes Zeek OT protocol fields to their Arkime equivalents for eight protocols so filters and drill-downs carry across Dashboards and Arkime; adds optional Community ID computation in Logstash for Zeek logs that arrive without one; and extends the Malcolm API's document lookup with sorting, paging, total hit counts, and backend search status, so a partial or timed-out search can no longer look like a normal empty result. Uploaded PCAP processing received two significant correctness fixes: Zeek now waits for its intelligence files to finish loading before processing packets, and Suricata's unix-socket instances are warmed up and reloaded so alerts carry rule metadata. A large supply-chain hardening effort pins and verifies nearly every external artifact consumed during the build. OpenSearch and OpenSearch Dashboards, Arkime, Keycloak, Logstash, Filebeat, NetBox, and other components have been updated. Many other bug fixes and general improvements are also included.

This release also fixes nine security issues, the most significant being stored cross-site scripting through the file-upload container, which allowed a user with upload access to run arbitrary JavaScript in another Malcolm user's session, and credential disclosure through the Hedgehog extracted-files proxy, which forwarded the caller's headers to a destination the caller names.

The Malcolm team would like to thank @sylvesterkaczmarek, @pavanchow, @Darshan060224, @sercanokur, @nagameTW, @katsugtgz, and @zaara2004 for their contributions to this release.

If you are upgrading from an existing Malcolm installation, run ./scripts/status for Malcolm to migrate some settings prior to running ./scripts/configure, ./scripts/start, or other Malcolm control scripts.

v26.09.0...v26.10.0

  • ✨ Features and enhancements

    • Update Zeek to 9.0.1 LTS, adding support for its new and changed log types: multicast_participants.log is parsed, mapped, and surfaced in a new Multicast dashboard, and ntp_control.log and ntp_private.log are parsed and added to the index template and Arkime field definitions #1119
    • Normalize Zeek OT protocol fields to Arkime's field names for BACnet, C12.22, DNP3, EtherNet/IP (ENIP/CIP), Modbus, OPC UA, S7comm, and Synchrophasor, so filters and drill-downs carry across Dashboards and Arkime rather than stopping at the tool boundary. Original zeek.* fields are preserved; the new top-level fields are added alongside them, with Modbus function and exception names converted to their numeric codes to match Arkime's integer typing. The OT dashboards and the OpenSearch composable template were updated to match #993, #1257, #986
    • Expand threat intelligence enrichment into normalized ECS threat.* fields across every source that provides it: Zeek intel kill chain phases become threat.tactic.*, MISP MITRE galaxy tags in cif_tags become threat.technique.* and threat.technique.subtechnique.*, Mandiant and Google Threat Intelligence actors become threat.group.name, and Suricata's Emerging Threats mitre_* metadata and capa's subtechnique results are mapped the same way. Tactic and technique names come from generated ATT&CK dictionaries so a given ID always resolves to one name, with reference URLs added and ATT&CK for ICS distinguished from Enterprise. The Threat Intelligence dashboard, Arkime field definitions, and the WISE field list were updated accordingly #695
    • Carry additional per-indicator intelligence metadata into Zeek intel: Mandiant categories, campaigns, reports, and the opt-in threat rating (meta.threat_score), and STIX 2.1 confidence values. Mandiant's M-Score is treated as confidence rather than severity, and absent values are left unset rather than invented #1185, #1190, #1201, #1244
    • Add ioc_types and include_autogenerated_tags parameters for Google Threat Intelligence collections, so a deployment can load only the indicator types it needs and leave out machine-generated tags; both reduce the memory a large collection costs in Zeek
    • Add optional Community ID computation in Logstash for Zeek logs that Zeek didn't already tag with one, controlled by LOGSTASH_ZEEK_COMPUTE_COMMUNITY_ID (enabled by default). An ID Zeek supplied is never overwritten; for logs with no proto column, the transport is inferred by log type only where the analyzer runs over a single transport #400
    • Extend the API's document lookup with sort, offset, and track_total_hits arguments and a total hit count with its exact-or-lower-bound relation, and return the backend's shards and timed_out status from both document lookup and field aggregations, so a client can tell a partial or timed-out search from a genuinely empty one. Argument validation errors now return HTTP 400 with an error string, and MALCOLM_API_DEBUG=true logs a compact per-request search summary #1130, #1131
    • Document PCAPNG as supported, now that Arkime's scheme reader is the default and handles it alongside Zeek and Suricata. Some link types, including IEEE 802.11, are still not fully decoded by Arkime #407
    • Prompt during Hedgehog Linux ISO installation whether to run the sensor status kiosk at startup, defaulting to yes. Answering no disables kiosk.service and removes the browser autostart entry; the question never appears on Malcolm installs #1264
    • Skip the NetBox device-type library import on restart once it has completed for a given library revision, using a marker stored in NetBox's own database rather than container-local storage, and add NETBOX_DEVICETYPE_IMPORT_VENDORS to limit the first import to selected manufacturers. Note that limiting vendors also limits which manufacturers can be matched during MAC OUI enrichment #1196
    • Warn during configuration when the full Malcolm stack is being set up on a local host with clearly less than Malcolm's documented 24 GiB minimum RAM, including in noninteractive installs, and note on Linux when no swap is configured. The warning is informational and changes no settings; it is skipped for capture-only Hedgehog and Kubernetes deployments, where the installer's reading of host memory doesn't describe the actual workload #1242, #483
    • Bring the local account management interface's header, footer, navigation, and styling in line with Malcolm's landing and extracted-files pages, layered after htadmin's own Bootstrap styles rather than vendoring the upstream application #1216, #531
    • Minor improvements to the Biomes dashboard, which is now documented in Analytics
    • Allow non-default ports to be registered for thirteen ICSNPP OT protocol analyzers through new ZEEK_PORTS_… environment variables: BACnet, C12.22, EtherNet/IP, GE-SRTP, Genisys, HART-IP, Modbus, Omron FINS, OPC UA Binary, Profinet IO-CM, ROC+, S7comm, and Synchrophasor. TCP and UDP entries are applied to whichever transports each analyzer supports, so a site running one of these protocols on a nonstandard port no longer needs a custom Zeek script to parse it. The three variables that already existed are renamed into this pattern (ZEEK_SYNCHROPHASOR_PORTS, ZEEK_GENISYS_PORTS, and ZEEK_ENIP_PORTS become ZEEK_PORTS_SYNCHROPHASOR, ZEEK_PORTS_GENISYS, and ZEEK_PORTS_ENIP), and existing values are migrated on upgrade #1272, #1273
  • 🛡️ Security Remediation & Hardening

    • Fix stored cross-site scripting through the file-upload container, where uploaded HTML and SVG files could be read back on the Malcolm origin as active content: the responses carried a content-derived Content-Type, Content-Disposition: inline on the FilePond PHP routes, no X-Content-Type-Options: nosniff, and a CSP with no script-src or default-src, so an inline script ran with the session of whoever opened the URL. Uploaded content was reachable through several static paths (including two that bypassed the location guarding the upload directory) and through FilePond's restore and load routes. Triage also found that the chunked-upload patch route accepted an unvalidated transfer ID, allowing files to be written into any directory writable by the web server user while bypassing the extension deny-list. Since Malcolm's upload page never reads uploads back, the read side is removed entirely: the upload container's nginx returns 404 by default and serves only the upload page, its static assets, and the two PHP entrypoints; those entrypoints accept only POST and DELETE; the restore, load, and patch handlers are stripped from the vendored PHP at build time with the build failing if any remain; and PHP responses now carry nosniff and default-src 'none'; sandbox GHSA-qv8w-m37v-6c7j
    • Fix credential and identity disclosure in the hh-extracted-files proxy, which built its upstream from a caller-supplied path segment and forwarded the caller's Authorization header, session Cookie, and the X-Forwarded-User, X-Forwarded-Groups, and X-Forwarded-Roles headers written by Malcolm's own auth layer to that destination. All five headers are now cleared before the request is proxied, in both nginx.conf and nginx_readonly.conf GHSA-325g-72fr-vj2x
    • Remove the shipped default Arkime WISE configuration PIN (WISE2019), which was published in both the example configuration and the documentation. ARKIME_WISE_CONFIG_PIN_CODE now ships empty, the Arkime container generates a random PIN when none is configured, existing installs still holding the known value are migrated to empty on upgrade, and ./scripts/auth_setup prompts for it (with --auth-arkime-wise-pin for noninteractive setup). The shipped default for the Arkime viewer password secret was removed in the same pass GHSA-w2jh-h3pq-273j #1133
    • Stop passing MaxMind and GitHub API token as Docker build arguments, which recorded them in image metadata and exposed them in process arguments during the build. Both are now mounted as BuildKit secrets in the GitHub Actions workflows and in scripts/build.sh, and the ISO and Raspberry Pi build scripts write environment.chroot with restrictive permissions and register cleanup before the write. While reviewing this, the MaxMind build arguments were found to have been declared in the Arkime image's final stage, recording their values in the layer history of published images; those credentials have been rotated, and the new images were verified to contain no credential values GHSA-3293-9mf2-qf68 #1134
    • Require client certificate authentication on Logstash's Beats input, which previously treated certificates as optional under BEATS_SSL=true and accepted a forwarder that presented none. Forwarders using the client certificate generated by ./scripts/auth_setup, including Hedgehog Linux sensors and Malcolm's internal Filebeat, are unaffected GHSA-f2gq-m4v3-2383 #1136
    • Stop following symlinks in the PCAP upload watcher, which runs as root and checked and changed ownership of new uploads by path in a directory writable through the upload service's SFTP server. Uploads are now inspected with lstat(), with symlinks and other non-regular entries removed rather than processed, opened once with O_NOFOLLOW, and handled entirely through that file descriptor GHSA-98jw-5wq7-6qjv #1137
    • Bound every outbound HTTP request made by the Malcolm API, which previously had no timeout, so a dependency that accepted a connection and stopped responding could occupy the API's single sync worker and take down the whole API including its /mapi/ping healthcheck. Timeouts are configurable through MALCOLM_API_HTTP_CONNECT_TIMEOUT_SEC and MALCOLM_API_HTTP_READ_TIMEOUT_SEC, each /mapi/ready check is bounded by MALCOLM_API_READY_TIMEOUT_SEC and the checks now run concurrently, Valkey connections set a socket timeout, and the API runs with multiple gunicorn workers and threads under both Docker Compose and Kubernetes GHSA-x9f2-h2rj-vp63 #1135
    • Tighten permissions on nginx/htpasswd and htadmin/metadata from 0644 to 0640, so web authentication hashes are no longer readable by other local users when the parent directories permit traversal. Because scripts/control.py applies this at startup, existing installations are corrected the next time Malcolm starts GHSA-rxpm-f8cm-2m5m
    • Harden temporary-file handling in the ISO installation hook, which created and truncated predictable paths in /tmp as root. The debconf template is now written to a private mktemp -d directory with cleanup on exit and on signal, and the unused answer-file writes were removed GHSA-wmrp-jr97-8xf4 #1132
    • Record who submitted each document created through the API's /mapi/event endpoint in a new event.submitter field, set after the submitted body is merged so a caller cannot supply it, and namespace caller-supplied document IDs as <date>-alert.<id>. Arkime session IDs and Zeek UIDs never contain a ., so a caller can no longer choose an ID that collides with, and overwrites, a record written by Malcolm's own pipeline
    • Restrict files restored by --import-malcolm-config-file to the configuration directory: entries whose path is absolute, empty, non-string, or resolves outside the destination (through .. components or symlinks) are skipped with a warning, imported files are limited to *.env to match what configuration export produces, and the number of files actually restored is reported. Thanks to @zaara2004 for reporting this via email #1147
    • Add role-based access control to the OpenSearch reverse proxy at /mapi/opensearch/, which matched nothing in the RBAC path table and therefore fell through to the default allow, letting any authenticated user reach cluster settings, snapshots, index deletion, and mapping changes. Access is now split into a reader tier holding a strict allowlist of method-independent query endpoints (_search, _msearch, _count, _field_caps, _validate/query), available to the same roles that can already read data through Dashboards, and an admin-only tier for everything else including scroll and point-in-time context management #1096
    • Reduce supply-chain exposure across the build. Malcolm's Dockerfiles, image builders, CI workflows, and setup scripts previously fetched some external images, source, packages, and binaries by mutable reference (latest tags, branch URLs, "latest release" lookups) and without verifying them. Release binaries (yq, supercronic, tini, evtx, the Arkime package and ja4plus plugin, pip for NetBox) now install through a shared install-verified-binary.sh helper that checks each download against a SHA-256 digest recorded in shared/checksums/binaries.sha256; source archives (ECS, freq, htadmin, filepond-server-php, openresty, the NetBox device-type library and importer, the nginx LDAP auth module) are pinned to commits and verified before extraction; the nginx landing page's third-party assets and htadmin's web assets are checksummed; the epel-release package is verified against a vendored EPEL 9 signing key; the images Malcolm controls are pinned by digest; Zeek packages are resolved to commits recorded in a lock file that the build installs from; CI dependencies and every GitHub Action are pinned; and downloads made by scripts outside the Dockerfiles are verified. A few sources intentionally continue to track upstream, because staying current is the point of fetching them at build time: reference data such as the IANA address registry, Wireshark's manufacturer database, and YARA rules, plus the Raspberry Pi kernel branch and upstream base images. Users who need hardened, verified builds today can continue to use Malcolm's Iron Bank images #1138
    • Upgrade OpenResty from v1.31.1.1 to v1.31.8.1 which incorporates 12 CVE fixes from Nginx 1.31.2 through 1.31.8.
    • Upgrade lua-resty-openidc from 1.7.5 to 1.9.3, which terminates Keycloak authentication for the entire reverse proxy and was roughly two years behind. The upgrade forces lua-resty-session to 4.2.0, whose flattened option schema silently ignores unknown keys, so the session options were rewritten to 4.x flat keys rather than carrying forward a 3.x-shaped table that would have dropped the Secure flag from the session cookie without logging anything. accept_unsupported_alg = false was also set, since the default is permissive and would otherwise accept an unverified payload for algorithms Keycloak advertises but lua-resty-jwt cannot verify. A session secret is now set explicitly through NGINX_SESSION_SECRET and generated and persisted when the operator hasn't supplied one, because 4.x otherwise derives a per-worker secret that invalidates every live session on container restart #1269
    • Make the risk of disabling remote Keycloak TLS certificate verification visible rather than changing the default, since Malcolm is commonly deployed against self-signed or internally issued certificates and a default that fails on first use would mostly lead operators to disable verification without understanding why. The nginx container now logs a warning at startup when NGINX_AUTH_MODE=keycloak_remote and KEYCLOAK_SSL_VERIFY is not true, and the authentication documentation explains the setting, recommends enabling it, and describes using nginx/ca-trust/ for certificates not issued by a publicly trusted CA GHSA-5wxm-fqf2-r23g
  • 🐛 Bug fixes

    • Fix Zeek missing intelligence hits when processing uploaded PCAP. When reading a trace, Zeek begins processing packets before the files in Intel::read_files have finished loading, so packets at the start of a PCAP (or all of a small one) were evaluated against an empty or partially loaded intel store, producing conn.log and dns.log entries normally but no intel.log hits. This got worse as intel feeds grew. Packet processing is now suspended until every existing intel file finishes its initial read, and resumed immediately if a file fails outright so the PCAP is still processed rather than left suspended. Live capture is unaffected #1260
    • Fix Suricata rule metadata never reaching alerts for uploaded PCAP. In unix-socket mode Suricata only begins building rule metadata once the alert output has been set up for a pcap-file run, so rules parsed at startup never carried it: across 106 uploaded-PCAP eve.json files, none of 33,303 alerts had alert.metadata, including 5,346 alerts from Emerging Threats rules that all carry it. Each instance is now warmed up with a minimal one-packet PCAP and given a blocking rule reload on first connection, once per instance start. As a result, the existing Logstash mappings from [suricata][alert][metadata] populate for uploaded PCAP for the first time, including ET's mitre_* metadata into threat.tactic.* and threat.technique.* and cve metadata into vulnerability.id, normalized to the standard CVE-YYYY-NNNN form from ET's underscored, prefix-less, lowercase, and multiple-ID-per-value variants. affected_product and attack_target are mapped to vulnerability.category for rules that also carry cve metadata, with the raw values kept on every alert under suricata.alert.metadata.*. Separately, scheduled rule updates reloaded only the live instance's socket, so the PCAP-processing instances kept the ruleset they started with until the container restarted; every suricata-command*.socket is now reloaded. Reported upstream to OISF as #9252 #1262
    • Fix the dashboards container exiting with Socket timeout a few minutes after startup, most reliably while dashboards-helper is creating index patterns and importing saved objects. This is an upstream OpenSearch Dashboards bug (opensearch-project/OpenSearch-Dashboards#12791) in which the legacy OpenSearch client leaks a connection after a request has already succeeded; the leaked connection times out two minutes later and nothing handles the error, so the server exits. Two small Node.js preload scripts are loaded through NODE_OPTIONS: one prevents the connection leak, and one catches a socket timeout if it still happens and logs it rather than letting Dashboards crash. Both are temporary and should be removed once the upstream fix ships in a release Malcolm moves to #1221
    • Fix the PCAP watchers treating read-only access to a watched file as activity on it. watch_common.py could not tell a reader's open from a writer's, and every file processed by pcap_watcher.py gets read immediately afterward by the watcher's own libmagic calls and then by Arkime, Zeek, and Suricata. A processed file went back into the tracking deck with a history that could never qualify for processing, was logged as ⏿ checking every few seconds until deleted, and stopped the deck being ordered oldest-first so files queued behind it were delayed. A file moved into processed/ under a name already seen, or opened by anything during its quiet period, was never processed at all and logged nothing. Read-only closes now retract one recorded open rather than being treated as activity, and each open is recorded separately so overlapping readers and a writer still holding the file are counted independently #1265
    • Fix several defects in STIX, TAXII, and MISP intelligence conversion found while reviewing the threat feed code: TAXII 2.1 feeds produced no Zeek intel at all, because each page arrives as an envelope rather than a STIX bundle and failed to parse; whole-bundle validation meant one malformed object discarded every indicator in a file, so none of CISA's AA26-204A indicators loaded; since was parsed but never sent to the server, so every run downloaded the entire collection; and collections reported as unreadable were requested anyway. Compound STIX patterns were also handled incorrectly in both directions: originally an AND within one observation became independent Zeek entries, so a file name matched any file with that name, and the first fix for that rejected such patterns outright, discarding hashes and addresses that match safely on their own. Patterns are now normalized to disjunctive normal form and each clause judged separately, with an AND containing a file or certificate hash reduced to the hash, and OR branches that can't be expressed dropped while the others are kept. Escaped apostrophes and backslashes in STIX string literals are decoded rather than left escaped. On the MISP side, the attributes of an object describing one thing are reduced to its hashes rather than each becoming an independently matching item, composite filename|hash attributes contribute only the hash, and attributes their authors marked to_ids: false are skipped by default #1143, #1148, #1149, #1150, #1158, #1144, #1161
    • Fix threat-intelligence timestamps shifting with the importing host's timezone. The Mandiant, STIX, and MISP mappings converted timezone-aware datetimes through mktime(value.timetuple()), which interprets their clock fields as local time and drops fractional seconds, so meta.firstseen, meta.lastseen, and the corresponding CIF fields could disagree with the original indicator by hours. MISP manifest filtering had the same problem plus an undefined fallback that caused events without a timestamp to be skipped entirely, including when no cutoff was requested #1139, #1141, #1140
    • Fix API time ranges shifting with the host's timezone despite documented UTC defaults, and include descendants of object and nested mappings in the /fields catalogue, which previously stopped at the parent entry and therefore omitted fields such as filescan.rules.scanner, suricata.alert.severity, and destination.device.id that hadn't yet appeared in indexed documents #1152, #1175
    • Fix OpenSearch index retention accounting. Distinct index patterns that resolve to the same concrete index were counted once per pattern, inflating reported usage and producing repeated deletion candidates, so pruning could trigger while actual usage was below the configured limit. Candidate sizes were also rounded down to whole megabytes before being subtracted from the deletion budget, so every sub-megabyte index contributed zero and the loop could select all of them. Each concrete index is now counted once, and exact byte values are requested and kept fractional during selection. Separately, opensearch_status.sh -w could report that logs exist when an index's document count was absent, null, or zero #1151, #1170, #1232
    • Fix several defects in PCAP upload and capture handling: duplicate detection was silently disabled whenever the startup health wait was turned off, so an already-processed capture was republished, and a failed duplicate lookup now logs and publishes rather than dropping the capture; the configured sensor name was interpolated into a regular expression, so a name containing regex characters could match another sensor's records or fail to match its own; a FileMovedEvent only retired the tracked source path when source and destination shared a parent directory, leaving stale names in the queue; and PCAPNG uploads acquired an unintended pcapng metadata tag because the extension was missing from the format-token exclusions #1145, #1153, #1146, #1238, #1240
    • Fix several latent defects in the file scanning pipeline: the health endpoint compared lowercase Supervisor states against uppercase sets so its error-state and exit-code branches were never taken, and paired configuration with processes by list position rather than by (group, name), either of which could report a failed process as healthy; cancelling queued scan workers raised unowned-token errors and left queued coroutines unstarted and unclosed; cancelling a Strelka upload could block the event loop on a pending future; a nonpositive configured chunk size could drop file content or produce endless empty chunks; normalizing an already-normalized Redis alias raised AssertionError and prevented subscriber handlers from being constructed; and finite task groups never returned, because their permanent signal listener sat in the same group as the finite work #1159, #1163, #1164, #1169, #1171, #1176, #1177
    • Fix Zeek log and archive handling in Filebeat: a one-entry Filebeat registry parses as a single object rather than a list, and iterating its keys raised an AttributeError that aborted log cleanup before it could process eligible logs; a losing contender for the cleanup lock unlinked the winner's locked inode, letting a third runner create a new inode at the same path and enter cleanup concurrently; an uploaded Zeek archive whose basename matched an earlier one silently replaced it; safe-extract.py left partially populated output behind when extraction failed for any reason other than an archive bomb, which the caller could mistake for completed input; and package_zeek_logs.sh split find output on whitespace, mishandling paths containing spaces, tabs, or newlines #1157, #1165, #1239, #1241, #1233
    • Fix NetBox prefix enrichment treating the first and last address of every subnet as unassignable, which is only true for prefixes of /30 or shorter, and misreading a site name containing a colon as a CIDR list #1125
    • Fix NetBox plugin discovery launching an unscoped rg search when no package locations were known, which could fall back to scanning the working directory, and registering plugins into a function-local PLUGINS assignment rather than the application's global setting, leaving the plugin unregistered. Discovery results are now sorted and deduplicated so the generated order is stable across runs #1247
    • Fix DTLS sessions being recorded with a TCP transport, since Zeek logs them to ssl.log alongside TLS; the transport is now derived from the version. MQTT logs were likewise labeled UDP rather than TCP #1119
    • Fix Suricata configuration generation writing repeated include: keys at the same level, which is deprecated as of Suricata 8, by emitting a single array as the last key in the file
    • Fix prune_files.sh using strict comparisons against the configured disk-usage percentage and path-size thresholds, so usage exactly at a limit never triggered cleanup or stopped it early, and failing on prune paths containing spaces #1231
    • Fix a crash in the container log formatter when a log line carried an ISO-8601 timestamp before its message, which called join on a list and therefore failed on every timestamped JSON and plain-text line #1243
    • Fix scripts/build.sh defects found while moving credentials to BuildKit secrets: a doubled = in the CONFIG_ENV_DIR assignment, reading MaxMind values from arkime-secret.env without testing that it exists, and a bad substitution in the <runtime>-compose fallback. In control.py, the fallback to the examples directory looked for .env rather than .env.example, and a mapped value of empty string was treated as absent during environment variable migration #1134, #1133
  • ✅ Component version updates

    • Zeek to v9.0.1 LTS
      • Several log formats changed with Zeek 9.x and Malcolm's parsing, index templates, and Arkime field definitions were updated to match: files.log no longer has ftime, and in intel.log the extended file_description column is now file_desc while associated and kill_chain_phases are mapped into threat.group.name and threat.tactic.* rather than kept as zeek.intel.* fields
      • New log types are parsed and indexed, including multicast_participants.log, ntp_control.log, and ntp_private.log
    • OpenSearch and OpenSearch Dashboards to v3.9.0
    • OpenResty to v1.31.8.1 (incorporates 12 CVE fixes from Nginx 1.31.2 through 1.31.8)
    • Arkime to v6.9.1
    • Keycloak to v26.8
    • Logstash to v9.5.5
    • Filebeat to v9.5.5
    • NetBox Python dependencies including GitPython to v3.2.1 (addressing GHSA-239g-whfq-7xj9, GHSA-g5vv-9gxw-82hx, GHSA-rw58-wc66-99g4, and GHSA-59cr-6r3x-644w), pillow to v13.0.0, setuptools to v84.1.0, and uv to v0.13.4 #1202
    • lua-resty-openidc to v1.9.3, with lua-resty-session to v4.2.0 and new lua-resty-openssl v1.8.0 and lua-ffi-zlib v0.6 dependencies
    • logstash-output-opensearch plugin pinned to v2.1.1
    • Elastic Common Schema (used for the Dashboards field catalogue) to v9.5.0
    • supercronic to v0.2.51
    • evtx (Windows Event Log parsing, used by Filebeat) to v0.13.0
    • htpdate (used by the ISO and Raspberry Pi images) to v2.0.2
    • grpcio and grpcio-tools (used by the file scanning pipeline) to v1.84.0
    • The API container's Python base image pinned to v3.14, and python-rapidjson to v1.25
  • 🧹 Code and project maintenance

    • Consolidate Malcolm's Python unit tests into shared/tests/. A shared conftest.py now puts each component's source directory on the path, so a test can import what it needs by name and a single pytest -q shared/tests run collects everything. Test coverage grew substantially over the course of this release alongside the fixes above.
    • Various documentation improvements.
  • 📄 Configuration changes for Malcolm (in environment variables in ./config/). The Malcolm control script (e.g., ./scripts/status, ./scripts/start) automatically handles creation and migration of variables according to ./config/env-var-actions.yml.

    • LOGSTASH_ZEEK_COMPUTE_COMMUNITY_ID in logstash.env controls whether Community ID is computed in Logstash for Zeek logs that arrive without one (default true)
    • NGINX_SESSION_SECRET in keycloak.env sets the OIDC session secret used by the Keycloak authentication modes. It is generated and persisted automatically when left blank, though a value set here is the supported way to keep logins alive across an nginx restart
    • NETBOX_DEVICETYPE_IMPORT_VENDORS and NETBOX_DEVICETYPE_IMPORT_FORCE in netbox.env limit the device-type library import to selected manufacturer slugs and force a re-import of a library revision that was already imported
    • ARKIME_WISE_CONFIG_PIN_CODE in arkime-secret.env now ships empty; installations still holding the former shipped default are migrated to empty on upgrade, which causes the Arkime container to generate a random PIN at startup
    • MALCOLM_API_HTTP_CONNECT_TIMEOUT_SEC (default 5), MALCOLM_API_HTTP_READ_TIMEOUT_SEC (default 30), and MALCOLM_API_READY_TIMEOUT_SEC (default 5) bound the API's outbound requests and each individual readiness check
    • SURICATA_SOCKET_WARMUP_TIMEOUT_SEC (default 60) and SURICATA_SOCKET_RULE_RELOAD_TIMEOUT_SEC (default 900) bound the warm-up and rule reload performed once per Suricata unix-socket instance start
    • SENSITIVE_COUNTRY_CODES in lookup-common.env adds KR, following the U.S. Department of Energy Sensitive Country List
    • ZEEK_PORTS_BACNET,ZEEK_PORTS_C1222,ZEEK_PORTS_ENIP,ZEEK_PORTS_GE_SRTP,ZEEK_PORTS_GENISYS,ZEEK_PORTS_HART_IP,ZEEK_PORTS_MODBUS,ZEEK_PORTS_OMRON_FINS,ZEEK_PORTS_OPCUA_BINARY,ZEEK_PORTS_PROFINET_IO_CM,ZEEK_PORTS_ROC_PLUS,ZEEK_PORTS_S7COMM,ZEEK_PORTS_SYNCHROPHASOR in zeek.env used to specify non-default ports to register certain Zeek analyzers formatted as a comma-separated list of Zeek ports (e.g., 12345/tcp or 4041/tcp,4042/udp)
      • ZEEK_SYNCHROPHASOR_PORTS, ZEEK_GENISYS_PORTS, and ZEEK_ENIP_PORTS have been renamed to ZEEK_PORTS_SYNCHROPHASOR, ZEEK_PORTS_GENISYS, and ZEEK_PORTS_ENIP; any values set are carried over automatically
  • ⚰️ Breaking changes and removed or deprecated functionality

    • Logstash's Beats input on port 5044 now requires client certificate authentication when BEATS_SSL=true, rather than accepting a forwarder that presents no certificate. Forwarders configured with the client certificate generated by ./scripts/auth_setup (in filebeat/certs/), including Hedgehog Linux sensors and Malcolm's internal Filebeat, are unaffected. Custom forwarders that connectED without a client certificate will need one.
    • MISP attributes whose to_ids flag is false are now skipped by default, since their authors use that flag to mark values that shouldn't be used for detection. Measured against six public MISP feeds over two months, this removed about 4.5% of items (9,630 of 211,705), mostly reassignable payload host addresses. To restore the previous behavior, set require_to_ids: False for a feed in misp.yaml.
    • ARKIME_WISE_CONFIG_PIN_CODE and the Arkime viewer password secret no longer ship with default values. Deployments that relied on the documented WISE PIN will need to set one explicitly through ./scripts/auth_setup, or read the randomly generated value from the Arkime container.

mmguero and others added 30 commits September 30, 2026 17:53
…s (3.9.0), and urllib3 (2.8.0)

Signed-off-by: Seth Grover <seth.d.grover@gmail.com>
Document lookups previously offered no control over result ordering or paging, and omitted the total hit count. Add optional sorting and offset paging to /mapi/document, along with total.value and total.relation. Callers can request exact counts through track_total_hits while existing requests retain the backend’s default counting behavior.

Both /mapi/document and /mapi/agg discarded shard status and timeout information, allowing incomplete searches to look like ordinary empty or short results. Return the full shard status and timed_out flag alongside available results. Preserve HTTP 200 for partial responses so clients can use the data and assess its completeness.

Validate the new arguments and return HTTP 400 for malformed input. Reject aggregation names that collide with response metadata. Add debug summaries with result counts and backend search status, log shard failure details when present, and remove duplicate aggregation argument logging.

Update the API documentation and examples, including outdated aggregation keys that used values instead of field names.

Validated against local OpenSearch with 138 passing checks and no failures in each of two runs, covering sorting on @timestamp and firstPacket. Checks exercised legacy requests, paging and count behavior, plus invalid inputs. Another 32 isolated scenarios checked response compatibility and metadata handling with simulated partial failures, timeouts, and missing metadata. Python syntax and all 13 JSON documentation examples passed validation. A Bash validation script was used locally to verify the results (see attachment to the bugs listed below).

Live fault injection, result-window rejection, and Elasticsearch verification remain outstanding.

Defer search_after because offset paging satisfies the current request. Reliable cursor paging needs a suitable unique tie-breaker across document types, and snapshot-consistent paging would also require point-in-time support. Existing backend result-window limits still apply.

Fixes #1130
Fixes #1131

Signed-off-by: Seth Grover <seth.d.grover@gmail.com>
Create the debconf template in a private mktemp directory and clean it up
on exit or interruption. Abort if directory creation fails.

Remove unused malcolm.answer writes.

Refs:

GHSA-wmrp-jr97-8xf4
https://chatgpt.com/c/6ac404ec-1ca8-832b-a722-3f612549dd32

Signed-off-by: Seth Grover <seth.d.grover@gmail.com>
ARKIME_WISE_CONFIG_PIN_CODE no longer ships with a default value. The
example env file leaves it empty, and the Arkime entrypoint generates a
random PIN at startup when none is set. Existing installs still using
WISE2019 are migrated to an empty value through env-var-actions.yml.

auth_setup now prompts for the WISE PIN and accepts
--auth-arkime-wise-pin in non-interactive mode. A blank answer leaves the
current value unchanged. --auth-arkime-password no longer defaults to
"Malcolm", and the Arkime secret values are only written when provided.

Fix two bugs in checkEnvFilesAndValues found while adding the migration:
value maps in copied_environment_variables dropped empty-string results
because of a truthiness check, and the examples-directory fallback looked
for .env files instead of .env.example. Remove the explicit 0644 mode
from that UpdateEnvFiles call so it no longer loosens permissions on
secret env files.

Update malcolm-config.md and authsetup.md to match.

Closes #1133

Signed-off-by: Seth Grover <seth.d.grover@gmail.com>
The MaxMind account ID, license key, and alternate download URL were
passed to the Arkime image build as build arguments. Because the ARGs
were declared in the final stage ahead of every RUN instruction, their
values were recorded in the history of each subsequent layer and
published with the image. GITHUB_TOKEN was also passed as a build
argument, and scripts/build.sh exposed all of these values on the
docker compose command line.

These values are now provided as BuildKit secrets and mounted as
environment variables only for the RUN instructions that need them, so
they no longer appear in image history or process arguments.

- arkime.Dockerfile: remove the MaxMind ARGs and mount them as secrets
  for the GeoIP download step
- nginx.Dockerfile: remove the GITHUB_TOKEN ARG from the docbuild stage
  and mount it as JEKYLL_GITHUB_TOKEN. The previous export ran in a
  subshell and never reached Jekyll, so the token is now passed to the
  docs build for the first time
- docker-build-push-scan.yml: move the credentials from build-args to
  the build-push-action secrets input
- docker-compose-dev.yml: add build secrets for arkime, arkime-live,
  and nginx-proxy, sourced from environment variables
- scripts/build.sh: export the credentials for the compose secrets
  instead of passing --build-arg values, and collapse the duplicated
  build invocation into one
- malcolm-iso and hedgehog-raspi build_via_vagrant.sh: write
  environment.chroot with umask 077 and register the cleanup trap
  before writing it

Also fix three existing bugs in scripts/build.sh: a doubled "=" in the
CONFIG_ENV_DIR assignment, missing -f tests when reading MaxMind values
from arkime-secret.env, and a bad substitution in the <runtime>-compose
fallback.

Signed-off-by: Seth Grover <seth.d.grover@gmail.com>
)

control.py and the appliance's agg-init.sh both set nginx/htpasswd and
htadmin/metadata to 0644, leaving the bcrypt password hashes readable by
any local user who can reach the Malcolm directory. The nginx and
htadmin worker processes run as PUID/PGID, which matches the files'
owner, so world-read access is not needed. Set both files to 0640.

control.py applies this in start(), so existing installs are corrected
the next time Malcolm starts.

Signed-off-by: Seth Grover <seth.d.grover@gmail.com>
Several API endpoints made outbound HTTP requests to other Malcolm
services without a timeout. A dependency that accepted a connection and
then stopped responding held the API request open until gunicorn killed
the worker, and with gunicorn's default single sync worker, that blocked
the entire API, including the /mapi/ping healthcheck.

- Add (connect, read) timeouts to every outbound requests call,
  configurable through MALCOLM_API_HTTP_CONNECT_TIMEOUT_SEC (default 5)
  and MALCOLM_API_HTTP_READ_TIMEOUT_SEC (default 30)
- Bound each /mapi/ready check with MALCOLM_API_READY_TIMEOUT_SEC
  (default 5) and run the checks concurrently, so the endpoint's total
  time is that of its slowest check rather than the sum of all of them
- Add socket_timeout to the Valkey connections used by /mapi/ready and
  the keyspace endpoint
- Add an optional timeout argument to malcolm_utils.check_socket, with
  the default unchanged at 10 seconds
- Run gunicorn with --workers 2 --threads 4 in both compose files and the
  Kubernetes manifest, so one slow request can't block the API or its
  healthcheck

Closes #1135

Signed-off-by: Seth Grover <seth.d.grover@gmail.com>
…2-r23g)

When NGINX_AUTH_MODE is keycloak_remote and KEYCLOAK_SSL_VERIFY is not
true, nginx does not verify the remote Keycloak's TLS certificate, so a
host able to intercept that traffic could impersonate the identity
provider. Verification stays disabled by default to accommodate
self-signed certificates, but the risk is now visible.

- docker_entrypoint.sh: log a warning at startup in keycloak_remote mode
  when verification is disabled, explaining the risk and how to enable
  it. The check uses the same exact "true" match as
  nginx_auth_helpers.lua
- authsetup.md: document the verification setting in the remote Keycloak
  walkthrough, recommend enabling it, and explain how to trust internal
  CAs through nginx/ca-trust/

Signed-off-by: Seth Grover <seth.d.grover@gmail.com>
…v3-2383)

The Beats input on port 5044 used ssl_client_authentication "optional",
so with BEATS_SSL enabled a forwarder that presented no client
certificate could still connect and submit events over TLS. Presented
certificates were verified, but omitting one bypassed authentication.
Set it to "required" so only forwarders holding a certificate issued by
Malcolm's CA can send events.

All supported producers already present the client certificate
generated by auth_setup, including Malcolm's internal Filebeat, Hedgehog
Linux sensors, and the documented external forwarder configurations.
When BEATS_SSL is false, logstash-start.sh still strips the ssl_
settings from the input, so the plaintext configuration is unaffected.

Also set ssl.client_authentication "required" explicitly on the Filebeat
TCP input (port 5045). It already required client certificates through
libbeat's default when certificate_authorities is set; this documents
the behavior and keeps it from silently changing if that setting is
removed.

Update the BEATS_SSL descriptions in malcolm-config.md and
third-party-logs.md to state that client certificate authentication is
required.

Closes #1136

Signed-off-by: Seth Grover <seth.d.grover@gmail.com>
watch-pcap-uploads-folder.py runs as root and called os.path.isfile()
and os.chown() on paths in the upload directory, both of which follow
symlinks. Because the upload service's SFTP server allows creating
symlinks, a symlink placed in the upload directory could cause the
watcher to change the ownership of its target, and the gap between the
check and the chown allowed a regular file to be swapped for a symlink.

- Inspect each upload with lstat() and remove symlinks and other
  non-regular entries instead of processing them
- Open uploads with O_NOFOLLOW and perform the ownership change, file
  type detection, and size check through that descriptor (fchown,
  magic.from_descriptor, fstat)
- Before moving or decompressing an upload, confirm the path still
  refers to the same file, and remove it if it changed
- Open compressed uploads through the same O_NOFOLLOW descriptor, and
  set ownership and mode on the decompression temp file through its
  descriptor rather than by path

Closes #1137

Signed-off-by: Seth Grover <seth.d.grover@gmail.com>
The hh-extracted-files location proxies to a host taken from the request
path, which Malcolm can't validate (see GHSA-j89q-3q7m-8vr6). Requests
were forwarded with the client's Authorization header and cookies
intact, so a crafted link could cause a user's Malcolm credentials (the
basic auth header, Keycloak session cookie, or the bearer token set by
the Lua auth code) to be sent to an arbitrary host on port 8006.

Clear the Authorization and Cookie headers in that location, as the
Arkime and upload locations already do for Authorization. Hedgehog's
extracted-file server authorizes Malcolm by its ACL and does not rely on
either header.

Signed-off-by: Seth Grover <seth.d.grover@gmail.com>
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
…XII 2.1, drop MISP context values

STIX patterns (#1148)
Builds on #1144. Patterns are normalized to disjunctive normal form and
each clause is judged on its own, so no emitted item can match traffic
the pattern wouldn't:
- OR alternatives and IN members emit one item each; OR branches that
  can't be expressed are dropped and the rest kept
- an AND within one observation that includes a file or certificate
  hash is reduced to the hash(es), noted in meta.desc
- network-traffic address-type tests are dropped in favor of the
  address; src_ref/dst_ref values map to Intel::ADDR
- other conjunctions, relations between observations, and qualifiers
  are rejected, with the reason in the warning
- object paths are normalized before the type map lookup, so quoted and
  unquoted hash names resolve the same

TAXII 2.1 and STIX parsing (#1149)
- TAXII 2.1 envelopes are parsed (previously every page failed)
- since is sent to the server as added_after
- collections reporting can_read: false are skipped
- indicators are parsed one at a time, so one malformed object no
  longer discards the whole bundle; STIX 2.0 bundle spec_version is
  passed to its objects
- empty indicator names no longer produce a leading ". " in meta.desc

MISP context values (#1150)
- MISP objects with a file, certificate, or key hash emit only their
  hashes, noted in meta.desc
- filename|hash composites emit only the hash, with the file name in
  the description
- attributes with to_ids: false are skipped by default; restore the
  previous behavior per feed with require_to_ids: False in misp.yaml,
  or globally with --misp-require-to-ids false

BEHAVIOR CHANGE: with the to_ids default, about 4.5% of MISP items from
public feeds are no longer emitted (mostly URLhaus payload host IPs).

Tests: test_stix_point_patterns.py extended; new test_misp_hash_reduction.py
and test_stix_bundle_parsing.py (103 tests total). Docs updated in
docs/zeek-intel.md.

Fixes #1148
Fixes #1149
Fixes #1150

Signed-off-by: Seth Grover <seth.d.grover@gmail.com>
When importing a settings file, _load_configuration_section reported
every configuration item without is_modified set as "missing from
settings file." Items whose imported value matched their default, and
items exported with the empty-value sentinel, were counted even though
they were present in the file, so a fresh export re-imported by the
same version reported a large number of missing items.

Count only keys absent from the file's configuration section, matching
how _load_installation_section already reports missing installation
items.

Signed-off-by: Seth Grover <seth.d.grover@gmail.com>
When importing a configuration file with --import-malcolm-config-file,
install.py restored each entry in its "raw" section with
base64_decode_files_to_dir, which joined the entry's key onto the config
directory without verifying the result. A key containing ".."
components, or an absolute path, wrote the decoded contents outside the
config directory, anywhere writable by the account running the
installer.

- Skip entries whose path is absolute, empty, or not a string, or which
  resolve outside the destination directory through ".." components or
  symlinks, logging a warning for each
- Add an optional filename pattern to base64_decode_files_to_dir, and
  restrict imported files to *.env, matching what configuration export
  produces
- Return the number of files written, and report it in the installer's
  restore message
- Verify that "raw" is a dictionary before processing it, and skip
  non-string Base64 values instead of raising

Reported by Sithum Shihara (@zaara2004).

Closes #1147

Signed-off-by: Seth Grover <seth.d.grover@gmail.com>
The suricataDisableSids DependencySpec was copied from
suricataRuleUpdate and kept its ValueRule with default_value=False.
Toggling autoSuricata or liveSuricata during import tried to apply
False to a list-of-strings item, which the validator rejects, so each
import logged "Failed to set default" twice.

Drop the ValueRule and keep only the visibility rule. The item's own
default_value=[] covers fresh installs, and leaving out an empty-list
rule avoids wiping imported SIDs whenever a Suricata toggle fires.

Signed-off-by: Seth Grover <seth.d.grover@gmail.com>
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
@mmguero mmguero added this to the v26.10.0 milestone Oct 29, 2026
@mmguero mmguero self-assigned this Oct 29, 2026
@mmguero mmguero added release Related to creation/packaging of Malcolm releases github Dealing with Malcolm's GitHub repository labels Oct 29, 2026
@mmguero mmguero added this to Malcolm Oct 29, 2026
@mmguero mmguero moved this to In Progress in Malcolm Oct 29, 2026
Comment thread api/project/__init__.py Dismissed
Comment thread api/project/__init__.py Dismissed
Comment thread suricata/scripts/suricata_socket.py Dismissed
Comment thread suricata/scripts/suricata_socket.py Dismissed
@mmguero
mmguero merged commit c6160d8 into main Oct 29, 2026
4 of 5 checks passed
@github-project-automation github-project-automation Bot moved this from In Progress to Done in Malcolm Oct 29, 2026
@mmguero mmguero moved this from Done to Released in Malcolm Oct 29, 2026
@sylvesterkaczmarek

Copy link
Copy Markdown
Contributor

Thank you for the kind acknowledgement and for all the work the Malcolm team puts into this project; it's a real pleasure to contribute.

@mmguero
mmguero deleted the v26.10.0_merge_cisagov branch November 11, 2026 22:02

This branch was successfully deployed

1 active deployment
github-pages — c6160d8e Deployed Oct 29, 2026 by mmguero via deploy #22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

github Dealing with Malcolm's GitHub repository release Related to creation/packaging of Malcolm releases

Projects

Status: Released

Development

Successfully merging this pull request may close these issues.

7 participants