From 131289a5301d849209a9b4cf4ca5690c6107804f Mon Sep 17 00:00:00 2001 From: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com> Date: Fri, 6 Nov 2026 07:48:14 +0000 Subject: [PATCH] Add Malcolm upgrade decision, preflight and recovery walkthrough Refresh the focused change onto contributor-prs at 7046399b. Validated **3 added Bash snippets**, **9 local links/anchors** and the current ISO step references. No operating-system upgrade, live Malcolm upgrade or restore was executed. --- docs/malcolm-upgrade.md | 121 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 121 insertions(+) diff --git a/docs/malcolm-upgrade.md b/docs/malcolm-upgrade.md index 0267dccbc..8f2c67d6a 100644 --- a/docs/malcolm-upgrade.md +++ b/docs/malcolm-upgrade.md @@ -1,6 +1,9 @@ # Upgrading Malcolm * [Upgrading Malcolm](#UpgradePlan) + - [Choose the upgrade procedure](#UpgradeChoose) + - [Preflight and backup checklist](#UpgradePreflight) + - [Recovery when an upgrade fails](#UpgradeRecovery) - [Scenario 1: Malcolm was installed from the ISO installer or a prepackaged or release tarball](#UpgradeISOEnv) - [Scenario 2: Malcolm was installed from a GitHub clone](#UpgradeGitEnv) - [Final Steps](#UpgradeFinalSteps) @@ -9,6 +12,124 @@ At this time there is not an "official" version upgrade procedure for Malcolm, a The Malcolm project uses [calendar versioning](https://calver.org/) for its [releases]({{ site.github.repository_url }}/releases/latest). A careful reading of the release notes between the source and target version before upgrading Malcolm is highly recommended. + +## Choose the upgrade procedure + +**Do not start with the operating-system upgrade commands until you have +identified how Malcolm was installed.** The two scenarios below change +different things: + +| Existing installation | Procedure | What it changes | +| --- | --- | --- | +| Malcolm/Hedgehog installer ISO | [Scenario 1](#UpgradeISOEnv) | Host operating system (only when required), Malcolm application files, and container images | +| Downloaded Malcolm release tarball or ZIP | [Scenario 1](#UpgradeISOEnv), skipping the ISO-only Debian steps on other Linux distributions | Malcolm application files and container images | +| Git clone (including a locally modified clone) | [Scenario 2](#UpgradeGitEnv) | Checked-out Git revision, local configuration, and container images | + +Hedgehog Linux ISO installations follow the ISO path because both variants +now share the same base distribution. For a Git clone used on a host with an +independently managed operating system, use the Git scenario, not the Debian +Bookworm-to-Trixie instructions. For a release archive installed on a +non-ISO Linux distribution, run Scenario 1's stop step, **skip its ISO-specific +steps 2 through 10**, and continue with the Malcolm application/image steps +only after the backups below have been verified. + +If you are unsure which kind of installation you have, inspect the installation +directory from your normal user account: + +~~~bash +cd ~/Malcolm +test -d .git && git remote -v && git status --short +test -f .os-info && cat .os-info +~~~ + +The presence of a Git working tree is a useful clue, but it is **not** a +substitute for knowing how the host was installed. The release archive and ISO +paths can have different operating-system and deployment responsibilities. + +## Preflight and backup checklist + +The steps below are non-destructive; perform them **before** stopping services +or following either upgrade scenario. + +1. Confirm the currently deployed Malcolm version and the target release. + Read **every intervening release note** and check whether the update also + requires a Debian major-version upgrade, container format changes, or a + migration of persistent data. +2. Confirm a maintenance window and access to the local console or another + recovery channel. Do not rely on being able to SSH back after changing + the OS networking or package sources. +3. Record the baseline, retaining command output in an administrator-controlled + location: + + ~~~bash + cd ~/Malcolm + ./scripts/status + docker compose version + df -h . + test -d .git && git status --short && git describe --tags --always + ~~~ + + Run the last command only for a Git checkout. In a deployment that uses + Kubernetes rather than Docker Compose, use the Kubernetes [status commands](kubernetes.md#Running) + instead; the Docker commands do not describe a Kubernetes installation. +4. **Back up both configuration and data.** The `rsync` application-directory + backups in the scenarios below intentionally **exclude** the large runtime + directories (OpenSearch indices, PCAPs, Zeek/Suricata logs, and extracted + files). Make a separate backup or snapshot of each dataset you need to + preserve, following your data retention and encryption requirements. +5. Protect application-specific configuration and credentials: `config/`, + authentication material (including generated certificates and keys), + deployment secrets, local `.env` files, and any customized proxy, ingest, + and sensor settings. Store backups outside the directory that is about + to be replaced and restrict their access. +6. **Verify the backups** before making changes: confirm the archives exist, + can be read, and the chosen OpenSearch backup/snapshot can be restored to a + separate test environment. A successful copy operation is not a recovery + test. Record the backup location, old version, and any image tags needed + for recovery. +7. For an ISO host migration between major Debian versions, take a + hypervisor/bare-metal image or snapshot *in addition to* application/data + backups. Simply restoring the old Malcolm application directory does not + undo an operating-system upgrade. + +Do not run the destructive image-removal and archive-cleanup commands below +until the source data, snapshots, and release artifacts are verified. + +## Recovery when an upgrade fails + +Stop and diagnose at the **first** failed checkpoint. Avoid repeating +the update blindly, resetting your data, or erasing the configuration in an +attempt to make services start. + +| Failure checkpoint | Safer recovery path | +| --- | --- | +| OS upgrade or reboot fails | Restore the verified *host/VM snapshot*, or repair the OS from console/recovery media. Restoring only the Malcolm directory will not revert Debian packages or system services. | +| New application files or Git checkout are incorrect | Keep Malcolm stopped, inspect the application backup and recorded previous Git revision, then restore the prior application and configuration together on a recovery/staging host. | +| Git stash or merge conflict | See [Scenario 2: resolving merge conflicts](#UpgradeGitEnv). Preserve the stash until the intended configuration has been inspected; do not run `git stash clear`. | +| OpenSearch fails after a version change | Preserve the existing indices and logs. Check the target version's supported data format and restore a **tested snapshot** to a compatible version when necessary. Never point an older OpenSearch version at data that may have been migrated by a newer version without confirming compatibility. | +| Authentication or sensor communication fails | Check the preserved TLS certificates, trust roots, secrets, and environment configuration first. Regenerating identity material can break previously enrolled sensors. | + +For a simple Git checkout with **no persistent data migration**, the +previous checkout can be identified using: + +~~~bash +cd ~/Malcolm +git reflog -n 10 +git stash list +~~~ + +This is an inspection step, **not** permission to reset branches or overwrite +changes. Restoring a previous release should be a controlled operation using +the previously recorded revision, matching container images, preserved +configuration, and compatible data. Check that a rollback has not invalidated +any migrations run by the newer release. + +After either a successful upgrade **or** a recovery, use the +[Final Steps](#UpgradeFinalSteps) checks to confirm web login, dashboards, +services, packet capture and data continuity before reopening ingestion to +normal traffic. + + ## Scenario 1: Malcolm was installed from the [ISO installer](malcolm-hedgehog-e2e-iso-install.md#ISOInstallMalcolm) or a [prepackaged or release tarball](development.md#Packager) This section walks through upgrading a Malcolm system (installed via the [ISO Installation](malcolm-hedgehog-e2e-iso-install.md#ISOInstallMalcolm)) to the [latest release]({{ site.github.repository_url }}/releases/latest) of Malcolm (`v{{ site.malcolm.version }}` at the time of this writing), including the underlying Linux system upgrade, Docker image refresh, Malcolm application file replacement, configuration review, and startup.