Add OPERATIONS.md

[ptr727]

Closes the OPERATIONS.md gap from the 2026-08-02 hub audit (audit run 2026-08-02T16:10Z | hub 69688ec).

The hub declares this file for every repo (appliesTo: "*" in spec/files.json) and checks it for presence only, so its content is entirely this repository's own. It is mandatory because the convention was emerging unevenly across the fleet, with the same operational material landing under ad-hoc names, which is the improvisation the declared destinations exist to prevent.

What it carries

The five declared headings, with real content rather than a stub:

• Runbooks - the local gate chain that mirrors CI (csharpier, build, dotnet format verify, dotnet test, dotnet husky run) and the containerized lint set, plus how to cut a release and how to run verify safely.
• Backup and Recovery - the state the application writes is the user's, not the repository's: the SQLite databases and the .bak files that undo restores.
• Logs and Debugging - what the three exit codes distinguish, so a calling script branches on them rather than on output, and why exiftool validation warnings are logged at debug level only.
• Tool Usage - exiftool, ffmpeg, and the Docker daemon that verify needs, plus reading the Immich API key from a file so it stays out of shell history and the process list.
• Configuration Layout - where each tree sits and why repo-config/ is outside the Actions-owned .github/.

The document describes only what is on develop. An earlier revision documented the verify command, the exiftool -validate flag and an exit code 2, none of which exist on this branch: they are on the unmerged verify work in #35. Copilot caught that across two rounds and it is corrected in 550565b. The verify operational content, including that it needs its own database and cannot run inside this project's own container, follows once #35 merges.

Note this repository is on the fleet's leading edge here. No repo carried OPERATIONS.md at audit time, since the requirement was promoted to hub main only in ptr727/ProjectTemplate#517.

Verification

markdownlint, editorconfig-checker, and the hub prose gate all report clean. Inline links are correct for this file, which spec/section-model.md names in the inline-link exception rather than the reference-style default.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 44.96%. Comparing base (1ab0f11) to head (78581a2).
⚠️ Report is 1 commits behind head on develop.

Additional details and impacted files

@@           Coverage Diff            @@
##           develop      #37   +/-   ##
========================================
  Coverage    44.96%   44.96%           
========================================
  Files           25       25           
  Lines         3398     3398           
  Branches       259      259           
========================================
  Hits          1528     1528           
  Misses        1824     1824           
  Partials        46       46           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

Adds a new OPERATIONS.md document to capture this repository’s operational runbooks (local gate chain, release mechanics, backup/recovery, logging, external tool expectations, and configuration layout), aligning with the fleet requirement to have a standard operations destination.

Changes:

• Introduces OPERATIONS.md with operational runbooks for local validation and release publishing.
• Documents backup/recovery expectations for user-owned state (*.db, *.bak) and basic troubleshooting guidance.
• Summarizes tool/runtime dependencies (exiftool, ffmpeg, Docker) and repository layout pointers.

Suppressed comments (3)

OPERATIONS.md:41

• This section documents a verify command (--quick, Docker/Immich decoder, its own DB), but the CLI currently only defines process, undo, import, index, and trash (see PhotoCleaner/CommandLine.cs). As written, this is misleading operational guidance.

### Verify a media library

`verify` runs Immich's own decoder inside the `immich-server` image, so it needs a reachable Docker daemon and cannot run from inside this project's own container. Use `--quick` where Docker is unavailable, which runs the structural checks only and cannot detect an undecodable file.

Give `verify` its own database. It records verified state in the same `is_processed` column that `process` writes, so pointing `--db` at a `Process.db` makes `verify` skip every file as already verified when they were only processed, and nothing detects it.

OPERATIONS.md:67

• The documented exit-code contract (including an exit code 2 for per-file failures) doesn't match the implementation: CommandRunner.RunAsync returns 0 on completion and 1 on cancellation/exception, and the commands log per-file failures instead of returning a distinct exit code.

Exit codes distinguish the two failure kinds, and a script should branch on them rather than on output:

- `0`: every file succeeded.
- `1`: the command could not run, meaning an unhandled exception, a fatal configuration error, a cancellation, or a failed `verify` preflight.
- `2`: the command ran to completion and one or more files failed.

OPERATIONS.md:75

• This bullet ties Docker specifically to a verify command that doesn't exist in the current CLI. Consider describing Docker in terms of the actual repo operations (containerized linters and building/running the published image).

- **Docker** is a runtime dependency of `verify` only, which pulls `ghcr.io/immich-app/immich-server:release` and runs Immich's own `MediaRepository` inside it. Behavior therefore tracks Immich across releases rather than being reimplemented here.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (4)

OPERATIONS.md:77

• This bullet states Docker is a runtime dependency of a verify feature that doesn’t exist in this repository. As written, it incorrectly implies the application needs Docker at runtime.

- **Docker** is a runtime dependency of `verify` only, which pulls `ghcr.io/immich-app/immich-server:release` and runs Immich's own `MediaRepository` inside it. Behavior therefore tracks Immich across releases rather than being reimplemented here.

OPERATIONS.md:43

• verify is referenced as an available command (and --quick/--db behavior is documented), but there is no verify subcommand or related implementation anywhere in the repo. This section is currently misleading and should either be removed or rewritten to reflect only existing commands.

This issue also appears on line 77 of the same file.

### Verify a media library

`verify` runs Immich's own decoder inside the `immich-server` image, so it needs a reachable Docker daemon and cannot run from inside this project's own container. Use `--quick` where Docker is unavailable, which runs the structural checks only and cannot detect an undecodable file.

Give `verify` its own database. It records verified state in the same `is_processed` column that `process` writes, so pointing `--db` at a `Process.db` makes `verify` skip every file as already verified when they were only processed, and nothing detects it.

OPERATIONS.md:70

• The documented exit code meanings don’t match the application: Program.Main/CommandRunner.RunAsync only ever return 0 (work completed) or 1 (cancellation/unhandled exception). There is no code path that returns exit code 2, and per-file failures currently still produce exit code 0.

Exit codes distinguish the two failure kinds, and a script should branch on them rather than on output:

- `0`: every file succeeded.
- `1`: the command could not run, meaning an unhandled exception, a fatal configuration error, a cancellation, or a failed `verify` preflight.
- `2`: the command ran to completion and one or more files failed.

OPERATIONS.md:75

• This claims exiftool is always passed -validate, but there are no exiftool invocations in the codebase that include -validate (for example MediaUtilities.GetExifToolJsonAsync uses -groupNames -json). Update this line so it doesn’t state a flag that isn’t actually used.

- **exiftool** reads and writes metadata, and is always passed `-validate`. It is installed in the Docker image, and a native run needs it on `PATH`.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (4)

OPERATIONS.md:77

• “The Docker linters pull :latest deliberately, so a local run matches whatever CI resolved” is not accurate for this repo: CI uses pinned GitHub Actions for markdownlint/cspell/actionlint (see .github/workflows/validate-task.yml), and only editorconfig-checker runs via Docker :latest. Reword to avoid implying local Docker tags mirror CI’s pinned tool versions.

The Docker linters pull `:latest` deliberately, so a local run matches whatever CI resolved.

OPERATIONS.md:62

• The exit-code section currently implies 1 only means cancellation/unhandled exception, but this program also bypasses startup on parse/validation errors and returns ParseResult.InvokeAsync() directly (Program.cs), which can yield a non-zero exit for invalid arguments. Document that 1 can also indicate invocation/validation failure so scripts don’t misclassify argument errors as crashes.

- `0`: the command ran to completion.
- `1`: the command could not run, meaning a cancellation or an unhandled exception.

OPERATIONS.md:7

• The PR description says OPERATIONS.md documents how to run a verify workflow safely and calls out two verify-specific traps, but this file does not mention verify, and the repo does not appear to define a verify command/subcommand or workflow (no verify in PhotoCleaner/CommandLine.cs or .github/workflows/). Either add the intended verify runbook section here or update the PR description to match the repository’s actual surfaces.

## Runbooks

### Run the gates the way CI runs them

OPERATIONS.md:26

• This command list is missing the workflow lint that exists in .vscode/tasks.json (Lint: Workflows runs actionlint). Add the actionlint container invocation so the documented “lint set” matches the local gate surface.

docker run --rm --pull=always -v "$PWD":/workdir --workdir /workdir davidanson/markdownlint-cli2:latest "**/*.md"
docker run --rm --pull=always -v "$PWD":/workdir --workdir /workdir ghcr.io/streetsidesoftware/cspell:latest --no-progress README.md HISTORY.md
docker run --rm --pull=always -v "$PWD":/check --workdir /check mstruebing/editorconfig-checker:latest

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (3)

OPERATIONS.md:19

• The sentence says CI runs dotnet test “unchanged”, but validate-task.yml runs tests with coverage collection (dotnet test --collect:"XPlat Code Coverage" --results-directory ./coverage). This is a concrete difference when trying to mirror CI locally; please either reflect the actual command or reword to avoid claiming it’s identical.

CI substitutes `dotnet csharpier check .` for the format step, after a `dotnet tool restore`, and runs the style verify and `dotnet test` unchanged. So a local run that formats a file leaves CI clean, while an unformatted commit fails there rather than being fixed.

OPERATIONS.md:9

• The PR description says this document includes guidance/traps for running a verify command (including DB isolation and Docker-in-Docker constraints), but this file doesn’t mention a verify command/workflow, and the repo doesn’t appear to define one. Either add the missing verify runbook content here, or update the PR description to remove that claim so the PR accurately reflects what’s being introduced.

## Runbooks

### Run the gates the way CI runs them

Local and CI runs read the same committed configuration, but they invoke it differently: locally the formatter writes, and in CI it only verifies. The [`.NET Format`](./.vscode/tasks.json) task is the local clean-compile chain, meaning `dotnet csharpier format`, then `dotnet build`, then the style verify. Run the chain and the suite before committing, since the chain never runs the tests and a change that compiles and formats cleanly can still be broken:

OPERATIONS.md:26

• This “lint set” snippet claims to match the Lint: tasks, but it omits the workflow lint (Lint: Workflows) command that exists in .vscode/tasks.json (and in CI via raven-actions/actionlint). As written, someone following this runbook won’t run the full local lint surface.

docker run --rm --pull=always -v "$PWD":/workdir --workdir /workdir davidanson/markdownlint-cli2:latest "**/*.md"
docker run --rm --pull=always -v "$PWD":/workdir --workdir /workdir ghcr.io/streetsidesoftware/cspell:latest --no-progress README.md HISTORY.md
docker run --rm --pull=always -v "$PWD":/check --workdir /check mstruebing/editorconfig-checker:latest

[ptr727]

Suppressed findings answered

Answering the collapsed findings across both rounds. All correct, and all one root error: I wrote this file describing the verify command, the exiftool -validate flag, and an exit code 2, none of which exist on develop. They live on the unmerged verify branch, and this branch was cut from develop, so the document described code that is not here. Fixed in 550565b.

Round 1 on c755194, Suppressed comments (3)

1. OPERATIONS.md:41 - Fixed in 550565b

This section documents a verify command (--quick, Docker/Immich decoder, its own DB), but the CLI currently only defines process, undo, import, index, and trash (see PhotoCleaner/CommandLine.cs).

Verified against develop rather than taken on trust: the command literals there are exactly process, undo, import, index, trash, with no verify. The runbook section is removed.

2. OPERATIONS.md:67 - Fixed in 550565b

The documented exit-code contract (including an exit code 2 for per-file failures) doesn't match the implementation: CommandRunner.RunAsync returns 0 on completion and 1 on cancellation/exception.

Confirmed in CommandRunner.cs on develop, which returns 1 twice and 0 once, with no 2. The section now documents 0 and 1 only, and says plainly that a per-file failure does not change the exit code, so 0 means the command finished rather than that every file succeeded.

3. OPERATIONS.md:75 - Fixed in 550565b

This bullet ties Docker specifically to a verify command that doesn't exist in the current CLI. Consider describing Docker in terms of the actual repo operations (containerized linters and building/running the published image).

Taken as suggested. The text now states the application needs no Docker daemon at runtime and that Docker here is packaging and tooling only.

Round 2 on 2487b39, Suppressed comments (4)

Findings 1 to 3 restate the round-1 set above (OPERATIONS.md:43, :70, :77) and are fixed by the same commit. The fourth is new:

4. OPERATIONS.md:75 - Fixed in 550565b

This claims exiftool is always passed -validate, but there are no exiftool invocations in the codebase that include -validate (for example MediaUtilities.GetExifToolJsonAsync uses -groupNames -json).

Correct. On develop that call is .WithArguments(["-groupNames", "-json", filePath]). The bullet now names the invoking method without claiming a flag it does not pass.

---

While correcting these I also verified every remaining option the file names against develop rather than assuming: --logfile, --loglevel, --apikey-file, --db, --trashdb and --skipbackup all exist, and the backup claim is backed by .bak handling in ProcessTask.cs and UndoTask.cs. The flag was named imprecisely as "backups disabled" and is now --skipbackup.

The verify operational content lands in a follow-up once #35 merges.

7 of 7 suppressed findings across both rounds are now answered.

[ptr727]

Suppressed findings answered (round 3)

Three more from review 4839547991 on cb8a354, Suppressed comments (3). All three correct, all fixed in 1f69db3 or in the PR description.

1. OPERATIONS.md:19 - Fixed in 1f69db3

The sentence says CI runs dotnet test "unchanged", but validate-task.yml runs tests with coverage collection (dotnet test --collect:"XPlat Code Coverage" --results-directory ./coverage). This is a concrete difference when trying to mirror CI locally.

Correct, and the irony is that this sentence was itself written to fix an earlier finding about overclaiming that CI and local runs are identical. I corrected the csharpier half and left a second inaccuracy in the same sentence. It now names both differences explicitly and says the style verify is the part that is identical.

2. OPERATIONS.md:9 - Fixed in the PR description

The PR description says this document includes guidance/traps for running a verify command (including DB isolation and Docker-in-Docker constraints), but this file doesn't mention a verify command/workflow, and the repo doesn't appear to define one.

Correct. I removed the verify content from the file after round 2 established it documented unmerged work, and left the PR description advertising it. The description now states that the document covers only what is on develop, records why the verify content was removed, and says it follows once #35 merges.

This is the second time on this repository today that a stale PR description outlived the change it described, the other being the pin mismatch on #36. Both were caught by suppressed findings rather than by any gate.

3. OPERATIONS.md:26 - Fixed in 1f69db3

This "lint set" snippet claims to match the Lint: tasks, but it omits the workflow lint (Lint: Workflows) command that exists in .vscode/tasks.json (and in CI via raven-actions/actionlint). As written, someone following this runbook won't run the full local lint surface.

Correct, and the failure mode is the one that matters for a runbook: someone follows it, sees four clean results, and believes they ran the full surface. The actionlint invocation is added, taken from .vscode/tasks.json rather than written from memory, and the two other commands are corrected to the -w form the tasks actually use. The text now states plainly that those four are the whole local surface and how CI reaches the same four.

---

14 of 14 suppressed findings across all four rounds on this PR are now answered.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (1)

OPERATIONS.md:80

• This implies the Docker linter runs will match CI, but CI pins markdownlint/cspell/actionlint via action wrappers (see .github/workflows/validate-task.yml) while the local tasks pull Docker :latest. That means local results can legitimately differ from CI over time; only editorconfig-checker is run via Docker in CI.

The Docker linters pull `:latest` deliberately, so a local run matches whatever CI resolved.

[ptr727]

Suppressed findings answered (round 4)

One from the round on 1f69db3, Suppressed comments (1). Correct, fixed in abf97af.

OPERATIONS.md:80 - Fixed in abf97af

This implies the Docker linter runs will match CI, but CI pins markdownlint/cspell/actionlint via action wrappers (see .github/workflows/validate-task.yml) while the local tasks pull Docker :latest. That means local results can legitimately differ from CI over time; only editorconfig-checker is run via Docker in CI.

Correct. The sentence I added to fix the previous round's finding, that the snippet omitted actionlint, then overclaimed in the other direction by saying CI "runs the same four" without qualifying how. Same rules and same committed configuration, different builds.

The text now states that CI reaches three of the four through SHA-pinned action wrappers and only editorconfig-checker as a container, that :latest locally can therefore drift ahead of a pinned wrapper, that CI is authoritative when they disagree, and that such a disagreement is a version gap rather than a rule change. That last part is the operationally useful half: someone hitting it needs to know which to trust and what it means, not just that a difference exists.

---

15 of 15 suppressed findings across all five rounds on this PR are now answered.

Three consecutive rounds on this file were caused by my own corrections overshooting: claiming CI is identical, then omitting a linter, then overclaiming again about the ones I added. Each was caught by a suppressed finding rather than by any gate.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (2)

OPERATIONS.md:78

• This implies the API key is only read from a file, but the CLI also accepts an inline --apikey value (mutually exclusive with --apikey-file). Consider rewording to recommend --apikey-file rather than stating it as the only supported mechanism.

The Immich API key is read from a file with `--apikey-file` rather than an argument, which keeps it out of shell history and out of the process list.

OPERATIONS.md:80

• This sentence contradicts the earlier note that local lint containers track :latest while CI uses SHA-pinned action wrappers (so local results can legitimately differ from CI). Reword to avoid implying local lint runs will match CI.

The Docker linters pull `:latest` deliberately, so a local run matches whatever CI resolved.

[ptr727]

Suppressed findings answered (round 5)

Two from review 4839577417 on abf97af, Suppressed comments (2). Both correct, fixed in c0e169b.

1. OPERATIONS.md:78 - Fixed in c0e169b

This implies the API key is only read from a file, but the CLI also accepts an inline --apikey value (mutually exclusive with --apikey-file). Consider rewording to recommend --apikey-file rather than stating it as the only supported mechanism.

Correct. Verified in CommandLine.cs on develop rather than taken on trust: both options exist, each description names the other as mutually exclusive, and a validator rejects supplying both with "Specify only one of --apikey or --apikey-file, not both."

The text now names both, says they are mutually exclusive, and recommends the file with the reason: an inline key lands in shell history and stays visible in the process list for the life of the command. That is more useful than the original, which stated a fact about one flag and left the other undocumented.

2. OPERATIONS.md:80 - Fixed in c0e169b

This sentence contradicts the earlier note that local lint containers track :latest while CI uses SHA-pinned action wrappers (so local results can legitimately differ from CI). Reword to avoid implying local lint runs will match CI.

Correct, and self-inflicted. That line survived from the first draft, where it was carried over from the hub's own OPERATIONS.md. It is true there, because the hub runs those linters as containers in CI, and false here, because this repository reaches three of the four through pinned action wrappers. Adding the accurate paragraph in the previous round left the stale sentence sitting four lines below it, contradicting it outright.

Removed rather than reworded, since the paragraph above now says all of it.

---

17 of 17 suppressed findings across all six rounds on this PR are now answered.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (1)

OPERATIONS.md:65

• Exit code 1 can also be returned for command-line parse/validation errors (e.g., CommandLine.BypassStartup() short-circuits when parseResult.Errors.Count > 0). The current wording implies 1 only means cancellation or an unhandled exception, which is incomplete and slightly misleading.

- `1`: the command could not run, meaning a cancellation or an unhandled exception.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (1)

OPERATIONS.md:16

• The local gate snippet runs dotnet csharpier/dotnet husky without first restoring local tools. On a fresh clone, this can fail because the repo uses a tool manifest (.config/dotnet-tools.json) and CI explicitly runs dotnet tool restore before CSharpier. Add dotnet tool restore to the recommended local command chain so it works out of the box and better matches CI.

dotnet csharpier format --log-level=debug .
dotnet build
dotnet format style --verify-no-changes --severity=info --verbosity=detailed
dotnet test
dotnet husky run

[ptr727]

Suppressed findings answered (round 6)

One from review 4839596687 on c0e169b, Suppressed comments (1). Correct, fixed in 78581a2.

OPERATIONS.md:65 - Fixed in 78581a2

Exit code 1 can also be returned for command-line parse/validation errors (e.g., CommandLine.BypassStartup() short-circuits when parseResult.Errors.Count > 0). The current wording implies 1 only means cancellation or an unhandled exception, which is incomplete and slightly misleading.

Correct, and verified in CommandLine.cs on develop rather than taken on trust:

internal static bool BypassStartup(ParseResult parseResult) =>
    parseResult.Errors.Count > 0
    || parseResult.CommandResult.Children.Any(...);

A parse or validation error therefore exits 1 on a path that never reaches CommandRunner.RunAsync, which is where the cancellation and exception cases I listed live. The --apikey and --apikey-file mutual-exclusion validator corrected two rounds ago is one such error, so the document already described a condition it then failed to account for in the exit codes.

The entry now names all three causes and adds the part a reader acts on: a parse error short-circuits before any work starts, so nothing was touched. That distinction matters for the same reason the 0 entry carries one, since 0 here means the command finished rather than that every file succeeded.

---

19 of 19 suppressed findings across all seven rounds on this PR are now answered.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

[ptr727]

Suppressed finding answered (round 8, on the merged head)

One collapsed finding from review 4839755012 on 78581a2, Suppressed comments (1), went unanswered when this merged. It surfaced while auditing every constituent PR ahead of the develop to main promotion. A suppressed finding carries no thread, so this PR read as fully resolved.

Correct, and fixed in 5e2eee0 via #44.

OPERATIONS.md:16 - Fixed in 5e2eee0

The local gate snippet runs dotnet csharpier/dotnet husky without first restoring local tools. On a fresh clone, this can fail because the repo uses a tool manifest (.config/dotnet-tools.json) and CI explicitly runs dotnet tool restore before CSharpier. Add dotnet tool restore to the recommended local command chain so it works out of the box and better matches CI.

Verified rather than taken on faith, and a warm package cache hides it, so the check needs a cold one. Against a clone with NUGET_PACKAGES and DOTNET_CLI_HOME pointed at empty directories, the chain died on its first command with Run "dotnet tool restore" to make the "csharpier" command available. and exit 1. With the restore ahead of it, the restore succeeded and CSharpier formatted all 54 files at exit 0.

The fix went further than the snippet on Copilot review of #44: the same gap applies to the .NET Format VS Code task, which depends on CSharpier Format and .NET Build and restores in none of the three. The wording therefore attaches the restore to the clone and names both surfaces, rather than to the chain alone.

The scope stops at OPERATIONS.md. The .NET Tools section of README.md documents building the manifest from scratch, where a restore would be wrong, and CODESTYLE.md names the commands in prose rather than offering a runnable chain.