Cherry-pick #47099: Fix Fleet startup crash on read-only filesystem without S3 bucket - #47166
Merged
nulmete merged 1 commit intoJun 9, 2026
Merged
Conversation
…7099) **Related issue:** Resolves #47090 Fleet crashes into `CrashLoopBackOff` on startup when deployed on Kubernetes with `readOnlyRootFilesystem: true` and **no** S3 software installers bucket configured: ``` Failed to start: initializing filesystem org logo store: mkdir /tmp/org-logos: read-only file system ``` I realised I was calling `initFatal` when failing to create a directory on the filesystem which doesn't match the pattern of `logging` + `creating a "failing" store` (one that is initialized but fails all operations) as we do for e.g. software title icons (see NewFailingSoftwareTitleIconStore). Per this slack conversation: https://fleetdm.slack.com/archives/C084F4MKYSJ/p1780931127976389, we decided to fall back to a database-backed storage: <img width="737" height="114" alt="Screenshot 2026-06-08 at 3 16 28 PM" src="https://github.com/user-attachments/assets/2a6ff75f-b382-40ba-81d9-3be3cfbd648a" /> # Checklist for submitter - [x] Changes file added for user-visible changes in `changes/`. ## Testing - [x] Added/updated automated tests - [x] QA'd all new/changed functionality manually Commented out this line to force filesystem usage: <img width="615" height="71" alt="Screenshot 2026-06-08 at 1 18 53 PM" src="https://github.com/user-attachments/assets/85043c88-5c8c-48a0-8145-098fba9513bd" /> #### Before Server crashes <img width="1278" height="124" alt="Screenshot 2026-06-08 at 1 18 17 PM" src="https://github.com/user-attachments/assets/7b788a24-131a-47a3-8580-fcd9fda8b449" /> #### After Server starts and logo upload works - Without --dev_license https://github.com/user-attachments/assets/58c5ebf9-cf52-4ba0-ac98-9675e7eef92c - With --dev_license https://github.com/user-attachments/assets/117bb812-31bd-4849-927c-93cafd1a71d7 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes * **New Features** - Organization logos now support database storage as the fallback option when S3 software installers bucket is not configured, replacing local filesystem storage for improved reliability. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
lucasmrod
approved these changes
Jun 9, 2026
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## rc-minor-fleet-v4.87.0 #47166 +/- ##
=========================================================
Coverage ? 67.17%
=========================================================
Files ? 2910
Lines ? 226171
Branches ? 11828
=========================================================
Hits ? 151924
Misses ? 60508
Partials ? 13739
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
nulmete
added a commit
that referenced
this pull request
Jun 9, 2026
Cherry-pick of #47168 into the `rc-minor-fleet-v4.87.0` RC branch. Follows the cherry-pick of #47099 (#47166, already merged here). This re-timestamps the `CreateOrgLogoTable` migration `20260608173427` → `20260527215818` so it sorts immediately after `AddHostCertificatesOriginDeletedAtIndex`(required so the same migration can also ship in the 4.86.2 patch without breaking the 4.86.2 → 4.87.0 upgrade path). Note: `schema.sql` conflicted (main regenerated it against main's migration set, which differs from this RC branch) because [20260608202705_AddVulnPerfIndexes.go](https://github.com/fleetdm/fleet/blob/main/server/datastore/mysql/migrations/tables/20260608202705_AddVulnPerfIndexes.go) and [20260608210432_CleanupSoftwareLastOpenedAtSentinels.go](https://github.com/fleetdm/fleet/blob/main/server/datastore/mysql/migrations/tables/20260608210432_CleanupSoftwareLastOpenedAtSentinels.go) are not part of the RC branch yet (looks like the PRs that introduced them on main have not been cherry-picked yet).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick of #47099 into the
rc-minor-fleet-v4.87.0RC branch.Will follow-up with another cherry-pick for the migration re-ordering (since we want to include this in 4.86.2 patch, and there are several migrations before this one and the last in 4.86.1).