Fix 5XXs on software install endpoints from software row locking

[juan-fdz-hawa]

Resolves #50165

The hourly Fleet-maintained apps refresh normalized software names with four multi-table UPDATE ... JOIN (<derived table>) statements in one transaction. The derived tables need GROUP BY, so MySQL materializes them, and at scale the optimizer flips to scanning the target table once per materialized row. An UPDATE locks every row it reads -- the WHERE filter is applied after the row is locked -- so that plan held exclusive next-key locks on all of software and software_titles for the whole run. Inserts needing a shared lock on a software_titles row then blocked on the foreign key check, timing out the software install endpoints. Forcing the plan locally takes 657 next-key locks on a 651-row table while matching zero rows.

The hazard is specific to UPDATE, so discovery still joins and only the writes changed:

• Find mismatched rows with one SELECT per pass and table, join order pinned by STRAIGHT_JOIN so the catalog is the outer table and every software row is reached by index. A SELECT here is a non-locking consistent read.
• Rename by primary key in batches of 500, each its own retried single-statement transaction, so locks are at most 500 record locks and release per statement. Every UPDATE re-checks name <> ?, keeping the pass idempotent. The comparison stays in SQL to preserve the column's utf8mb4_unicode_ci collation.
• Move the pass to its own schedule (CronMacOSMaintainedAppNames) so a failed catalog fetch no longer skips it, with a back-dated first run so existing mismatches heal on upgrade. The refresh still triggers it on success, best effort, so a name change applies immediately.
• Stop applying a macOS app's name to the iOS and iPadOS titles sharing its bundle identifier.

Checklist for submitter

If some of the following don't apply, delete the relevant line.

• Changes file added for user-visible changes in changes/, orbit/changes/ or ee/fleetd-chrome/changes.
  See Changes files for more information.

• Input data is properly validated, SELECT * is avoided, SQL injection is prevented (using placeholders for values in statements), JS inline code is prevented especially for url redirects, and untrusted data interpolated into shell scripts/commands is validated against shell metacharacters.

Testing

• Added/updated automated tests
• QA'd all new/changed functionality manually

Summary by CodeRabbit

• New Features
  • Added an independent hourly schedule to reconcile macOS maintained app names.
  • Catalog refresh now reconciles names only after a successful sync.
• Bug Fixes
  • Reconciliation now processes names in bounded batches and retries transient database conflicts.
  • Reconciliation warnings no longer fail scheduled jobs, while sync failures still do.
  • Improved handling for orphaned installers, naming precedence, and macOS versus iOS/iPadOS software.
• Tests
  • Added coverage for batch processing and reconciliation edge cases.

[juan-fdz-hawa]

@coderabbitai full review

[coderabbitai]

✅ Action performed

Full review finished.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

Walkthrough

Maintained app name reconciliation is removed from catalog upserts and added to a separate premium macOS cron schedule. Successful catalog refreshes still trigger best-effort reconciliation. The datastore implementation uses primary reads and bounded, retryable rename batches. Tests cover batching, discovery windows, idempotency, orphaned installers, precedence, multi-team installers, and mobile sibling handling.

Possibly related PRs

• fleetdm/fleet#48963: Refactors and reschedules the same maintained-app reconciliation flow.
• fleetdm/fleet#50885: Modifies macOS maintained-app name reconciliation and installer/title mappings.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 20.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the primary fix for software-install endpoint 5XX errors caused by software row locking.
Description check	✅ Passed	The description explains the failure, implementation, testing, and related issue, with the main required sections substantially completed.
Linked Issues check	✅ Passed	The changes address issue #50165 by reducing locking during name normalization and preventing software-installation lock wait timeouts.
Out of Scope Changes check	✅ Passed	The schedule, reconciliation, SQL, tests, and documentation changes directly support the linked issue and stated objectives.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch 50165-5xxs-on-software-installs-endpoints-and-software-row-locking

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

❌ Patch coverage is 73.21429% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 68.58%. Comparing base (07493d8) to head (9a578a6).
⚠️ Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
server/datastore/mysql/maintained_apps.go	77.35%	7 Missing and 5 partials ⚠️
cmd/fleet/cron.go	0.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #50226      +/-   ##
==========================================
+ Coverage   62.10%   68.58%   +6.48%     
==========================================
  Files        3986     3986              
  Lines      256775   256788      +13     
  Branches    13684    13684              
==========================================
+ Hits       159463   176121   +16658     
+ Misses      82758    65012   -17746     
- Partials    14554    15655    +1101     

Flag	Coverage Δ
backend	69.67% <73.21%> (+7.73%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[coderabbitai]

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@server/datastore/mysql/maintained_apps.go`:
- Around line 237-243: Update the mismatchedTitlesByIdentifier query to restrict
target software_titles rows to Darwin/macOS sources, excluding ios_apps and
ipados_apps before reconciliation. Preserve the existing identifier and
additional_identifier conditions, and keep the iOS/iPadOS sibling-title
regression test.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: c86eecb7-4ae0-4a85-b28f-5521ebeb11e0

📥 Commits

Reviewing files that changed from the base of the PR and between 55518a0 and 5fc55ab.

📒 Files selected for processing (2)

• server/datastore/mysql/maintained_apps.go
• server/datastore/mysql/maintained_apps_test.go

🚧 Files skipped from review as they are similar to previous changes (1)

• server/datastore/mysql/maintained_apps_test.go

[getvictor]

Done with review. The core changes are solid.