Skip to content

Fix macOS software titles mis-named from embedded helper bundles (#44199) - #47831

Merged
mostlikelee merged 18 commits into
mainfrom
fix-embedded-bundle-titles-44199
Jul 9, 2026
Merged

Fix macOS software titles mis-named from embedded helper bundles (#44199)#47831
mostlikelee merged 18 commits into
mainfrom
fix-embedded-bundle-titles-44199

Conversation

@mostlikelee

@mostlikelee mostlikelee commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Related issue: Resolves #44199

osquery reports an app's embedded login helper under Contents/Library/LoginItems/ as a separate apps row sharing the parent's bundle identifier; whichever row created the software_titles row wins its name, often the helper. Two changes pair:

  • Filter (queries.go): drop rows whose path matches %.app/Contents/%. Prospective.
  • Backfill migration: recompute existing apps titles from sibling software names (FMA → longest-common-prefix → shortest), UPDATE only on diff.

cmd/osquery-perf gains --embedded_bundle_paths to nest duplicate-bundle paths so the bug is reproducible at scale.

Migration perf (osquery-perf library, the largest realistic dataset available)

Titles Software rows Candidates Wall-clock
18,565 60,372 1,146 (6.2%) ~191 ms

EXPLAIN ANALYZE: PK scan on software_titles + title_id-indexed lookups on software. No table scans, no filesort. Linear extrapolation to ~5× scale stays well under 1s.

Checklist for submitter

  • Changes file added for user-visible changes in changes/

  • Input data is properly validated, SELECT * is avoided, SQL injection is prevented (using placeholders for values in statements), JS inline code is prevented especially for url redirects, and untrusted data interpolated into shell scripts/commands is validated against shell metacharacters.

Testing

  • Added/updated automated tests
  • QA'd all new/changed functionality manually

For unreleased bug fixes in a release candidate, one of:

  • Confirmed that the fix is not expected to adversely impact load test results

Database migrations

  • Checked schema for all modified table for columns that will auto-update timestamps during migration.
  • Confirmed that updating the timestamps is acceptable, and will not cause unwanted side effects.
  • Ensured the correct collation is explicitly set for character columns (COLLATE utf8mb4_unicode_ci).

Summary by CodeRabbit

Release Notes

  • New Features

    • Added a macOS option to generate duplicate bundle paths using nested embedded locations (off by default).
  • Bug Fixes

    • Prevented macOS app display names from being replaced by embedded login-helper names when bundle identifiers match.
    • Improved macOS software inventory filtering by excluding app subpaths under Contents.
  • Database / Migration

    • Added a one-time migration to correct misnamed macOS software titles by recomputing them from related app title data.
    • Updated migration status table seed/auto-increment values.

Tim Lee added 4 commits June 18, 2026 06:24
…es under parent

Reproduces #44199 by emitting embedded-helper paths
(/some/path/Common_N.app/Contents/Library/LoginItems/<name>.app) when
the new flag is set, enabling load tests of the ingestion filter and
title backfill migration that follow. Default off — existing
duplicate-bundle behavior unchanged.
…uery

osquery's apps table reports both a parent app and any embedded login
helper under Contents/Library/LoginItems as separate rows, often sharing
a bundle identifier. The helper's name would clobber the software
title's display name (e.g. AmphetamineLoginHelper instead of
Amphetamine). Exclude paths nested under .app/Contents/ so only
top-level bundles are ingested.

Fixes #44199 prospectively. Existing mis-named titles are addressed by
the follow-up backfill migration.
For macOS app titles whose software_titles.name was set from an
embedded helper bundle (e.g. AmphetamineLoginHelper) rather than the
parent app, recompute the name from the title's sibling software rows
using the same precedence as title creation:

  1. Fleet-maintained app canonical name (if bundle id matches), else
  2. longest-common-prefix of sibling names (trailing non-word chars
     trimmed), else
  3. shortest sibling name.

The migration drives off the indexed software_titles JOIN software
join (title_id) and only UPDATEs titles whose name actually changes —
deliberately not path-based, since host_software_installed_paths has
no global per-software source of truth and would force a full
unindexed TEXT scan inside the startup-blocking migration
transaction.

Pairs with the queries.go filter that prevents new mis-named titles
from forming.
@mostlikelee

Copy link
Copy Markdown
Contributor Author

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Full review finished.

@coderabbitai

coderabbitai Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 8e938b8f-3b76-4cc5-bf3d-f82954eb2c7f

📥 Commits

Reviewing files that changed from the base of the PR and between bdb9cb2 and e55ea24.

📒 Files selected for processing (1)
  • server/datastore/mysql/schema.sql
🚧 Files skipped from review as they are similar to previous changes (1)
  • server/datastore/mysql/schema.sql

Walkthrough

This PR fixes a macOS software title naming issue where a login-helper app nested inside another app bundle could be treated as the canonical title name. Future ingestion excludes %.app/Contents/% paths from the macOS software query. A new migration updates existing software_titles.name values by using Fleet maintained app names, then longest common prefix, then shortest sibling name. The osquery-perf agent also adds a flag to generate nested embedded bundle paths for this scenario.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 11.11% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title is concise and clearly describes the main macOS helper-bundle name fix.
Description check ✅ Passed The description includes the related issue, change summary, testing, and migration notes, and it mostly matches the template.
Linked Issues check ✅ Passed The path filter and migration/backfill address the duplicate-bundle name bug by favoring the non-helper title, matching #44199.
Out of Scope Changes check ✅ Passed The schema, changes file, and osquery-perf updates all support the same macOS title fix and do not appear unrelated.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix-embedded-bundle-titles-44199
⚔️ Resolve merge conflicts
  • Resolve merge conflict in branch fix-embedded-bundle-titles-44199

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
server/datastore/mysql/migrations/tables/20260618124430_FixEmbeddedBundleTitleNames.go (1)

105-120: 💤 Low value

Consider deterministic tie-breaking for equal-length names.

The slice at line 105 is built from map iteration, which is non-deterministic in Go. If two sibling names have identical length, the "shortest" selection depends on iteration order.

For this one-shot migration with typical helper naming patterns, the practical impact is minimal, but sorting names lexicographically before the shortest-selection loop would make results reproducible.

♻️ Optional fix for deterministic selection
+import "sort"
+
 func fixEmbeddedPickTitleName(siblings map[string]struct{}, bundleID string, fmaNames map[string]string) string {
 	if name, ok := fmaNames[bundleID]; ok && name != "" {
 		return name
 	}
 	names := make([]string, 0, len(siblings))
 	for n := range siblings {
 		names = append(names, n)
 	}
+	sort.Strings(names)
 	prefix := fixEmbeddedLongestCommonPrefix(names)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@server/datastore/mysql/migrations/tables/20260618124430_FixEmbeddedBundleTitleNames.go`
around lines 105 - 120, The names slice is built from non-deterministic map
iteration over the siblings map, which means when multiple sibling names have
identical length, the shortest name selection depends on the iteration order.
Sort the names slice lexicographically before the loop that finds the shortest
name by calling sort.Strings(names) after the prefix check and before the loop
that iterates through names[1:] to select the shortest, ensuring deterministic
results regardless of map iteration order.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In
`@server/datastore/mysql/migrations/tables/20260618124430_FixEmbeddedBundleTitleNames.go`:
- Around line 105-120: The names slice is built from non-deterministic map
iteration over the siblings map, which means when multiple sibling names have
identical length, the shortest name selection depends on the iteration order.
Sort the names slice lexicographically before the loop that finds the shortest
name by calling sort.Strings(names) after the prefix check and before the loop
that iterates through names[1:] to select the shortest, ensuring deterministic
results regardless of map iteration order.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: cfb47ebb-5c84-4c09-90ca-4bffb1a89949

📥 Commits

Reviewing files that changed from the base of the PR and between 443d82d and 2d018f2.

⛔ Files ignored due to path filters (1)
  • docs/Contributing/product-groups/orchestration/understanding-host-vitals.md is excluded by !**/*.md
📒 Files selected for processing (5)
  • changes/44199-embedded-bundle-title-name
  • cmd/osquery-perf/agent.go
  • server/datastore/mysql/migrations/tables/20260618124430_FixEmbeddedBundleTitleNames.go
  • server/datastore/mysql/migrations/tables/20260618124430_FixEmbeddedBundleTitleNames_test.go
  • server/service/osquery_utils/queries.go

@codecov

codecov Bot commented Jun 18, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 61.90476% with 40 lines in your changes missing coverage. Please review.
✅ Project coverage is 68.07%. Comparing base (cb09b42) to head (5d30086).
⚠️ Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
...bles/20260708153912_FixEmbeddedBundleTitleNames.go 68.42% 16 Missing and 14 partials ⚠️
cmd/osquery-perf/agent.go 0.00% 10 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #47831      +/-   ##
==========================================
- Coverage   68.11%   68.07%   -0.04%     
==========================================
  Files        3709     3689      -20     
  Lines      235099   234997     -102     
  Branches    12352    12274      -78     
==========================================
- Hits       160129   159976     -153     
- Misses      60594    60634      +40     
- Partials    14376    14387      +11     
Flag Coverage Δ
backend 69.73% <61.90%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Refactor Up_20260618124430 to delegate the title scan to
fixEmbeddedScanTitles, where defer rows.Close() satisfies
sqlclosecheck without the manual error-path close.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

  • Copilot's review of this pull request may be incomplete because some of the changed files are excluded by your Copilot content exclusion settings. See Excluding content from Copilot for details.

Pull request overview

This PR fixes incorrect macOS software title names caused by embedded helper bundles (e.g., LoginItems) sharing the parent app’s bundle identifier, ensuring the parent app name wins both for new ingests and for existing data.

Changes:

  • Filter macOS apps rows whose path indicates an embedded bundle under *.app/Contents/* during osquery ingestion.
  • Add a one-time MySQL migration to recompute affected software_titles.name values using FMA name override → longest common prefix → shortest fallback.
  • Extend cmd/osquery-perf to optionally generate embedded-bundle installed paths so the scenario is reproducible at scale.

Reviewed changes

Copilot reviewed 5 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
server/service/osquery_utils/queries.go Excludes embedded *.app/Contents/* paths from the macOS apps software query to prevent future mis-naming.
server/datastore/mysql/schema.sql Advances migration_status_tables seed data / AUTO_INCREMENT to include the new migration.
server/datastore/mysql/migrations/tables/20260618124430_FixEmbeddedBundleTitleNames.go Implements the backfill migration that recomputes software_titles.name for affected app titles.
server/datastore/mysql/migrations/tables/20260618124430_FixEmbeddedBundleTitleNames_test.go Adds coverage validating the migration’s name-picking precedence and out-of-scope behavior.
cmd/osquery-perf/agent.go Adds --embedded_bundle_paths option to generate nested Contents/Library/LoginItems/... paths for duplicate-bundle scenarios.
changes/44199-embedded-bundle-title-name User-visible change entry (content excluded from review by policy).
docs/Contributing/product-groups/orchestration/understanding-host-vitals.md Documentation update (content excluded from review by policy).
Files excluded by content exclusion policy (2)
  • changes/44199-embedded-bundle-title-name
  • docs/Contributing/product-groups/orchestration/understanding-host-vitals.md

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

fixEmbeddedLoadFMANamesDarwin now wraps query/scan/iteration errors
with descriptive messages, matching the sibling fixEmbeddedScanTitles
helper. Makes migration failure traces pinpoint the failing step.
@mostlikelee
mostlikelee marked this pull request as ready for review June 18, 2026 18:49
@mostlikelee
mostlikelee requested review from a team, sgress454 and sharon-fdm as code owners June 18, 2026 18:49

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.

Tip: disable this comment in your organization's Code Review settings.

@qodo-free-for-open-source-projects

Copy link
Copy Markdown

CI Feedback 🧐

A test triggered by this PR failed. Here is an AI-generated analysis of the failure:

Action: test-go (fleetctl, mysql:8.0.44) / test

Failed stage: Run Go Tests [❌]

Failed test name: TestGitOpsFullGlobal

Failure summary:

The action failed because Go tests in ./cmd/fleetctl/... exited non-zero after TestGitOpsFullGlobal
failed in both subtests:
- TestGitOpsFullGlobal/useDeprecatedKeys=false and
TestGitOpsFullGlobal/useDeprecatedKeys=true failed at cmd/fleetctl/fleetctl/testing_utils_test.go:20
(called from cmd/fleetctl/fleetctl/gitops_test.go:2244).
- Failure cause: the test attempted to
apply MDM custom settings via POST /api/latest/fleet/mdm/profiles/batch, but the API returned 422
Validation Failed with the message cannot set custom settings: Windows MDM isn't turned on.
- This
caused make .run-go-tests to fail (Makefile:302), which then failed make test-go (Makefile:417) and
ended the job with exit code 2.

Relevant error logs:
1:  Runner name: 'ubuntu-8core-1000964423'
2:  Runner group name: 'default larger runners'
...

1252:  �[36;1mattempt=1�[0m
1253:  �[36;1m�[0m
1254:  �[36;1mwhile [ $attempt -le $max_attempts ]; do�[0m
1255:  �[36;1m  echo "Attempt $attempt of $max_attempts"�[0m
1256:  �[36;1m�[0m
1257:  �[36;1m  # Try to connect to MySQL�[0m
1258:  �[36;1m  if wait_for_mysql "mysql_test"; then�[0m
1259:  �[36;1m    # If MySQL is ready, try to connect to MySQL replica�[0m
1260:  �[36;1m    if wait_for_mysql "mysql_replica_test"; then�[0m
1261:  �[36;1m      # Both are ready, we're done�[0m
1262:  �[36;1m      echo "All MySQL connections successful"�[0m
1263:  �[36;1m      exit 0�[0m
1264:  �[36;1m    fi�[0m
1265:  �[36;1m  fi�[0m
1266:  �[36;1m�[0m
1267:  �[36;1m  # If we get here, at least one connection failed�[0m
1268:  �[36;1m  echo "Failed to connect to MySQL on attempt $attempt"�[0m
1269:  �[36;1m�[0m
1270:  �[36;1m  if [ $attempt -lt $max_attempts ]; then�[0m
1271:  �[36;1m    echo "Restarting containers and trying again..."�[0m
1272:  �[36;1m    restart_containers�[0m
1273:  �[36;1m  else�[0m
1274:  �[36;1m    echo "Maximum attempts reached. Failing the job."�[0m
1275:  �[36;1m    exit 1�[0m
...

1423:  ARTIFACT_PREFIX: fleetctl-mysql8.0.44
1424:  GOTOOLCHAIN: local
1425:  FLEET_PREVIEW_TAG: dev
1426:  ##[endgroup]
1427:  make .run-go-tests PKG_TO_TEST="./cmd/fleetctl/..."
1428:  make[1]: Entering directory '/home/runner/work/fleet/fleet'
1429:  Running Go tests with gotestsum:
1430:  gotestsum --format=testdox --jsonfile=/tmp/test-output.json -- -tags full,fts5,netgo -run=  -v -race=false -timeout=20m  -parallel 8 -coverprofile=coverage.txt -covermode=atomic -coverpkg=github.com/fleetdm/fleet/v4/... ././cmd/fleetctl/... 
1431:  github.com/fleetdm/fleet/v4/cmd/fleetctl:
1432:  github.com/fleetdm/fleet/v4/cmd/fleetctl/integrationtest:
1433:  github.com/fleetdm/fleet/v4/cmd/fleetctl/fleetctl/goquerycmd:
1434:  github.com/fleetdm/fleet/v4/cmd/fleetctl/fleetctl/testing_utils:
1435:  github.com/fleetdm/fleet/v4/cmd/fleetctl/fleetctl/fleetctltest:
1436:  github.com/fleetdm/fleet/v4/cmd/fleetctl/integrationtest/preview:
1437:  �[32m✓�[0m Integrations preview (44.44s)
1438:  �[32m✓�[0m Preview fails on invalid license key (0.00s)
1439:  github.com/fleetdm/fleet/v4/cmd/fleetctl/integrationtest/package:
...

1550:  �[32m✓�[0m Apply specs deprecated keys app config windows updates.grace period days not a number (0.57s)
1551:  �[32m✓�[0m Apply specs deprecated keys app config windows updates.grace period days out of range (0.47s)
1552:  �[32m✓�[0m Apply specs deprecated keys config with FIM values for agent options (#869 9) (0.53s)
1553:  �[32m✓�[0m Apply specs deprecated keys config with blank required org name (0.47s)
1554:  �[32m✓�[0m Apply specs deprecated keys config with blank required server url (0.43s)
1555:  �[32m✓�[0m Apply specs deprecated keys config with invalid agent options command-line flags (0.54s)
1556:  �[32m✓�[0m Apply specs deprecated keys config with invalid agent options data type in dry-run (0.61s)
1557:  �[32m✓�[0m Apply specs deprecated keys config with invalid agent options data type with force (0.50s)
1558:  �[32m✓�[0m Apply specs deprecated keys config with invalid agent options in dry-run (0.64s)
1559:  �[32m✓�[0m Apply specs deprecated keys config with invalid key type (0.55s)
1560:  �[32m✓�[0m Apply specs deprecated keys config with invalid value for agent options command-line flags (0.53s)
1561:  �[32m✓�[0m Apply specs deprecated keys config with unknown key (0.38s)
1562:  �[32m✓�[0m Apply specs deprecated keys config with valid agent options command-line flags (0.42s)
1563:  �[32m✓�[0m Apply specs deprecated keys dry-run set with unsupported spec (0.48s)
1564:  �[32m✓�[0m Apply specs deprecated keys dry-run set with various specs, appconfig warning for legacy (0.48s)
1565:  �[32m✓�[0m Apply specs deprecated keys dry-run set with various specs, no errors (0.69s)
1566:  �[32m✓�[0m Apply specs deprecated keys empty config (0.49s)
...

1569:  �[32m✓�[0m Apply specs deprecated keys invalid agent options dry-run (0.39s)
1570:  �[32m✓�[0m Apply specs deprecated keys invalid agent options field type (0.52s)
1571:  �[32m✓�[0m Apply specs deprecated keys invalid agent options field type in overrides (0.48s)
1572:  �[32m✓�[0m Apply specs deprecated keys invalid agent options for existing team (0.69s)
1573:  �[32m✓�[0m Apply specs deprecated keys invalid agent options for new team (0.46s)
1574:  �[32m✓�[0m Apply specs deprecated keys invalid agent options force (0.41s)
1575:  �[32m✓�[0m Apply specs deprecated keys invalid known key's value type for team cannot be forced (0.50s)
1576:  �[32m✓�[0m Apply specs deprecated keys invalid team agent options command-line flag (0.51s)
1577:  �[32m✓�[0m Apply specs deprecated keys invalid top-level key for team (0.48s)
1578:  �[32m✓�[0m Apply specs deprecated keys macos updates deadline set but minimum version empty (0.58s)
1579:  �[32m✓�[0m Apply specs deprecated keys macos updates minimum version set but deadline empty (0.48s)
1580:  �[32m✓�[0m Apply specs deprecated keys macos updates.deadline with incomplete date (0.45s)
1581:  �[32m✓�[0m Apply specs deprecated keys macos updates.deadline with invalid date (0.47s)
1582:  �[32m✓�[0m Apply specs deprecated keys macos updates.deadline with timestamp (0.55s)
1583:  �[32m✓�[0m Apply specs deprecated keys macos updates.minimum version with build version (0.66s)
1584:  �[32m✓�[0m Apply specs deprecated keys missing required failing policies destination url (0.48s)
1585:  �[32m✓�[0m Apply specs deprecated keys missing required host status days count (0.40s)
...

1593:  �[32m✓�[0m Apply specs deprecated keys team config macos settings.enable disk encryption true (0.36s)
1594:  �[32m✓�[0m Apply specs deprecated keys team config macos settings.enable disk encryption with invalid value type (0.50s)
1595:  �[32m✓�[0m Apply specs deprecated keys team config macos settings.enable disk encryption without a value (0.66s)
1596:  �[32m✓�[0m Apply specs deprecated keys unknown key for team can be forced (0.70s)
1597:  �[32m✓�[0m Apply specs deprecated keys valid team agent options command-line flag (0.44s)
1598:  �[32m✓�[0m Apply specs deprecated keys windows updates unset valid (0.57s)
1599:  �[32m✓�[0m Apply specs deprecated keys windows updates valid (0.51s)
1600:  �[32m✓�[0m Apply specs deprecated keys windows updates.deadline days but grace period empty (0.44s)
1601:  �[32m✓�[0m Apply specs deprecated keys windows updates.deadline days not a number (0.40s)
1602:  �[32m✓�[0m Apply specs deprecated keys windows updates.deadline days out of range (0.48s)
1603:  �[32m✓�[0m Apply specs deprecated keys windows updates.grace period days but deadline empty (0.50s)
1604:  �[32m✓�[0m Apply specs deprecated keys windows updates.grace period days not a number (0.54s)
1605:  �[32m✓�[0m Apply specs deprecated keys windows updates.grace period days out of range (0.41s)
1606:  �[32m✓�[0m Apply specs dry-run set with unsupported spec (0.49s)
1607:  �[32m✓�[0m Apply specs dry-run set with various specs, appconfig warning for legacy (0.54s)
1608:  �[32m✓�[0m Apply specs dry-run set with various specs, no errors (0.53s)
1609:  �[32m✓�[0m Apply specs empty config (0.43s)
...

1612:  �[32m✓�[0m Apply specs invalid agent options dry-run (0.41s)
1613:  �[32m✓�[0m Apply specs invalid agent options field type (0.43s)
1614:  �[32m✓�[0m Apply specs invalid agent options field type in overrides (0.47s)
1615:  �[32m✓�[0m Apply specs invalid agent options for existing team (0.38s)
1616:  �[32m✓�[0m Apply specs invalid agent options for new team (0.60s)
1617:  �[32m✓�[0m Apply specs invalid agent options force (0.45s)
1618:  �[32m✓�[0m Apply specs invalid known key's value type for team cannot be forced (0.45s)
1619:  �[32m✓�[0m Apply specs invalid team agent options command-line flag (0.41s)
1620:  �[32m✓�[0m Apply specs invalid top-level key for team (0.48s)
1621:  �[32m✓�[0m Apply specs macos updates deadline set but minimum version empty (0.63s)
1622:  �[32m✓�[0m Apply specs macos updates minimum version set but deadline empty (0.39s)
1623:  �[32m✓�[0m Apply specs macos updates.deadline with incomplete date (0.51s)
1624:  �[32m✓�[0m Apply specs macos updates.deadline with invalid date (0.39s)
1625:  �[32m✓�[0m Apply specs macos updates.deadline with timestamp (0.45s)
1626:  �[32m✓�[0m Apply specs macos updates.minimum version with build version (0.47s)
1627:  �[32m✓�[0m Apply specs missing required failing policies destination url (0.48s)
1628:  �[32m✓�[0m Apply specs missing required host status days count (0.46s)
...

1647:  �[32m✓�[0m Apply specs windows updates.grace period days not a number (0.39s)
1648:  �[32m✓�[0m Apply specs windows updates.grace period days out of range (0.40s)
1649:  �[32m✓�[0m Apply team specs (0.48s)
1650:  �[32m✓�[0m Apply user roles (0.45s)
1651:  �[32m✓�[0m Apply user roles deprecated (0.45s)
1652:  �[32m✓�[0m Apply windows updates (0.36s)
1653:  �[32m✓�[0m Apply windows updates field omitted (0.00s)
1654:  �[32m✓�[0m Apply windows updates with null values (0.00s)
1655:  �[32m✓�[0m Apply windows updates with values (0.00s)
1656:  �[32m✓�[0m Can apply intervals in nanoseconds (0.49s)
1657:  �[32m✓�[0m Can apply intervals using durations (0.49s)
1658:  �[32m✓�[0m Clean status code err (0.00s)
1659:  �[32m✓�[0m Clean status code err bare wrapped status code err (0.00s)
1660:  �[32m✓�[0m Clean status code err nil (0.00s)
1661:  �[32m✓�[0m Clean status code err outer-wrapped status code err (0.00s)
1662:  �[32m✓�[0m Clean status code err plain error untouched (0.00s)
1663:  �[32m✓�[0m Compute label changes (0.00s)
...

1719:  �[32m✓�[0m Filename functions (0.00s)
1720:  �[32m✓�[0m Filename functions outfile name builds a file name using the name provided + current time (0.00s)
1721:  �[32m✓�[0m Filename functions outfile name with ext builds a file name using the name and extension provided + current time (0.00s)
1722:  �[32m✓�[0m FleetctlUpgradePacks empty packs (0.43s)
1723:  �[32m✓�[0m FleetctlUpgradePacks no pack (0.49s)
1724:  �[32m✓�[0m FleetctlUpgradePacks non empty (0.38s)
1725:  �[32m✓�[0m FleetctlUpgradePacks not admin (0.50s)
1726:  �[32m✓�[0m Format XML (0.00s)
1727:  �[32m✓�[0m Format XML XML with attributes (0.00s)
1728:  �[32m✓�[0m Format XML basic XML (0.00s)
1729:  �[32m✓�[0m Format XML empty XML (0.00s)
1730:  �[32m✓�[0m Format XML invalid XML (0.00s)
1731:  �[32m✓�[0m Format XML nested XML (0.00s)
1732:  �[32m✓�[0m Generate MDM apple (0.82s)
1733:  �[32m✓�[0m Generate MDM apple BM (0.39s)
1734:  �[32m✓�[0m Generate MDM apple CSR API call fails (0.42s)
1735:  �[32m✓�[0m Generate MDM apple successful run (0.40s)
1736:  �[32m✓�[0m Generate MDMVPP tokens (0.00s)
1737:  �[32m✓�[0m Generate MDMVPP tokens get VPP tokens error (0.00s)
1738:  �[32m✓�[0m Generate MDMVPP tokens multiple tokens with different teams (0.00s)
...

1756:  �[32m✓�[0m Generate org settings masked google workspace api key (0.00s)
1757:  �[32m✓�[0m Generate policies (0.00s)
1758:  �[32m✓�[0m Generate policies patch policy orphaned from fleet maintained app (0.00s)
1759:  �[32m✓�[0m Generate queries (0.00s)
1760:  �[32m✓�[0m Generate software (0.00s)
1761:  �[32m✓�[0m Generate software auto update schedule (0.00s)
1762:  �[32m✓�[0m Generate software script packages (0.00s)
1763:  �[32m✓�[0m Generate team settings (0.00s)
1764:  �[32m✓�[0m Generate team settings insecure (0.00s)
1765:  �[32m✓�[0m Generated org settings no SSO (0.00s)
1766:  �[32m✓�[0m Generated org settings okta conditional access not included (0.00s)
1767:  �[32m✓�[0m Get MDM command results (0.45s)
1768:  �[32m✓�[0m Get MDM command results command flag required (0.00s)
1769:  �[32m✓�[0m Get MDM command results command not found (0.01s)
1770:  �[32m✓�[0m Get MDM command results command results empty (0.01s)
1771:  �[32m✓�[0m Get MDM command results command results error (0.01s)
1772:  �[32m✓�[0m Get MDM command results darwin command results (0.00s)
1773:  �[32m✓�[0m Get MDM command results host specific results (0.00s)
1774:  �[32m✓�[0m Get MDM command results windows command results (0.00s)
1775:  �[32m✓�[0m Get MDM commands (0.39s)
1776:  �[32m✓�[0m Get apple BM (1.73s)
1777:  �[32m✓�[0m Get apple BM free license (0.41s)
1778:  �[32m✓�[0m Get apple BM premium license, multiple tokens (0.43s)
1779:  �[32m✓�[0m Get apple BM premium license, no token (0.41s)
1780:  �[32m✓�[0m Get apple BM premium license, single token (0.49s)
1781:  �[32m✓�[0m Get apple MDM (0.35s)
1782:  �[32m✓�[0m Get carve (0.37s)
1783:  �[32m✓�[0m Get carve with error (0.43s)
1784:  �[32m✓�[0m Get carves (0.47s)
...

1798:  �[32m✓�[0m Get hosts MDM get hosts - -mdm - -mdm-pending - (0.00s)
1799:  �[32m✓�[0m Get hosts MDM get hosts - -mdm-pending - -yaml - expected list hosts yaml.yml (0.00s)
1800:  �[32m✓�[0m Get hosts get hosts - -json - -remove-deprecated-keys (0.00s)
1801:  �[32m✓�[0m Get hosts get hosts - -json - expected list hosts json.json (0.00s)
1802:  �[32m✓�[0m Get hosts get hosts - -json test host - expected host detail response json.json (0.00s)
1803:  �[32m✓�[0m Get hosts get hosts - -yaml - expected list hosts yaml.yml (0.01s)
1804:  �[32m✓�[0m Get hosts get hosts - -yaml test host - expected host detail response yaml.yml (0.00s)
1805:  �[32m✓�[0m Get label (0.49s)
1806:  �[32m✓�[0m Get label usage include and exclude allowed (0.00s)
1807:  �[32m✓�[0m Get label usage include and exclude allowed macos (0.00s)
1808:  �[32m✓�[0m Get label usage include and exclude allowed macos# 01 (0.00s)
1809:  �[32m✓�[0m Get label usage include and exclude allowed macos# 02 (0.00s)
1810:  �[32m✓�[0m Get label usage include and exclude allowed windows (0.00s)
1811:  �[32m✓�[0m Get label usage include and exclude allowed windows# 01 (0.00s)
1812:  �[32m✓�[0m Get label usage include and exclude allowed windows# 02 (0.00s)
1813:  �[32m✓�[0m Get label usage include exclude overlap error (0.00s)
1814:  �[32m✓�[0m Get label usage include exclude overlap error macos (0.00s)
1815:  �[32m✓�[0m Get label usage include exclude overlap error macos# 01 (0.00s)
1816:  �[32m✓�[0m Get label usage include exclude overlap error macos# 02 (0.00s)
1817:  �[32m✓�[0m Get label usage include exclude overlap error windows (0.00s)
1818:  �[32m✓�[0m Get label usage include exclude overlap error windows# 01 (0.00s)
1819:  �[32m✓�[0m Get label usage include exclude overlap error windows# 02 (0.00s)
1820:  �[32m✓�[0m Get label usage multiple label keys error (0.00s)
1821:  �[32m✓�[0m Get label usage multiple label keys error macos (0.00s)
1822:  �[32m✓�[0m Get label usage multiple label keys error windows (0.00s)
1823:  �[32m✓�[0m Get label usage policy scopes (0.00s)
...

1839:  �[32m✓�[0m Get queries as observer team observer (0.01s)
1840:  �[32m✓�[0m Get query (0.52s)
1841:  �[32m✓�[0m Get query labels include all (0.43s)
1842:  �[32m✓�[0m Get reports labels include all (0.41s)
1843:  �[32m✓�[0m Get software titles (0.44s)
1844:  �[32m✓�[0m Get software versions (0.37s)
1845:  �[32m✓�[0m Get teams (1.02s)
1846:  �[32m✓�[0m Get teams YAML and apply (0.50s)
1847:  �[32m✓�[0m Get teams by name (0.47s)
1848:  �[32m✓�[0m Get teams expired license (0.46s)
1849:  �[32m✓�[0m Get teams not expired license (0.56s)
1850:  �[32m✓�[0m Get teams software from source of truth (0.46s)
1851:  �[32m✓�[0m Get user roles (0.35s)
1852:  �[32m✓�[0m Git ops ABM (6.57s)
1853:  �[32m✓�[0m Git ops ABM backwards compat (0.51s)
1854:  �[32m✓�[0m Git ops ABM both keys errors (0.64s)
1855:  �[32m✓�[0m Git ops ABM deprecated config with two tokens in the db fails (0.66s)
1856:  �[32m✓�[0m Git ops ABM new key all valid (0.80s)
1857:  �[32m✓�[0m Git ops ABM new key multiple elements (0.87s)
1858:  �[32m✓�[0m Git ops ABM no team is supported (0.54s)
1859:  �[32m✓�[0m Git ops ABM non existent org name fails (0.55s)
1860:  �[32m✓�[0m Git ops ABM not provided teams defaults to no team (0.53s)
1861:  �[32m✓�[0m Git ops ABM renamed new key all valid (0.79s)
1862:  �[32m✓�[0m Git ops ABM using an undefined team errors (0.67s)
1863:  �[32m✓�[0m Git ops EULA setting (4.22s)
...

1866:  �[32m✓�[0m Git ops EULA setting not a PDF file (0.42s)
1867:  �[32m✓�[0m Git ops EULA setting relative path to working dir to pdf file (no existing EULA uploaded) (0.50s)
1868:  �[32m✓�[0m Git ops EULA setting relative path to yaml file to pdf file (no existing EULA uploaded) (0.50s)
1869:  �[32m✓�[0m Git ops EULA setting uploading the same EULA again (0.56s)
1870:  �[32m✓�[0m Git ops EULA setting valid new pdf file (different EULA already uploaded) (0.47s)
1871:  �[32m✓�[0m Git ops EULA setting valid pdf file (no existing EULA uploaded) (0.57s)
1872:  �[32m✓�[0m Git ops MDM auth settings (0.59s)
1873:  �[32m✓�[0m Git ops SMTP settings (0.66s)
1874:  �[32m✓�[0m Git ops SSO server URL (0.82s)
1875:  �[32m✓�[0m Git ops SSO settings (0.47s)
1876:  �[32m✓�[0m Git ops android certificates add (0.65s)
1877:  �[32m✓�[0m Git ops android certificates change (0.57s)
1878:  �[32m✓�[0m Git ops android certificates delete all (0.65s)
1879:  �[32m✓�[0m Git ops android certificates delete one (0.66s)
1880:  �[32m✓�[0m Git ops app store app auto update (0.52s)
1881:  �[32m✓�[0m Git ops app store app auto update invalid auto-update window triggers error and does not call update software title auto update config (0.01s)
1882:  �[32m✓�[0m Git ops app store app auto update no auto update settings and no existing schedule does not call update software title auto update config (0.02s)
1883:  �[32m✓�[0m Git ops app store app auto update update software title auto update config is applied for i OS VPP apps (0.02s)
1884:  �[32m✓�[0m Git ops app store app auto update update software title auto update config is not called when no VPP apps provided (0.03s)
1885:  �[32m✓�[0m Git ops apple OS updates (0.44s)
1886:  �[32m✓�[0m Git ops apple OS updates ios updates (0.01s)
1887:  �[32m✓�[0m Git ops apple OS updates ios updates os updated when existing OS update declaration (0.01s)
1888:  �[32m✓�[0m Git ops apple OS updates ipados updates (0.01s)
1889:  �[32m✓�[0m Git ops apple OS updates ipados updates os updated when existing OS update declaration (0.01s)
1890:  �[32m✓�[0m Git ops apple OS updates macos updates (0.01s)
1891:  �[32m✓�[0m Git ops apple OS updates macos updates os updated when existing OS update declaration (0.01s)
1892:  �[32m✓�[0m Git ops basic global and no team (0.66s)
1893:  �[32m✓�[0m Git ops basic global and no team basic global and no-team.yml (0.05s)
1894:  �[32m✓�[0m Git ops basic global and no team both global and no-team.yml define controls -- should fail (0.02s)
1895:  �[32m✓�[0m Git ops basic global and no team controls only defined in no-team.yml (0.05s)
1896:  �[32m✓�[0m Git ops basic global and no team global DOES NOT define controls -- should fail (0.01s)
1897:  �[32m✓�[0m Git ops basic global and no team global and no-team.yml DO NOT define controls -- should fail (0.01s)
1898:  �[32m✓�[0m Git ops basic global and no team global defines software -- should fail (0.01s)
1899:  �[32m✓�[0m Git ops basic global and no team no-team provided without global -- should fail (0.01s)
1900:  �[32m✓�[0m Git ops basic global and no team no-team.yml defines policy with calendar events enabled -- should fail (0.01s)
1901:  �[32m✓�[0m Git ops basic global and no team unassigned provided without global -- should fail (0.01s)
1902:  �[32m✓�[0m Git ops basic global and team (0.59s)
...

1908:  �[32m✓�[0m Git ops custom settings global macos windows custom settings valid.yml (0.63s)
1909:  �[32m✓�[0m Git ops custom settings global windows custom settings invalid label mix 2 .yml (0.58s)
1910:  �[32m✓�[0m Git ops custom settings global windows custom settings invalid label mix.yml (0.52s)
1911:  �[32m✓�[0m Git ops custom settings global windows custom settings unknown label.yml (0.56s)
1912:  �[32m✓�[0m Git ops custom settings team macos custom settings valid deprecated.yml (0.56s)
1913:  �[32m✓�[0m Git ops custom settings team macos windows custom settings invalid labels mix 2 .yml (0.53s)
1914:  �[32m✓�[0m Git ops custom settings team macos windows custom settings invalid labels mix.yml (0.48s)
1915:  �[32m✓�[0m Git ops custom settings team macos windows custom settings unknown label.yml (0.52s)
1916:  �[32m✓�[0m Git ops custom settings team macos windows custom settings valid.yml (0.64s)
1917:  �[32m✓�[0m Git ops dry run rejects invalid label platform (0.35s)
1918:  �[32m✓�[0m Git ops exception enforcement (0.60s)
1919:  �[32m✓�[0m Git ops exception enforcement free tier (0.40s)
1920:  �[32m✓�[0m Git ops exceptions preserve omitted keys (0.45s)
1921:  �[32m✓�[0m Git ops features (0.62s)
1922:  �[32m✓�[0m Git ops filename validation (0.00s)
1923:  �[32m✓�[0m Git ops fleet failing policies webhook policy IDs (0.53s)
1924:  �[32m✓�[0m Git ops fleet webhooks and tickets enabled (0.67s)
...

2081:  �[32m✓�[0m New basic file structure has expected files (0.00s)
2082:  �[32m✓�[0m New basic file structure replaces and escapes org name template var (0.00s)
2083:  �[32m✓�[0m New basic file structure strips .template. from output filenames (0.00s)
2084:  �[32m✓�[0m New dir flag (0.01s)
2085:  �[32m✓�[0m New existing dir with force (0.01s)
2086:  �[32m✓�[0m New existing dir without force (0.00s)
2087:  �[32m✓�[0m New org name YAML quoting (0.01s)
2088:  �[32m✓�[0m New org name validation (0.01s)
2089:  �[32m✓�[0m New org name validation at max length (0.01s)
2090:  �[32m✓�[0m New org name validation control characters stripped (0.01s)
2091:  �[32m✓�[0m New org name validation only control characters (0.00s)
2092:  �[32m✓�[0m New org name validation only whitespace (0.00s)
2093:  �[32m✓�[0m New org name validation too long (0.00s)
2094:  �[32m✓�[0m New output messages (0.01s)
2095:  �[32m✓�[0m New template stripping (0.01s)
2096:  �[32m✓�[0m Print auth error (0.41s)
2097:  �[32m✓�[0m Print auth error SSO disabled shows default login message (0.00s)
2098:  �[32m✓�[0m Print auth error SSO enabled shows SSO instructions (0.00s)
2099:  �[32m✓�[0m Render template (0.00s)
...

2119:  �[32m✓�[0m Run api command get scripts full path missing (0.00s)
2120:  �[32m✓�[0m Run api command get scripts team (0.00s)
2121:  �[32m✓�[0m Run api command get scripts team no cache (0.00s)
2122:  �[32m✓�[0m Run api command get typo (0.00s)
2123:  �[32m✓�[0m Run api command upload script (0.00s)
2124:  �[32m✓�[0m Run script command (0.64s)
2125:  �[32m✓�[0m Run script command disabled scripts globally (0.00s)
2126:  �[32m✓�[0m Run script command host not found (0.00s)
2127:  �[32m✓�[0m Run script command invalid file type (0.00s)
2128:  �[32m✓�[0m Run script command invalid hashbang (0.00s)
2129:  �[32m✓�[0m Run script command invalid utf 8 (0.00s)
2130:  �[32m✓�[0m Run script command missing one of script-path and script-nqme (0.00s)
2131:  �[32m✓�[0m Run script command output truncated (0.01s)
2132:  �[32m✓�[0m Run script command posix shell hashbang (0.01s)
2133:  �[32m✓�[0m Run script command script empty (0.00s)
2134:  �[32m✓�[0m Run script command script failed (0.01s)
2135:  �[32m✓�[0m Run script command script killed (0.01s)
...

2190:  �[32m✓�[0m Validate git ops group EUA global-only run degrades id p but the team's in-run file disables EU A: accepted (0.00s)
2191:  �[32m✓�[0m Validate git ops group EUA global-only run degrades id p while a stored team keeps EUA on: rejected (#4337 1) (0.00s)
2192:  �[32m✓�[0m Validate git ops group EUA no EUA enabled anywhere is accepted (0.00s)
2193:  �[32m✓�[0m Validate git ops group EUA team enables EU A, global file adds complete id P: accepted (0.00s)
2194:  �[32m✓�[0m Validate git ops group EUA team enables EU A, global file adds id p missing entity id: rejected (0.00s)
2195:  �[32m✓�[0m Validate git ops group EUA team enables EU A, global file omits id P, stored has id P: rejected (overwrite clears) (0.00s)
2196:  �[32m✓�[0m Validate git ops group EUA team enables EU A, stored has id P, no global file: accepted (0.00s)
2197:  �[32m✓�[0m Validate git ops group EUA team enables EU A, stored has no id P, no global file: rejected (0.00s)
2198:  github.com/fleetdm/fleet/v4/cmd/fleetctl/integrationtest/gitops:
2199:  �[32m✓�[0m Git ops VPP (5.35s)
2200:  �[32m✓�[0m Git ops VPP all fleets is supported (0.64s)
2201:  �[32m✓�[0m Git ops VPP all teams is supported (0.60s)
2202:  �[32m✓�[0m Git ops VPP new key all valid (0.67s)
2203:  �[32m✓�[0m Git ops VPP new key multiple elements (0.70s)
2204:  �[32m✓�[0m Git ops VPP no team is supported (0.72s)
2205:  �[32m✓�[0m Git ops VPP non existent location fails (0.70s)
2206:  �[32m✓�[0m Git ops VPP not provided teams defaults to no team (0.66s)
2207:  �[32m✓�[0m Git ops VPP using an undefined team errors (0.65s)
2208:  �[32m✓�[0m Git ops existing team VPP apps with missing team (0.52s)
...

2301:  �[32m✓�[0m Git ops team software installers team software installer with display name.yml (1.74s)
2302:  �[32m✓�[0m Integrations enterprise gitops (316.35s)
2303:  �[32m✓�[0m Integrations enterprise gitops test CA integrations (3.89s)
2304:  �[32m✓�[0m Integrations enterprise gitops test FMA labels include all (6.00s)
2305:  �[32m✓�[0m Integrations enterprise gitops test IPA software installers (10.43s)
2306:  �[32m✓�[0m Integrations enterprise gitops test JSON configuration profile escaping (1.29s)
2307:  �[32m✓�[0m Integrations enterprise gitops test add manual labels (1.54s)
2308:  �[32m✓�[0m Integrations enterprise gitops test configuration profile escaping (1.33s)
2309:  �[32m✓�[0m Integrations enterprise gitops test delete CA with certificate templates (5.90s)
2310:  �[32m✓�[0m Integrations enterprise gitops test delete mac OS setup (5.01s)
2311:  �[32m✓�[0m Integrations enterprise gitops test deleting no team YAML (2.65s)
2312:  �[32m✓�[0m Integrations enterprise gitops test disallow software setup experience (123.70s)
2313:  �[32m✓�[0m Integrations enterprise gitops test disallow software setup experience all VPP with setup experience (1.25s)
2314:  �[32m✓�[0m Integrations enterprise gitops test disallow software setup experience no team VPP (1.14s)
2315:  �[32m✓�[0m Integrations enterprise gitops test disallow software setup experience no team installers (60.50s)
2316:  �[32m✓�[0m Integrations enterprise gitops test disallow software setup experience packages fail (60.63s)
2317:  �[32m✓�[0m Integrations enterprise gitops test dry run mac OS setup script with manual agent install conflict (0.41s)
...

2347:  �[32m✓�[0m Integrations enterprise gitops test omitted top level keys global (2.47s)
2348:  �[32m✓�[0m Integrations enterprise gitops test remove custom settings from default YAML (2.53s)
2349:  �[32m✓�[0m Integrations enterprise gitops test special case teams VPP apps (3.76s)
2350:  �[32m✓�[0m Integrations enterprise gitops test special case teams VPP apps all teams (2.36s)
2351:  �[32m✓�[0m Integrations enterprise gitops test special case teams VPP apps no team (1.22s)
2352:  �[32m✓�[0m Integrations enterprise gitops test unset configuration profile labels (4.87s)
2353:  �[32m✓�[0m Integrations enterprise gitops test unset software installer labels (12.31s)
2354:  �[32m✓�[0m Integrations enterprise starter library (4.92s)
2355:  �[32m✓�[0m Integrations enterprise starter library test apply starter library premium (3.52s)
2356:  �[32m✓�[0m Integrations gitops (2.29s)
2357:  �[32m✓�[0m Integrations gitops test fleet gitops (0.46s)
2358:  �[32m✓�[0m Integrations gitops test fleet gitops DDM fleet vars requires premium (0.12s)
2359:  �[32m✓�[0m Integrations gitops test fleet gitops with fleet secrets (0.22s)
2360:  �[32m✓�[0m Integrations starter library (1.55s)
2361:  �[32m✓�[0m Integrations starter library test apply starter library free (0.18s)
2362:  === �[31mFailed�[0m
2363:  === �[31mFAIL�[0m: cmd/fleetctl/fleetctl TestGitOpsFullGlobal/useDeprecatedKeys=false (0.04s)
2364:  time=level=INFO msg="request error" path=/api/latest/fleet/setup_experience/eula/metadata took=123.023µs uuid=73376fb7-99ad-4e2a-b2b4-0fb8d03cca80 err="not found"
2365:  [-] would've deleted report Query to delete
2366:  time=level=INFO msg="request error" path=/api/latest/fleet/setup_experience/eula/metadata took=118.436µs uuid=4f638e17-51ff-45d8-95fe-8bd396c55713 err="not found"
2367:  testing_utils_test.go:20: 
2368:  Error Trace:	/home/runner/work/fleet/fleet/cmd/fleetctl/fleetctl/testing_utils_test.go:20
2369:  /home/runner/work/fleet/fleet/cmd/fleetctl/fleetctl/gitops_test.go:2244
2370:  Error:      	Received unexpected error:
2371:  applying custom settings: POST /api/latest/fleet/mdm/profiles/batch received status 422 Validation Failed: cannot set custom settings: Windows MDM isn't turned on. For more information about setting up MDM, please visit https://fleetdm.com/learn-more-about/windows-mdm (API time: 1ms)
2372:  Test:       	TestGitOpsFullGlobal/useDeprecatedKeys=false
2373:  --- FAIL: TestGitOpsFullGlobal/useDeprecatedKeys=false (0.04s)
2374:  === �[31mFAIL�[0m: cmd/fleetctl/fleetctl TestGitOpsFullGlobal/useDeprecatedKeys=true (0.04s)
2375:  time=level=INFO msg="request error" path=/api/latest/fleet/setup_experience/eula/metadata took=118.295µs uuid=57936a82-27f8-4194-b033-6ea7b0fd41ad err="not found"
2376:  [-] would've deleted report Query to delete
2377:  time=level=INFO msg="request error" path=/api/latest/fleet/setup_experience/eula/metadata took=132.497µs uuid=48a40b00-54e3-488c-b164-6cd61acc1849 err="not found"
2378:  testing_utils_test.go:20: 
2379:  Error Trace:	/home/runner/work/fleet/fleet/cmd/fleetctl/fleetctl/testing_utils_test.go:20
2380:  /home/runner/work/fleet/fleet/cmd/fleetctl/fleetctl/gitops_test.go:2244
2381:  Error:      	Received unexpected error:
2382:  applying custom settings: POST /api/latest/fleet/mdm/profiles/batch received status 422 Validation Failed: cannot set custom settings: Windows MDM isn't turned on. For more information about setting up MDM, please visit https://fleetdm.com/learn-more-about/windows-mdm (API time: 1ms)
2383:  Test:       	TestGitOpsFullGlobal/useDeprecatedKeys=true
2384:  --- FAIL: TestGitOpsFullGlobal/useDeprecatedKeys=true (0.04s)
2385:  === �[31mFAIL�[0m: cmd/fleetctl/fleetctl TestGitOpsFullGlobal (0.54s)
2386:  DONE 921 tests, 3 failures in 649.137s
2387:  make[1]: *** [Makefile:302: .run-go-tests] Error 1
2388:  make[1]: Leaving directory '/home/runner/work/fleet/fleet'
2389:  make: *** [Makefile:417: test-go] Error 2
2390:  ##[error]Process completed with exit code 2.
2391:  Node 20 is being deprecated. This workflow is running with Node 24 by default. If you need to temporarily use Node 20, you can set the ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true environment variable. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
2392:  ##[group]Run actions/upload-artifact@834a144ee995460fba8ed112a2fc961b36a5ec5a
2393:  with:
2394:  name: fleetctl-mysql8.0.44-coverage
2395:  path: ./coverage.txt
2396:  if-no-files-found: error
2397:  compression-level: 6
...

2400:  RACE_ENABLED: false
2401:  GO_TEST_TIMEOUT: 20m
2402:  DOCKER_COMMAND: docker compose -f docker-compose.yml -f docker-compose-redis-cluster.yml up -d mysql_test mysql_replica_test redis redis-cluster-1 redis-cluster-2 redis-cluster-3 redis-cluster-4 redis-cluster-5 redis-cluster-6 redis-cluster-setup s3 saml_idp mailhog mailpit smtp4dev_test
2403:  RUN_TESTS_ARG: 
2404:  CI_TEST_PKG: fleetctl
2405:  NEED_DOCKER: 1
2406:  ARTIFACT_PREFIX: fleetctl-mysql8.0.44
2407:  GOTOOLCHAIN: local
2408:  ##[endgroup]
2409:  (node:44528) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
2410:  (Use `node --trace-deprecation ...` to show where the warning was created)
2411:  With the provided path, there will be 1 file uploaded
2412:  Artifact name is valid!
2413:  Root directory input is valid!
2414:  Beginning upload of artifact content to blob storage
2415:  (node:44528) [DEP0169] DeprecationWarning: `url.parse()` behavior is not standardized and prone to errors that have security implications. Use the WHATWG URL API instead. CVEs are not issued for `url.parse()` vulnerabilities.
2416:  Uploaded bytes 2309572
2417:  Finished uploading artifact content to blob storage!
2418:  SHA256 hash of uploaded artifact zip is 4b136f9603f90c75fd2fa88d0b5057cc5fcc139302098649e64a83e4aa8ea962
2419:  Finalizing artifact upload
2420:  Artifact fleetctl-mysql8.0.44-coverage.zip successfully finalized. Artifact ID 8017808100
2421:  Artifact fleetctl-mysql8.0.44-coverage has been successfully uploaded! Final size is 2309572 bytes. Artifact ID is 8017808100
2422:  Artifact download URL: https://github.com/fleetdm/fleet/actions/runs/28534951474/artifacts/8017808100
2423:  ##[group]Run c1grep() { grep "$@" || test $? = 1; }
2424:  �[36;1mc1grep() { grep "$@" || test $? = 1; }�[0m
2425:  �[36;1mc1grep -oP 'FAIL: .*$' /tmp/gotest.log > /tmp/summary.txt�[0m
2426:  �[36;1mc1grep 'test timed out after' /tmp/gotest.log >> /tmp/summary.txt�[0m
2427:  �[36;1mc1grep 'fatal error:' /tmp/gotest.log >> /tmp/summary.txt�[0m
2428:  �[36;1mc1grep -A 10 'panic: runtime error: ' /tmp/gotest.log >> /tmp/summary.txt�[0m
2429:  �[36;1mc1grep ' FAIL\t' /tmp/gotest.log >> /tmp/summary.txt�[0m
2430:  �[36;1mGO_FAIL_SUMMARY=$(head -n 5 /tmp/summary.txt | sed ':a;N;$!ba;s/\n/\\n/g')�[0m
2431:  �[36;1mecho "GO_FAIL_SUMMARY=$GO_FAIL_SUMMARY"�[0m
2432:  �[36;1mif [[ -z "$GO_FAIL_SUMMARY" ]]; then�[0m
2433:  �[36;1m  GO_FAIL_SUMMARY="unknown, please check the build URL"�[0m
2434:  �[36;1mfi�[0m
2435:  �[36;1mGO_FAIL_SUMMARY=$GO_FAIL_SUMMARY envsubst < .github/workflows/config/slack_payload_template.json > ./payload.json�[0m
2436:  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2437:  env:
2438:  RACE_ENABLED: false
2439:  GO_TEST_TIMEOUT: 20m
2440:  DOCKER_COMMAND: docker compose -f docker-compose.yml -f docker-compose-redis-cluster.yml up -d mysql_test mysql_replica_test redis redis-cluster-1 redis-cluster-2 redis-cluster-3 redis-cluster-4 redis-cluster-5 redis-cluster-6 redis-cluster-setup s3 saml_idp mailhog mailpit smtp4dev_test
2441:  RUN_TESTS_ARG: 
2442:  CI_TEST_PKG: fleetctl
2443:  NEED_DOCKER: 1
2444:  ARTIFACT_PREFIX: fleetctl-mysql8.0.44
2445:  GOTOOLCHAIN: local
2446:  ##[endgroup]
2447:  GO_FAIL_SUMMARY=FAIL: TestGitOpsFullGlobal/useDeprecatedKeys=false (0.04s)\nFAIL: TestGitOpsFullGlobal/useDeprecatedKeys=true (0.04s)
2448:  Node 20 is being deprecated. This workflow is running with Node 24 by default. If you need to temporarily use Node 20, you can set the ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true environment variable. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
2449:  ##[group]Run actions/upload-artifact@834a144ee995460fba8ed112a2fc961b36a5ec5a
2450:  with:
2451:  name: fleetctl-mysql8.0.44-test-log
2452:  path: /tmp/gotest.log
2453:  if-no-files-found: error
2454:  compression-level: 6
...

2457:  RACE_ENABLED: false
2458:  GO_TEST_TIMEOUT: 20m
2459:  DOCKER_COMMAND: docker compose -f docker-compose.yml -f docker-compose-redis-cluster.yml up -d mysql_test mysql_replica_test redis redis-cluster-1 redis-cluster-2 redis-cluster-3 redis-cluster-4 redis-cluster-5 redis-cluster-6 redis-cluster-setup s3 saml_idp mailhog mailpit smtp4dev_test
2460:  RUN_TESTS_ARG: 
2461:  CI_TEST_PKG: fleetctl
2462:  NEED_DOCKER: 1
2463:  ARTIFACT_PREFIX: fleetctl-mysql8.0.44
2464:  GOTOOLCHAIN: local
2465:  ##[endgroup]
2466:  (node:44550) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
2467:  (Use `node --trace-deprecation ...` to show where the warning was created)
2468:  With the provided path, there will be 1 file uploaded
2469:  Artifact name is valid!
2470:  Root directory input is valid!
2471:  Beginning upload of artifact content to blob storage
2472:  (node:44550) [DEP0169] DeprecationWarning: `url.parse()` behavior is not standardized and prone to errors that have security implications. Use the WHATWG URL API instead. CVEs are not issued for `url.parse()` vulnerabilities.
2473:  Uploaded bytes 10965
...

2489:  RACE_ENABLED: false
2490:  GO_TEST_TIMEOUT: 20m
2491:  DOCKER_COMMAND: docker compose -f docker-compose.yml -f docker-compose-redis-cluster.yml up -d mysql_test mysql_replica_test redis redis-cluster-1 redis-cluster-2 redis-cluster-3 redis-cluster-4 redis-cluster-5 redis-cluster-6 redis-cluster-setup s3 saml_idp mailhog mailpit smtp4dev_test
2492:  RUN_TESTS_ARG: 
2493:  CI_TEST_PKG: fleetctl
2494:  NEED_DOCKER: 1
2495:  ARTIFACT_PREFIX: fleetctl-mysql8.0.44
2496:  GOTOOLCHAIN: local
2497:  ##[endgroup]
2498:  (node:44562) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
2499:  (Use `node --trace-deprecation ...` to show where the warning was created)
2500:  With the provided path, there will be 1 file uploaded
2501:  Artifact name is valid!
2502:  Root directory input is valid!
2503:  Beginning upload of artifact content to blob storage
2504:  (node:44562) [DEP0169] DeprecationWarning: `url.parse()` behavior is not standardized and prone to errors that have security implications. Use the WHATWG URL API instead. CVEs are not issued for `url.parse()` vulnerabilities.
2505:  Uploaded bytes 205
...

2521:  RACE_ENABLED: false
2522:  GO_TEST_TIMEOUT: 20m
2523:  DOCKER_COMMAND: docker compose -f docker-compose.yml -f docker-compose-redis-cluster.yml up -d mysql_test mysql_replica_test redis redis-cluster-1 redis-cluster-2 redis-cluster-3 redis-cluster-4 redis-cluster-5 redis-cluster-6 redis-cluster-setup s3 saml_idp mailhog mailpit smtp4dev_test
2524:  RUN_TESTS_ARG: 
2525:  CI_TEST_PKG: fleetctl
2526:  NEED_DOCKER: 1
2527:  ARTIFACT_PREFIX: fleetctl-mysql8.0.44
2528:  GOTOOLCHAIN: local
2529:  ##[endgroup]
2530:  (node:44574) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
2531:  (Use `node --trace-deprecation ...` to show where the warning was created)
2532:  With the provided path, there will be 1 file uploaded
2533:  Artifact name is valid!
2534:  Root directory input is valid!
2535:  Beginning upload of artifact content to blob storage
2536:  (node:44574) [DEP0169] DeprecationWarning: `url.parse()` behavior is not standardized and prone to errors that have security implications. Use the WHATWG URL API instead. CVEs are not issued for `url.parse()` vulnerabilities.
2537:  Uploaded bytes 104931
...

2570:  RACE_ENABLED: false
2571:  GO_TEST_TIMEOUT: 20m
2572:  DOCKER_COMMAND: docker compose -f docker-compose.yml -f docker-compose-redis-cluster.yml up -d mysql_test mysql_replica_test redis redis-cluster-1 redis-cluster-2 redis-cluster-3 redis-cluster-4 redis-cluster-5 redis-cluster-6 redis-cluster-setup s3 saml_idp mailhog mailpit smtp4dev_test
2573:  RUN_TESTS_ARG: 
2574:  CI_TEST_PKG: fleetctl
2575:  NEED_DOCKER: 1
2576:  ARTIFACT_PREFIX: fleetctl-mysql8.0.44
2577:  GOTOOLCHAIN: local
2578:  ##[endgroup]
2579:  (node:44609) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
2580:  (Use `node --trace-deprecation ...` to show where the warning was created)
2581:  With the provided path, there will be 1 file uploaded
2582:  Artifact name is valid!
2583:  Root directory input is valid!
2584:  Beginning upload of artifact content to blob storage
2585:  (node:44609) [DEP0169] DeprecationWarning: `url.parse()` behavior is not standardized and prone to errors that have security implications. Use the WHATWG URL API instead. CVEs are not issued for `url.parse()` vulnerabilities.
2586:  Uploaded bytes 133

last_opened_time AS last_opened_at,
path AS installed_path
FROM apps
WHERE path NOT LIKE '%%.app/Contents/%%'

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Neat little change

@dantecatalfamo

Copy link
Copy Markdown
Member

@mostlikelee Does this include dropping embedded apps that have different AppIDs?

@mostlikelee

Copy link
Copy Markdown
Contributor Author

@mostlikelee Does this include dropping embedded apps that have different AppIDs?

Indeed, that's how we spec'd with product

@dantecatalfamo

Copy link
Copy Markdown
Member

@mostlikelee Re-dump the schema and then it's good to go 👍

…itles-44199

# Conflicts:
#	server/datastore/mysql/schema.sql
@mostlikelee

Copy link
Copy Markdown
Contributor Author

@dantecatalfamo should be g2g

@mostlikelee

Copy link
Copy Markdown
Contributor Author

@sharon-fdm this needs an approval for host vitals docs

@sharon-fdm sharon-fdm left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I only reviewed docs/Contributing/product-groups/orchestration/understanding-host-vitals.md

Relying on Dante's approval for the rest.

@mostlikelee
mostlikelee merged commit 56a3c75 into main Jul 9, 2026
46 checks passed
@mostlikelee
mostlikelee deleted the fix-embedded-bundle-titles-44199 branch July 9, 2026 16:01
allenhouchins added a commit that referenced this pull request Aug 10, 2026
**Related issue:** Resolves #50875

Docker Desktop on macOS never reported an installed version or
"Installed" status, and offered "Install" on hosts that already had it.

## Root cause

The FMA's `unique_identifier` was `com.electron.dockerdesktop`, which
belongs to the embedded Electron bundle. The installed app reports a
different identifier:

| Path | CFBundleIdentifier |
|---|---|
| `/Applications/Docker.app` | `com.docker.docker` |
| `/Applications/Docker.app/Contents/MacOS/Docker Desktop.app` |
`com.electron.dockerdesktop` |

The identifier was changed from `com.docker.docker` →
`com.electron.dockerdesktop` in #37670 (Jan 5) as "the new bundle
identifier … reflecting the current packaging". The top-level bundle
never changed.

That was latent until #47831 (#44199, Jul 9) added an embedded-bundle
filter to the macOS software inventory query in
`server/service/osquery_utils/queries.go`:

```sql
FROM apps
WHERE path NOT LIKE '%.app/Contents/%'
```

`/Applications/Docker.app/Contents/MacOS/Docker Desktop.app` matches
that pattern, so the only row carrying the embedded identifier is
filtered out and inventory keeps just `/Applications/Docker.app` →
`com.docker.docker`. Since FMA↔inventory matching is by bundle
identifier (`addSoftwareTitleToMatchingSoftware`), the title the FMA
owns had zero installed versions.

**Patch policies kept passing**, because they run the FMA's
`exists`/`patched` SQL directly against the host's *unfiltered* `apps`
table. That's also why #50041 needed a `.back` path exclusion, and why
patch status and the software UI have disagreed since July.

## What changed

**Catalog** — `unique_identifier` is now `com.docker.docker`, with
`docker-desktop/darwin.json` regenerated. `outputs/apps.json` needed a
hand-edit because `updateAppsListFile` (`cmd/maintained-apps/main.go`)
only appends new apps and never updates an existing entry's identifier —
filed separately.

The remaining `com.electron.dockerdesktop` references are intentional
and untouched: the install script's quit/relaunch targets (the Electron
bundle is what responds to AppleScript) and the cask's zap paths. The
`.back` exclusion in the patched query also stays —
`/Applications/Docker.app.back` is a *top-level* bundle reporting
`com.docker.docker` at a path the nested-bundle filter does not match,
so a stale `.back` would otherwise show a false "Update available".

**Migration** (`20260810152924_FixDockerDesktopBundleIdentifier`) —
`fleet_maintained_apps` self-heals on catalog sync
(`UpsertMaintainedApp` updates `unique_identifier` on duplicate slug)
and `ReconcileMaintainedAppSoftwareNames` renames the existing "Docker"
title, but an already-added installer's `software_installers.title_id`
binding does not:

- **No `com.docker.docker` title yet** → relabel the stale title in
place, so everything already pointing at it stays correct.
- **Title already exists** (the normal case — any host with Docker
creates it) → merge the stale title into it: installer, install history,
queued installs, patch policy, `software.title_id`, and per-team
settings (icons, display names, pins, update schedules), then drop the
stale title.

## Notes for reviewers

- **Teams that already have an installer on the target title are
skipped** rather than ending up with two installers on one title.
`dedup_token` is the *version* for FMAs and the *storage_id* otherwise,
so `idx_software_installers_dedup` would not have caught that collision.
Those teams keep the pre-migration state instead of having data silently
reshaped.
- **The stale title is only deleted once nothing depends on it.**
`fk_software_installers_title` is `ON DELETE SET NULL`, so deleting it
while an installer still pointed at it would orphan that installer. Note
also that `fk_patch_software_title_id` is `ON DELETE CASCADE` —
re-pointing the patch policy is what keeps it from being deleted
outright.
- **Dangling-reference check:** the only title-referencing columns
without an FK to `software_titles` are `software.title_id` (re-pointed),
`software_titles_host_counts` (deleted; the cron recomputes), and
`kernel_host_counts` / `in_house_app_install_tokens`, neither of which
can hold a macOS app title.
- **Naming lags briefly.** The existing title is named "Docker" (from
osquery); `ReconcileMaintainedAppSoftwareNames` renames it to "Docker
Desktop" on the next catalog sync, so there is a window after upgrade
where the name is still "Docker". I left that to the sync rather than
duplicating the rename logic in the migration.
- **This fixes one app, not the class.** Any other macOS FMA keyed on a
nested bundle fails the same silent way, and a green patch policy will
not reveal it. An audit is filed separately.

# Checklist for submitter

- [x] Changes file added for user-visible changes in `changes/`.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements).
Table/column names in the migration's generated SQL come from hardcoded
struct literals, never from data; all values are placeholders.

## Testing

- [x] Added/updated automated tests

Five migration tests cover: relabel-in-place,
merge-into-inventory-title, duplicate per-team settings dropped, teams
with an existing installer skipped (and the stale title consequently
retained), and no-op when the FMA was never added. The merge test also
asserts `updated_at` is not restamped. Updated the homebrew ingester
test expectations for the new identifier.

Verified: full `server/datastore/mysql/migrations/tables` suite passes
(201s), `ee/maintained-apps/...`, `cmd/maintained-apps/...`, and the FMA
datastore tests pass, `make lint-go-incremental` clean.

Root cause was confirmed against real bundles rather than inferred —
`PlistBuddy` on both Docker bundles for the identifiers above, and
`lsregister -dump` to confirm LaunchServices registers the nested
bundles (which is why the raw `apps` table sees them and patch policies
pass).

- [ ] QA'd all new/changed functionality manually

Needs QA on a real instance: add the Docker Desktop FMA, confirm the
installed version and "Installed" status appear on a host that already
has it, and confirm an upgrade over an instance that already had the FMA
added re-points the existing installer.

For unreleased bug fixes in a release candidate, one of:

- [x] Confirmed that the fix is not expected to adversely impact load
test results

The migration touches only rows tied to a single software title, and is
a no-op on instances that never added the Docker Desktop FMA.

## Database migrations

- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.

Five of the written tables have `updated_at` as `ON UPDATE
CURRENT_TIMESTAMP`: `software_installers`, `host_software_installs`,
`software_install_upcoming_activities`, `policies`, and
`software_title_team_pins`. Since this re-points a foreign key rather
than modifying the records, each statement assigns `updated_at =
updated_at` so MySQL leaves them alone, with a test asserting it.
(Bumping them would have been cosmetic — none of these columns drives
ordering, scheduling, or invalidation; policy membership uses
`policy_membership.updated_at` and `hosts.policy_updated_at`, and the
activity feed orders by the activities table — but preserving them is
more faithful to what the migration actually does.)

- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`). No columns added or altered;
this is a data-only migration.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved macOS Docker Desktop detection so installed versions and
“Installed” status are reported accurately.
* Ensured existing Docker Desktop installations and upgrade history
remain correctly associated after detection updates.
* Improved handling of stale application bundles during patch
evaluation.
* **Maintenance**
* Updated detection data and migration coverage to support the corrected
Docker Desktop identification.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Amphetamine app display name is obtained from LoginHelper instead of correct app name

4 participants