New log destination: Splunk

[noahtalerman]

• @allenhouchins: Consider just making a generic HTTP event collector endpoint so we aren't making one thing specific for Splunk, another thing specific to SumoLogic, another one specific to XYZ, etc.

Implementation: Merged in #48455 (resolves #25574).

Goal

User story
As a Fleet user,
I want to configure Splunk as my log destination for reports and other osquery logs
so that I can send data directly to Splunk w/o having to setup Firehose as a "middle man".

Changes

Product

• UI changes: Minor -- added "Splunk" to LogDestinationIndicator display name, tooltip, and TypeScript LogDestination union type. Same pattern as NATS (Add support for publishing logs to NATS. #36527). No new pages, components, or Figma needed. Screenshots in PR Add native Splunk HEC log destination #48455.
• CLI (fleetctl) usage changes: No changes. Splunk is configured via server flags/env vars, not fleetctl.
• YAML changes: No changes. Server-side config only, not GitOps YAML.
• REST API changes: Added splunk_settings to app config API response (GET /api/v1/fleet/config). Token excluded from response for security. Same pattern as other log destinations.
• Fleet's agent (fleetd) changes: No changes. This is server-side log forwarding.
• Fleet server configuration changes: 6 new flags: splunk.url, splunk.token, splunk.index, splunk.source, splunk.source_type, splunk.insecure_skip_verify. Documented in docs/Configuration/fleet-server-configuration.md.
• Exposed, public API endpoint changes: No changes.
• fleetdm.com changes: No changes.
• GitOps mode UI changes: No changes. Log destination indicator is read-only in all modes.
• GitOps generation changes: No changes.
• Activity changes: No changes.
• Permissions changes: No changes. Log destination config is admin-only (same as existing destinations). No changes to the permissions doc page.
• Changes to paid features or tiers: Fleet Free. Available in all deployments (same as other log destinations).
• My device and fleetdm.com/better changes: No changes.
• Usage statistics: No changes.
• Other reference documentation changes: Updated docs/Get started/FAQ.md (added splunk to plugin list). Updated articles/log-destinations.md (rewrote Splunk section with native HEC as primary, Firehose as alternative).
• First draft of test plan added
• Once shipped, requester has been notified
• Once shipped, dogfooding issue has been filed

Engineering

• Test plan is finalized
• Contributor API changes: No changes. Uses existing app config API; adds "splunk" as an allowed plugin value.
• Feature guide changes: Updated articles/log-destinations.md with native HEC documentation.
• Database schema migrations: None. Splunk config is server config (flags/env vars), not stored in DB.
• Load testing: Not required. Same batching pattern as Firehose (1MB max batch, retry on 503).
• Pre-QA load test: Not required. Load profile identical to existing log destinations.
• Load testing/osquery-perf improvements: Not required.

ℹ️ Please read this issue carefully and understand it. Pay special attention to UI wireframes, especially "dev notes".

Risk assessment

• Risk level: Low
• Follows identical patterns to existing log destinations (Firehose, NATS). No DB changes, no new API surface beyond an allowed config value. No performance implications -- same batching and retry logic as other destinations.

Test plan

Make sure to go through the list and consider all events that might be related to this story, so we catch edge cases earlier.

Core flow

1. Configure Fleet server with --osquery_status_log_plugin=splunk --osquery_result_log_plugin=splunk and the Splunk flags (splunk.url, splunk.token, etc.) pointing at a Splunk HEC endpoint
2. Verify Fleet starts successfully (health check against HEC /services/collector/health passes)
3. Enroll a host and confirm status/result logs appear in the configured Splunk index with correct source, sourcetype, and timestamps
4. In the Fleet UI, navigate to Manage automations modal and verify "Log destination: Splunk" is displayed
5. Navigate to a report details page and verify "Log destination: Splunk" appears
6. Hover the log destination indicator and verify the tooltip reads "Each time a report runs, the data is sent to Splunk."
7. Open the Edit report form and verify "sent to your log destination: Splunk" appears
8. Open the Save new report modal and verify "sent to your log destination: Splunk" appears

Edge cases

• Invalid HEC token: Fleet starts (health check passes regardless of token) but first Write() returns HTTP 403
• Unreachable Splunk URL: Fleet fails to start with a health check error
• Self-signed TLS cert without insecure_skip_verify: Fleet fails to start with a TLS error
• Self-signed TLS cert with insecure_skip_verify=true: Fleet starts and sends logs successfully
• Missing URL or token: Fleet fails to start with a descriptive validation error ("URL must not be empty" / "HEC token must not be empty")
• Single event > 1MB: Dropped with a log warning; other events in the same batch still sent
• Batch total > 1MB: Automatically split into multiple HTTP requests
• Splunk temporarily unavailable (HTTP 503 or 429): Retried with exponential backoff up to 8 times
• Splunk unavailable beyond retries: Returns error after 9 total attempts (1 initial + 8 retries)
• Switching from another log destination to Splunk: Verify previous destination stops receiving and Splunk starts receiving

UI

• Verify that all UI changes are correctly implemented (display name, tooltip, TypeScript type)
• Verify expected UI states (log destination indicator shows "Splunk" when configured)

API

• Verify GET /api/v1/fleet/config returns splunk_settings when Splunk is configured
• Verify splunk_settings does NOT include the HEC token (security)
• Verify error handling for invalid inputs (missing URL/token)

Upgrade / data migration

• No DB migration, so upgrade from prior version works seamlessly
• Existing non-Splunk log destinations remain unaffected

Supplemental testing

Testing notes

Splunk Docker setup for local testing:

docker run -d --name splunk-test --platform linux/amd64 \
  -p 8000:8000 -p 8088:8088 -p 8089:8089 \
  -e SPLUNK_GENERAL_TERMS=--accept-sgt-current-at-splunk-com \
  -e SPLUNK_START_ARGS=--accept-license \
  -e SPLUNK_PASSWORD=changeme123 \
  -e SPLUNK_HEC_TOKEN=test-hec-token-1234 \
  splunk/splunk:latest

Automated tests:

• 9 unit tests in server/logging/splunk_test.go (run with go test ./server/logging/ -run TestSplunk)
• 3 integration tests in server/logging/splunk_integration_test.go (run with SPLUNK_INTEGRATION_TEST=1 go test ./server/logging/ -run TestSplunkIntegration -v -- requires running Splunk Docker)

E2E verified: 83 real osquery status log events from a macOS host were indexed in Splunk with correct metadata. Screenshot in PR #48455.

Confirmation

1. Engineer: Added comment to user story confirming successful completion of test plan (include any special setup, test data, or configuration used during development/testing if applicable).
2. QA: Added comment to user story confirming successful completion of test plan.
3. QA: Determined whether this story needs Playwright automation.
   • Needs automation: Yes / No
   • If yes, filed a follow-up issue in the :help-qa project with status "Needs automation":

[sharon-fdm]

Duplication of #25574. Closing.

[fleet-release]

Splunk meets Fleet's wings,
Data flows like cloud city's streams,
Security sings.

[noahtalerman]

Hey @sharon-fdm thanks for picking this up!

This issue isn't a duplicate. It's the user story for the following original request:

• Send osquery data to Splunk #25574

I re-opened this story, assigned you, and unassigned you from the original request.

Can you please fill out the TODOs in the "Product" and "Engineering" section and write up a test plan? That way, we know what's changing and how to test it, we make sure to update reference docs, and more.

Also, can you please bring this to the next #g-orchestration design review? All changes to the Fleet product go through design review.

FYI @rachaelshaw