Add native Splunk HEC log destination - #48455
Merged
Merged
Conversation
Add a new "splunk" log plugin that sends osquery status, result, and audit logs directly to Splunk via the HTTP Event Collector (HEC) API, removing the need for intermediate services like AWS Firehose. Closes #25574 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #48455 +/- ##
==========================================
+ Coverage 67.90% 68.00% +0.09%
==========================================
Files 3678 3679 +1
Lines 233673 233808 +135
Branches 12415 12414 -1
==========================================
+ Hits 158686 159001 +315
+ Misses 60724 60504 -220
- Partials 14263 14303 +40
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…lunk - Added splunk.insecure_skip_verify config for self-signed certs - Added integration tests that run against a real Splunk Docker container (gated behind SPLUNK_INTEGRATION_TEST=1 env var) - Tests verify: event delivery, batch sending (100 events), bad token rejection - All events were confirmed searchable in the Splunk index Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The app config endpoint validates log plugin names. Without this, the Fleet API returned a 500 "unrecognized logging plugin: splunk" when any API client fetched the config. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Use assert instead of require inside http.HandlerFunc (testifylint)
- Replace interface{} with any (modernize)
- Use fleethttp.NewClient instead of http.Client{} (gocritic ruleguard)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Drop events exceeding 1MB with a warning log (matches Firehose behavior) - Retry on HTTP 503 with exponential backoff, up to 8 retries - Add 4 new tests: RecordTooBig, SplitBatchBySize, RetryOnServiceUnavailable, RetryExhausted - Add Splunk to LogDestination type and LogDestinationIndicator component - Add Splunk config docs in fleet-server-configuration.md - Add splunk to plugin lists in FAQ and config docs Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The Splunk section previously only documented the Firehose workaround. Now documents the native HEC integration as the primary method and keeps the Firehose route as an alternative under a subsection. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.
Tip: disable this comment in your organization's Code Review settings.
Contributor
There was a problem hiding this comment.
Warning
- Copilot's review of this pull request may be incomplete because some of the changed files are excluded by your Copilot content exclusion settings. See Excluding content from Copilot for details.
Pull request overview
This PR adds a new native Splunk HTTP Event Collector (HEC) log destination so Fleet can send osquery logs directly to Splunk without intermediary services, and wires the new destination through server config, API response types, and the UI log-destination indicator.
Changes:
- Implement a new Splunk HEC JSON log writer with startup health check, batching, and retry logic (
server/logging/splunk.go) plus unit/integration tests. - Wire Splunk config through Fleet server configuration + logging factory and expose a token-less subset via the appconfig API.
- Add
"splunk"to the frontend log destination type union and display/tooltip mapping.
Reviewed changes
Copilot reviewed 10 out of 14 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| server/logging/splunk.go | New Splunk HEC writer (health check, batching, retries). |
| server/logging/splunk_test.go | Unit tests for Splunk writer behavior. |
| server/logging/splunk_integration_test.go | Optional integration tests against a real Splunk instance. |
| server/logging/logging.go | Add Splunk config and splunk plugin case in JSON logger factory. |
| server/config/config.go | Add Splunk config struct + config flags/env mapping. |
| cmd/fleet/logging.go | Wire loaded Splunk config into the logging config builder. |
| server/fleet/app.go | Add API-facing SplunkConfig that excludes the token. |
| server/service/service_appconfig.go | Include splunk plugin config in appconfig response (token excluded). |
| frontend/interfaces/config.ts | Add "splunk" to LogDestination union. |
| frontend/components/LogDestinationIndicator/LogDestinationIndicator.tsx | Add Splunk display name and tooltip text. |
| docs/Configuration/fleet-server-configuration.md | Documentation update (contents excluded by policy). |
| docs/Get started/FAQ.md | Documentation update (contents excluded by policy). |
| articles/log-destinations.md | Documentation update (contents excluded by policy). |
| changes/25574-splunk-log-destination | Change entry (contents excluded by policy). |
Files excluded by content exclusion policy (4)
- articles/log-destinations.md
- changes/25574-splunk-log-destination
- docs/Configuration/fleet-server-configuration.md
- docs/Get started/FAQ.md
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+158
to
+160
| if resp.StatusCode == http.StatusServiceUnavailable && try < splunkMaxRetries { | ||
| return w.sendWithRetry(ctx, payload, try+1) | ||
| } |
Collaborator
Author
There was a problem hiding this comment.
Fixed in 3ee256a -- body is now drained and closed before retrying.
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (4)
💤 Files with no reviewable changes (1)
🚧 Files skipped from review as they are similar to previous changes (2)
WalkthroughThis PR adds a native Splunk HTTP Event Collector log destination for osquery status, result, and audit logs. It adds Splunk configuration types and config-key wiring in server config, routes the values into logging setup, implements a Splunk log writer with batching, retries, and health checks, and updates the logging plugin switch and service mapping to use it. The frontend accepts and displays the new Possibly related issues
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (2)
server/config/config.go (1)
687-696: 🎯 Functional Correctness | 🔵 Trivial | ⚖️ Poor tradeoffSplunk config has no per-log-type routing, unlike every other multi-destination plugin.
SplunkConfigexposes a single sharedIndex/Source/SourceType, whereas Firehose, Kinesis, Lambda, PubSub, KafkaREST, and Nats all define distinctStatus*/Result*/Audit*fields thatcmd/fleet/logging.gooverrides per log type. As written, status, result, and audit logs sent to Splunk will all land with identicalsource/sourcetype/index, with no way to distinguish them at the destination — a capability every other supported backend provides.If this is intentional (e.g., relying on event content alone to differentiate log types), consider noting it in the docs; otherwise, add per-log-type source/sourcetype fields to match the established pattern.
Also applies to: 809-809, 1697-1704, 2078-2085
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@server/config/config.go` around lines 687 - 696, SplunkConfig currently only exposes shared Index/Source/SourceType fields, so Splunk logs cannot be routed differently by log type like the other multi-destination plugins. Update SplunkConfig and the Splunk setup path in cmd/fleet/logging.go to add and use distinct Status*/Result*/Audit* source/sourcetype/index fields, matching the pattern used by Firehose, Kinesis, Lambda, PubSub, KafkaREST, and Nats. Make sure the per-log-type overrides are wired through the existing logging configuration flow so status, result, and audit events can be distinguished at the destination.server/logging/splunk_integration_test.go (1)
58-63: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winPoll for indexed events instead of sleeping a fixed 5s.
Splunk indexing latency is variable, so these sleeps will make the env-gated integration tests flaky on slower runners. Poll
searchSplunkuntil the expected count arrives or a deadline expires.Suggested change
- time.Sleep(5 * time.Second) - - events := searchSplunk(t, marker) + events := waitForSplunkEvents(t, marker, 3) require.Len(t, events, 3, "should find all 3 test events in Splunk")func waitForSplunkEvents(t *testing.T, marker string, want int) []string { t.Helper() deadline := time.Now().Add(30 * time.Second) for { events := searchSplunk(t, marker) if len(events) == want || time.Now().After(deadline) { return events } time.Sleep(500 * time.Millisecond) } }Also applies to: 97-100
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@server/logging/splunk_integration_test.go` around lines 58 - 63, Replace the fixed sleep in the Splunk integration test with polling so the test waits for indexed events up to a deadline instead of assuming 5s is enough. Add or reuse a helper like waitForSplunkEvents near searchSplunk that repeatedly calls searchSplunk until the expected count is reached or timeout expires, then use it in the test cases that currently sleep before asserting the event count.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@server/logging/logging.go`:
- Around line 233-246: The Splunk logger setup is failing fast because
initOsqueryLogging treats NewSplunkLogWriter errors as fatal during
construction. Update the splunk case to avoid making the live checkHealth inside
NewSplunkLogWriter a startup blocker, either by deferring the health check until
the first write or by converting that check to a warning-only path so fleet can
boot when Splunk HEC is temporarily unavailable. Keep the change localized to
NewSplunkLogWriter and the splunk branch in initOsqueryLogging.
In `@server/logging/splunk.go`:
- Around line 111-113: The splunk drop log in the logger path currently includes
a raw `event_prefix`, which can leak customer payload data into Fleet logs when
oversized events are rejected. Update the `w.logger.InfoContext` call in the
Splunk logging flow to remove the payload snippet and keep only safe metadata
such as the size and other non-content fields, preserving the existing
oversize-drop behavior without echoing event contents.
- Around line 141-159: The retry backoff in sendWithRetry uses time.Sleep, which
ignores ctx cancellation and makes retry tests wait the full exponential delay.
Replace the direct sleep in sendWithRetry with a context-aware wait using the
existing ctx, and move the delay calculation behind a small helper on the splunk
writer (or package-level function) that tests can override/stub. Keep the retry
flow and the try-based backoff behavior intact while ensuring canceled contexts
return immediately.
---
Nitpick comments:
In `@server/config/config.go`:
- Around line 687-696: SplunkConfig currently only exposes shared
Index/Source/SourceType fields, so Splunk logs cannot be routed differently by
log type like the other multi-destination plugins. Update SplunkConfig and the
Splunk setup path in cmd/fleet/logging.go to add and use distinct
Status*/Result*/Audit* source/sourcetype/index fields, matching the pattern used
by Firehose, Kinesis, Lambda, PubSub, KafkaREST, and Nats. Make sure the
per-log-type overrides are wired through the existing logging configuration flow
so status, result, and audit events can be distinguished at the destination.
In `@server/logging/splunk_integration_test.go`:
- Around line 58-63: Replace the fixed sleep in the Splunk integration test with
polling so the test waits for indexed events up to a deadline instead of
assuming 5s is enough. Add or reuse a helper like waitForSplunkEvents near
searchSplunk that repeatedly calls searchSplunk until the expected count is
reached or timeout expires, then use it in the test cases that currently sleep
before asserting the event count.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 3f33d1b3-f438-4f8a-ae67-26cf08fa0781
⛔ Files ignored due to path filters (3)
articles/log-destinations.mdis excluded by!**/*.mddocs/Configuration/fleet-server-configuration.mdis excluded by!**/*.mddocs/Get started/FAQ.mdis excluded by!**/*.md
📒 Files selected for processing (11)
changes/25574-splunk-log-destinationcmd/fleet/logging.gofrontend/components/LogDestinationIndicator/LogDestinationIndicator.tsxfrontend/interfaces/config.tsserver/config/config.goserver/fleet/app.goserver/logging/logging.goserver/logging/splunk.goserver/logging/splunk_integration_test.goserver/logging/splunk_test.goserver/service/service_appconfig.go
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ackoff - Remove event_prefix from oversized record log (security: avoid leaking payload data) - Replace time.Sleep with timer/select that respects context cancellation - Extract retry delay to stubable var (retry tests: 51s -> 0.01s) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
nulmete
reviewed
Jul 1, 2026
nulmete
left a comment
nulmete
left a comment
Member
There was a problem hiding this comment.
Overall LGTM, please see my comments below
Comment on lines
+20
to
+33
| // TestSplunkIntegration tests the Splunk HEC writer against a real Splunk instance. | ||
| // | ||
| // Prerequisites: | ||
| // | ||
| // docker run -d --name splunk-test --platform linux/amd64 \ | ||
| // -p 8000:8000 -p 8088:8088 -p 8089:8089 \ | ||
| // -e SPLUNK_GENERAL_TERMS=--accept-sgt-current-at-splunk-com \ | ||
| // -e SPLUNK_START_ARGS=--accept-license \ | ||
| // -e SPLUNK_PASSWORD=changeme123 \ | ||
| // -e SPLUNK_HEC_TOKEN=test-hec-token-1234 \ | ||
| // splunk/splunk:latest | ||
| // | ||
| // Run with: SPLUNK_INTEGRATION_TEST=1 go test ./server/logging/ -run TestSplunkIntegration -v | ||
| func TestSplunkIntegration(t *testing.T) { |
Member
There was a problem hiding this comment.
This means we're not running this test in CI, right? If that's the case, then I'd look into mocking the Splunk server so that we get some value out of this set of tests. (Otherwise, we'd have to manually run this locally before merging a PR to detect regressions.)
Collaborator
Author
There was a problem hiding this comment.
The unit tests (TestSplunkWrite, etc.) already use httptest servers and run in CI. These integration tests exist for manual validation against real Splunk. Same pattern as Firehose/Kinesis which also have no CI integration tests. Happy to remove the integration test file if you prefer keeping only the unit tests.
Member
There was a problem hiding this comment.
I think the unit tests should be enough, and since we're not running these integration tests in CI I don't see a reason to keep them. My personal preference is to delete them.
… retry tests - Move URL/token validation from NewSplunkLogWriter to the factory in logging.go - Truncate error response bodies to 512 bytes - Add TestSplunkRetryBodyIntegrity: verifies payload is identical on every retry - Add TestSplunkRetryNoNestedRetries: verifies exactly maxRetries+1 calls (no OOM) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CI Feedback 🧐A test triggered by this PR failed. Here is an AI-generated analysis of the failure:
|
nulmete
approved these changes
Jul 2, 2026
nulmete
left a comment
nulmete
left a comment
Member
There was a problem hiding this comment.
LGTM. I'd remove the integration tests since we won't be running those in CI.
Comment on lines
+20
to
+33
| // TestSplunkIntegration tests the Splunk HEC writer against a real Splunk instance. | ||
| // | ||
| // Prerequisites: | ||
| // | ||
| // docker run -d --name splunk-test --platform linux/amd64 \ | ||
| // -p 8000:8000 -p 8088:8088 -p 8089:8089 \ | ||
| // -e SPLUNK_GENERAL_TERMS=--accept-sgt-current-at-splunk-com \ | ||
| // -e SPLUNK_START_ARGS=--accept-license \ | ||
| // -e SPLUNK_PASSWORD=changeme123 \ | ||
| // -e SPLUNK_HEC_TOKEN=test-hec-token-1234 \ | ||
| // splunk/splunk:latest | ||
| // | ||
| // Run with: SPLUNK_INTEGRATION_TEST=1 go test ./server/logging/ -run TestSplunkIntegration -v | ||
| func TestSplunkIntegration(t *testing.T) { |
Member
There was a problem hiding this comment.
I think the unit tests should be enough, and since we're not running these integration tests in CI I don't see a reason to keep them. My personal preference is to delete them.
mike-j-thomas
removed their request for review
July 3, 2026 12:21
rachaelshaw
approved these changes
Jul 3, 2026
36 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related issue: Resolves #25574
Checklist for submitter
changes/SELECT *is avoided, SQL injection is prevented (using placeholders for values in statements), JS inline code is prevented especially for url redirects, and untrusted data interpolated into shell scripts/commands is validated against shell metacharacters.Testing
Summary
splunklog plugin that sends osquery logs directly to Splunk's HTTP Event Collector (HEC) endpointinsecure_skip_verifyoption for environments with self-signed TLS certsUI changes
Follows the same pattern as the NATS log destination PR (#36527) -- adding "Splunk" to the display name, tooltip, and TypeScript type union. No new components, pages, or styles.
Manage automations modal -- "Log destination: Splunk"
Query details page -- "Log destination: Splunk"
Tooltip on hover
Edit query form -- "sent to your log destination: Splunk"
Save new query modal -- "sent to your log destination: Splunk"
How it works
The Splunk writer (
server/logging/splunk.go) implements thefleet.JSONLoggerinterface. On startup it performs a health check against the HEC/services/collector/healthendpoint. On eachWrite()call, it wraps each log entry in Splunk's HEC event format (addingtime,index,source,sourcetype), batches them up to 1 MB, and POSTs to/services/collector/eventwith theAuthorization: Splunk <token>header. If a batch exceeds 1 MB it flushes and starts a new one. Events over 1 MB are dropped with a log warning. Transient errors (HTTP 503) are retried with exponential backoff (up to 8 retries).Configuration
Or via environment variables:
Files changed
server/logging/splunk.go-- Splunk HEC log writer with batching, retry, and health checkserver/logging/splunk_test.go-- 9 unit testsserver/logging/splunk_integration_test.go-- 3 integration tests against real Splunk (gated by env var)server/logging/logging.go-- AddedSplunkConfigandcase "splunk"to factoryserver/config/config.go-- AddedSplunkConfigstruct and config flagscmd/fleet/logging.go-- Wired Splunk config into logging builderserver/fleet/app.go-- AddedSplunkConfigtype for API responses (excludes token)server/service/service_appconfig.go-- Addedcase "splunk"to logging plugin validationfrontend/interfaces/config.ts-- Added"splunk"to LogDestination typefrontend/components/LogDestinationIndicator/LogDestinationIndicator.tsx-- Added Splunk display name and tooltipdocs/Configuration/fleet-server-configuration.md-- Splunk config documentationdocs/Get started/FAQ.md-- Updated plugin listarticles/log-destinations.md-- Updated Splunk section with native HEC docschanges/25574-splunk-log-destination-- Change fileTest plan
Unit tests (9 tests)
TestSplunkWrite-- sends 3 events, verifies HEC format, auth header, index/source/sourcetypeTestSplunkWriteEmpty-- empty logs don't trigger HTTP requestTestSplunkServerError-- HEC 403 propagates as errorTestSplunkHealthCheckFailure-- constructor fails on bad healthTestSplunkRecordTooBig-- oversized events (>1MB) are dropped, normal events still sentTestSplunkSplitBatchBySize-- logs exceeding 1MB batch limit are split into multiple requestsTestSplunkRetryOnServiceUnavailable-- 503 retried with backoff, succeeds on 3rd attemptTestSplunkRetryExhausted-- after 9 attempts (1 + 8 retries) returns errorTestSplunkMissingConfig-- empty URL/token returns descriptive errorIntegration tests (3 tests, gated by
SPLUNK_INTEGRATION_TEST=1)TestSplunkIntegration-- 3 events sent via writer, queried back from Splunk REST APITestSplunkIntegrationBatch-- 100 events in one Write(), all confirmed indexedTestSplunkIntegrationBadToken-- bad token Write() returns 403End-to-end test (macOS ARM64, real osquery agent)
--osquery_status_log_plugin=splunkhostIdentifier,host_uuid,calendarTime,severity,message,decorations)Splunk showing real osquery events from Fleet
Generated with Claude Code
Summary by CodeRabbit