feat(dogstatsd): add Windows named pipe intake by thieman · Pull Request #1956 · DataDog/saluki

thieman · 2026-06-30T19:12:04Z

Summary

Add a reusable Windows named-pipe listen address/transport in saluki-io.
Wire DogStatsD config to enable dogstatsd_pipe_name and dogstatsd_windows_pipe_security_descriptor.
Use core-Agent-compatible raw newline framing for Windows named-pipe DogStatsD payloads.
Drop oversized named-pipe frames that fill the I/O buffer before a newline arrives, matching the core Agent's oversized-message behavior.
Match core Agent pipe creation behavior by applying dogstatsd_buffer_size as the Windows pipe input buffer and using an output buffer size of 0.
Accept either bare pipe names or full \\.\pipe\... values when constructing named-pipe listen addresses.
Use npipe:// for URL-style display/parsing of Windows named-pipe addresses, matching existing Agent conventions for named-pipe socket paths.
Update DogStatsD config inventory/docs from planned unsupported to supported and cover named-pipe newline/stream behavior.
Add a Windows integration case that executes a PowerShell named-pipe client via target_exec, writes to the named pipe, and asserts ADP decodes the frame.

Test plan

cargo test -p panoramic
cargo run --release --bin panoramic -- list -d test/integration/cases --runtime windows --json | rg 'dogstatsd-named-pipe-windows'
cargo check --workspace
cargo check --workspace --tests
cargo test -p saluki-io
cargo test -p saluki-components dogstatsd
cargo xwin check -p saluki-io --target x86_64-pc-windows-msvc
cargo xwin check -p saluki-components --target x86_64-pc-windows-msvc
make fmt

CI investigation note: earlier Windows integration runs proved the action connected/wrote to the pipe, then exposed two bugs: ADP initially treated named-pipe bytes as length-delimited stream frames, and a mounted helper script used a container path Docker rejected on Windows. The test now runs an inline PowerShell client through target_exec, and ADP uses raw newline framing for named pipes.

pr-commenter · 2026-06-30T19:19:58Z

Binary Size Analysis (Agent Data Plane)

Baseline: 94ecf81 · Comparison: 3d25c15 · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 41.53 MiB (baseline) vs 41.55 MiB (comparison)
Size Change: +21.50 KiB (+0.05%)

✅ Binary size difference within threshold

Changes by Module

Module	File Size	Symbols
`figment`	+56.86 KiB	527
`tokio`	-31.70 KiB	3508
`tonic`	-24.73 KiB	375
`crossbeam_channel`	-22.09 KiB	92
`hashbrown`	-19.83 KiB	877
`&mut serde_json`	+19.09 KiB	91
`otlp_protos::otlp_include::opentelemetry`	+16.78 KiB	186
`piecemeal`	+16.51 KiB	41
`tracing_appender`	+16.24 KiB	3
`datadog_protos::trace_piecemeal_include::datadog`	-15.80 KiB	20
`saluki_components::common::datadog`	-15.42 KiB	561
`http_body_util`	+14.82 KiB	155
`anon.c9cad71f0a7107d02d3a8ab0feeee5cd.412.llvm.1316936464131686473`	+14.07 KiB	1
`anon.6ca6cdaf5f719bf571b27f19b205580b.35.llvm.3703347104893547974`	-14.07 KiB	1
`saluki_components::sources::dogstatsd`	+11.69 KiB	366
`saluki_io::net::client`	-10.97 KiB	38
`anon.4b2437a356b91c4941797a9ac8064d8f.0.llvm.4033812997682673044`	-9.81 KiB	1
`anon.d35bf39ab8c1a6f621834974fe00c369.0.llvm.18152056829563145063`	+9.72 KiB	1
`saluki_common::cache::CacheBuilder<K,V,W,H>`	-8.54 KiB	5
`saluki_components::sources::otlp`	+8.20 KiB	226

Detailed Symbol Changes

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  +0.4% +74.4Ki  +0.8%  +107Ki    [36804 Others]
  [NEW] +56.4Ki  [NEW] +56.2Ki    saluki_components::common::datadog::io::run_endpoint_io_loop::_{{closure}}::h864dcd3f24c6e4ef
  [NEW] +44.4Ki  [NEW] +44.3Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::hc76c79a2b14f65f6
  [NEW] +39.0Ki  [NEW] +38.8Ki    _<saluki_components::forwarders::otlp::OtlpForwarder as saluki_core::components::forwarders::Forwarder>::run::_{{closure}}::hf061731a7e18b6ce
  [NEW] +29.9Ki  [NEW] +29.7Ki    _<saluki_components::transforms::aggregate::Aggregate as saluki_core::components::transforms::Transform>::run::_{{closure}}::h5255c5fdf9a6142d
  [NEW] +29.6Ki  [NEW] +29.5Ki    saluki_components::sources::otlp::metrics::translator::OtlpMetricsTranslator::translate_metrics::h762652f9497ab51c
  [NEW] +28.8Ki  [NEW] +28.7Ki    agent_data_plane::cli::dogstatsd::handle_dogstatsd_command::_{{closure}}::hbb422c48c74eced3
  [NEW] +28.3Ki  [NEW] +28.0Ki    _<saluki_components::sources::dogstatsd::_::<impl serde_core::de::Deserialize for saluki_components::sources::dogstatsd::DogStatsDConfiguration>::deserialize::__Visitor as serde_core::de::Visitor>::visit_map::hf9734c02c96e53ea
  [NEW] +25.7Ki  [NEW] +25.6Ki    saluki_components::sources::dogstatsd::drive_stream::_{{closure}}::hc4388ae93a328f58
  [NEW] +24.9Ki  [NEW] +24.8Ki    agent_data_plane::internal::remote_agent::run_remote_agent_registration_loop::_{{closure}}::h45de6159a7a3ece6
  [DEL] -24.6Ki  [DEL] -24.5Ki    saluki_components::sources::dogstatsd::drive_stream::_{{closure}}::hf19a0326c8f722dc
  [DEL] -25.0Ki  [DEL] -24.8Ki    agent_data_plane::internal::remote_agent::run_remote_agent_registration_loop::_{{closure}}::h9eb2ca20d8f6c771
  [DEL] -25.6Ki  [DEL] -25.3Ki    _<saluki_components::sources::dogstatsd::_::<impl serde_core::de::Deserialize for saluki_components::sources::dogstatsd::DogStatsDConfiguration>::deserialize::__Visitor as serde_core::de::Visitor>::visit_map::h0103c168425238c2
  [DEL] -28.2Ki  [DEL] -27.9Ki    _<saluki_components::sources::dogstatsd::DogStatsDConfiguration as saluki_core::components::sources::builder::SourceBuilder>::build::_{{closure}}::h404793891308bafd
  [DEL] -28.6Ki  [DEL] -28.4Ki    _<saluki_components::transforms::aggregate::Aggregate as saluki_core::components::transforms::Transform>::run::_{{closure}}::hddf6db314d6a5b7b
  [DEL] -28.8Ki  [DEL] -28.7Ki    agent_data_plane::cli::dogstatsd::handle_dogstatsd_command::_{{closure}}::h930fe88d5710a074
  [DEL] -28.9Ki  [DEL] -28.7Ki    saluki_components::sources::otlp::metrics::translator::OtlpMetricsTranslator::translate_metrics::hc70a768805bd3303
  [DEL] -30.8Ki  [DEL] -30.7Ki    saluki_components::encoders::datadog::metrics::encode_v3_metrics_batch::h21043daf6d8d5f0a
  [DEL] -38.3Ki  [DEL] -38.1Ki    _<saluki_components::forwarders::otlp::OtlpForwarder as saluki_core::components::forwarders::Forwarder>::run::_{{closure}}::hfe25375016005a13
  [DEL] -44.5Ki  [DEL] -44.3Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::h897f166a83600cf6
  [DEL] -56.8Ki  [DEL] -56.6Ki    saluki_components::common::datadog::io::run_endpoint_io_loop::_{{closure}}::h2e07bfc47ccf6007
  +0.1% +21.5Ki  +0.2% +54.6Ki    TOTAL

pr-commenter · 2026-06-30T19:35:20Z

Regression Detector (Agent Data Plane)

Run ID: a7bd2e84-981f-453f-b552-8222b118ceda
Baseline: 94ecf81c · Comparison: 3d25c158 · diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment (5)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment	goal	Δ mean %	links
quality_gates_rss_idle	memory	⚪ +0.37	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory	⚪ +0.28	metrics profiles logs
quality_gates_rss_dsd_heavy	memory	⚪ +0.07	metrics profiles logs
quality_gates_rss_dsd_medium	memory	⚪ -0.02	metrics profiles logs
quality_gates_rss_dsd_low	memory	⚪ -0.32	metrics profiles logs

Bounds Checks: ✅ Passed (5)

experiment	check	replicates	observed	links
quality_gates_rss_dsd_heavy	memory_usage	10/10	✅ 138 MiB ≤ 140 MiB	metrics profiles logs
quality_gates_rss_dsd_low	memory_usage	10/10	✅ 42.9 MiB ≤ 50 MiB	metrics profiles logs
quality_gates_rss_dsd_medium	memory_usage	10/10	✅ 64.8 MiB ≤ 75 MiB	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory_usage	10/10	✅ 191 MiB ≤ 200 MiB	metrics profiles logs
quality_gates_rss_idle	memory_usage	10/10	✅ 28.9 MiB ≤ 40 MiB	metrics profiles logs

Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression.

…atsd-named-pipe # Conflicts: # lib/saluki-io/src/net/addr.rs

Replace the DogStatsD-specific Panoramic action with a reusable container_exec action. The named-pipe integration case now mounts a small PowerShell script and executes it inside the Windows target container, keeping DogStatsD-specific behavior in the test case instead of the harness API. This still verifies the end-to-end Windows named-pipe path: ADP starts the listener, PowerShell connects to the named pipe, writes a raw newline-delimited DogStatsD payload, and the test asserts ADP decodes and logs the invalid frame.

Rename the integration-test helper from container_exec to target_exec and make it run in either target environment. Container-backed runtimes still use Docker exec, while host-process runtimes execute the command directly with tokio::process::Command. The Windows named-pipe integration case keeps using a mounted PowerShell script, but no longer requires a DogStatsD-specific or container-only Panoramic action.

thieman · 2026-07-01T14:10:34Z

    uds: bool,
+    // The core Agent exposes `named_pipe` as a separate `dogstatsd_eol_required` target.
+    // Named pipes use raw newline framing like UDP datagrams, but they have their own switch
+    // so Windows clients can require newline-terminated pipe frames without affecting UDS.


Remove this comment

target_exec is a generic command action, not a readiness poller. Run the command once with timeout/cancellation semantics so script errors fail immediately with their stdout/stderr instead of being retried until the action timeout expires. Keep host-process support by executing commands with tokio::process::Command and kill_on_drop enabled, and keep container support by wrapping a single Docker exec call in the same timeout/cancellation behavior.

Remove the explanatory comment on the named_pipe eol_required field per PR review. The surrounding code and tests already document the behavior sufficiently.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 496d200708

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

webern

Config changes look good. You did the right thing with the overlay and the generated diff flows from there and looks like what we expect.

thieman · 2026-07-01T14:36:05Z

I'll tell Claude he did a great job

Named-pipe DogStatsD traffic uses raw newline framing to match the core Agent. Unlike UDP datagrams, named pipes are connection-oriented, so a partial frame can fill the fixed I/O buffer before a newline arrives. Detect that full-buffer/no-newline state for named-pipe listeners, count it as a framing error, clear the buffer, and continue reading. This matches the core Agent behavior of dropping oversized named-pipe messages instead of decoding truncated data or disconnecting the client.

Avoid mounting the PowerShell helper into the Windows container. Docker rejected the POSIX-style container destination path in CI, so keep the named-pipe client script inline in the target_exec command. Also relax the target_exec nonzero-exit unit test so it asserts preserved stderr rather than Unix-specific exit status formatting. Windows reports process status differently, but the behavior under test is immediate failure with diagnostic output.

…atsd-named-pipe # Conflicts: # docs/agent-data-plane/configuration/dogstatsd.md # lib/datadog-agent/config-testing/src/config_registry/dogstatsd.rs # lib/saluki-components/src/sources/dogstatsd/mod.rs

Use memchr::memchr in the named-pipe oversized-frame guard instead of slice::contains. This matches the surrounding framing code's optimized delimiter search pattern.

Record the Cargo.lock update from adding memchr as a direct saluki-components dependency. The named-pipe oversized-frame guard uses memchr for newline detection.

Extend the Windows named-pipe integration case with two behaviors covered by the core Agent tests: multiple newline-delimited messages on one connection, and an oversized unterminated write followed by a valid message on a new connection. The test lowers dogstatsd_buffer_size to make the oversized path deterministic, then asserts ADP still decodes messages sent after the oversized frame is dropped.

Do not lower dogstatsd_buffer_size for the named-pipe integration test. The core Agent also writes DogStatsD telemetry to the configured named pipe, and a tiny pipe/read buffer fragments those background metrics into noisy parse failures. Keep the oversized-frame coverage deterministic by writing a payload larger than the default DogStatsD buffer instead, then assert a subsequent valid frame is decoded on a new connection.

…atsd-named-pipe # Conflicts: # lib/datadog-agent/config/src/classifier/classifier_data.rs

Extend the Windows named-pipe integration test with paired payloads that differ only by newline termination. With DD_DOGSTATSD_EOL_REQUIRED=named_pipe, the newline-terminated invalid payload should reach DogStatsD parsing and appear in logs, while the short unterminated payload should be rejected by the framer and stay out of parse logs.

## Summary - Add a reusable Windows named-pipe listen address/transport in `saluki-io`. - Wire DogStatsD config to enable `dogstatsd_pipe_name` and `dogstatsd_windows_pipe_security_descriptor`. - Use core-Agent-compatible raw newline framing for Windows named-pipe DogStatsD payloads. - Drop oversized named-pipe frames that fill the I/O buffer before a newline arrives, matching the core Agent's oversized-message behavior. - Match core Agent pipe creation behavior by applying `dogstatsd_buffer_size` as the Windows pipe input buffer and using an output buffer size of 0. - Accept either bare pipe names or full `\\.\pipe\...` values when constructing named-pipe listen addresses. - Use `npipe://` for URL-style display/parsing of Windows named-pipe addresses, matching existing Agent conventions for named-pipe socket paths. - Update DogStatsD config inventory/docs from planned unsupported to supported and cover named-pipe newline/stream behavior. - Add a Windows integration case that executes a PowerShell named-pipe client via `target_exec`, writes to the named pipe, and asserts ADP decodes the frame. ## Test plan - `cargo test -p panoramic` - `cargo run --release --bin panoramic -- list -d test/integration/cases --runtime windows --json | rg 'dogstatsd-named-pipe-windows'` - `cargo check --workspace` - `cargo check --workspace --tests` - `cargo test -p saluki-io` - `cargo test -p saluki-components dogstatsd` - `cargo xwin check -p saluki-io --target x86_64-pc-windows-msvc` - `cargo xwin check -p saluki-components --target x86_64-pc-windows-msvc` - `make fmt` CI investigation note: earlier Windows integration runs proved the action connected/wrote to the pipe, then exposed two bugs: ADP initially treated named-pipe bytes as length-delimited stream frames, and a mounted helper script used a container path Docker rejected on Windows. The test now runs an inline PowerShell client through `target_exec`, and ADP uses raw newline framing for named pipes. Co-authored-by: travis.thieman <travis.thieman@datadoghq.com> 54d3177

thieman added 6 commits June 30, 2026 14:46

docs: design Windows DogStatsD named pipe support

75b4392

docs: plan Windows DogStatsD named pipe support

101a3d0

feat(dogstatsd): configure Windows named pipe listener

e9c847a

feat(io): add Windows named pipe listener support

e61caa0

chore: update lockfile for named pipe support

42138a0

feat(dogstatsd): enable Windows named pipe intake

ba1541b

dd-octo-sts Bot added area/io General I/O and networking. area/components Sources, transforms, and destinations. source/dogstatsd DogStatsD source. area/docs Reference documentation. labels Jun 30, 2026

test(integration): cover Windows DogStatsD named pipe intake

5881fb9

dd-octo-sts Bot added the area/test All things testing: unit/integration, correctness, SMP regression, etc. label Jun 30, 2026

This comment has been minimized.

Sign in to view

thieman added 2 commits June 30, 2026 15:42

fix(dogstatsd): use newline framing for Windows named pipes

9e6df47

chore(docs): remove internal planning artifacts

023d004

thieman commented Jun 30, 2026

View reviewed changes

Comment thread lib/datadog-agent/config/src/generated.rs

thieman commented Jun 30, 2026

View reviewed changes

Comment thread lib/saluki-components/src/sources/dogstatsd/mod.rs

thieman commented Jun 30, 2026

View reviewed changes

Comment thread lib/saluki-io/src/net/addr.rs

thieman added 7 commits June 30, 2026 15:57

fix(dogstatsd): clarify named pipe configuration behavior

ff356d0

fix(dogstatsd): address named pipe review feedback

71c61f3

fix(io): use npipe scheme for named pipe addresses

c8ea35b

fix(dogstatsd): match Agent named pipe buffer sizing

d56f0eb

Merge remote-tracking branch 'origin/main' into thieman/dadp-99-dogst…

41e1cf0

…atsd-named-pipe # Conflicts: # lib/saluki-io/src/net/addr.rs

thieman commented Jul 1, 2026

View reviewed changes

thieman added 2 commits July 1, 2026 10:11

chore(dogstatsd): remove named pipe eol comment

496d200

Remove the explanatory comment on the named_pipe eol_required field per PR review. The surrounding code and tests already document the behavior sufficiently.

thieman marked this pull request as ready for review July 1, 2026 14:16

thieman requested a review from a team as a code owner July 1, 2026 14:16

chatgpt-codex-connector Bot reviewed Jul 1, 2026

View reviewed changes

Comment thread lib/saluki-components/src/sources/dogstatsd/framer.rs

webern reviewed Jul 1, 2026

View reviewed changes

thieman added 3 commits July 1, 2026 10:45

Merge remote-tracking branch 'origin/main' into thieman/dadp-99-dogst…

8750b19

…atsd-named-pipe # Conflicts: # docs/agent-data-plane/configuration/dogstatsd.md # lib/datadog-agent/config-testing/src/config_registry/dogstatsd.rs # lib/saluki-components/src/sources/dogstatsd/mod.rs

tobz reviewed Jul 1, 2026

View reviewed changes

Comment thread lib/saluki-components/src/sources/dogstatsd/mod.rs

Comment thread lib/saluki-components/src/sources/dogstatsd/mod.rs Outdated

thieman added 5 commits July 1, 2026 13:24

chore(dogstatsd): use memchr for named pipe newline search

2a4af3a

Use memchr::memchr in the named-pipe oversized-frame guard instead of slice::contains. This matches the surrounding framing code's optimized delimiter search pattern.

chore(deps): record memchr dependency for components

772ec34

Record the Cargo.lock update from adding memchr as a direct saluki-components dependency. The named-pipe oversized-frame guard uses memchr for newline detection.

Merge remote-tracking branch 'origin/main' into thieman/dadp-99-dogst…

fae843a

…atsd-named-pipe # Conflicts: # lib/datadog-agent/config/src/classifier/classifier_data.rs

dd-octo-sts Bot added the area/core Core functionality, event model, etc. label Jul 1, 2026

tobz approved these changes Jul 2, 2026

View reviewed changes

gh-worker-dd-devflow-36fce6 Bot added mergequeue-status: queued mergequeue-status: in_progress and removed mergequeue-status: queued labels Jul 2, 2026

gh-worker-dd-mergequeue-cf854d Bot merged commit 54d3177 into main Jul 2, 2026
85 checks passed

gh-worker-dd-devflow-36fce6 Bot added mergequeue-status: done and removed mergequeue-status: in_progress labels Jul 2, 2026

gh-worker-dd-mergequeue-cf854d Bot deleted the thieman/dadp-99-dogstatsd-named-pipe branch July 2, 2026 16:10

jszwedko mentioned this pull request Jul 7, 2026

Support Windows named pipe listener for DogStatsD (dogstatsd_pipe_name, dogstatsd_windows_pipe_security_descriptor). #1466

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(dogstatsd): add Windows named pipe intake#1956

feat(dogstatsd): add Windows named pipe intake#1956
gh-worker-dd-mergequeue-cf854d[bot] merged 27 commits into
mainfrom
thieman/dadp-99-dogstatsd-named-pipe

thieman commented Jun 30, 2026 •

edited

Loading

Uh oh!

pr-commenter Bot commented Jun 30, 2026 •

edited

Loading

Uh oh!

pr-commenter Bot commented Jun 30, 2026 •

edited

Loading

Uh oh!

This comment has been minimized.

Uh oh!

Uh oh!

Uh oh!

thieman Jul 1, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

webern left a comment

Uh oh!

thieman commented Jul 1, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

thieman commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

pr-commenter Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Binary Size Analysis (Agent Data Plane)

✅ Binary size difference within threshold

Uh oh!

pr-commenter Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Regression Detector (Agent Data Plane)

Optimization Goals: ✅ No significant changes detected

Uh oh!

This comment has been minimized.

Uh oh!

Uh oh!

Uh oh!

thieman Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

webern left a comment

Choose a reason for hiding this comment

Uh oh!

thieman commented Jul 1, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

thieman commented Jun 30, 2026 •

edited

Loading

pr-commenter Bot commented Jun 30, 2026 •

edited

Loading

pr-commenter Bot commented Jun 30, 2026 •

edited

Loading