Skip to content

fix(sidecar): configure OTLP endpoint for FFE metrics#2076

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 5 commits into
mainfrom
leo.romanovsky/ffe-metrics-test-token
Jun 4, 2026
Merged

fix(sidecar): configure OTLP endpoint for FFE metrics#2076
gh-worker-dd-mergequeue-cf854d[bot] merged 5 commits into
mainfrom
leo.romanovsky/ffe-metrics-test-token

Conversation

@leoromanovsky
Copy link
Copy Markdown
Contributor

@leoromanovsky leoromanovsky commented Jun 3, 2026

Motivation

Companion PHP evaluation-metrics work needs FFE metrics to follow the sidecar architecture used by the rest of the tracer: configure endpoints once on the session, then send typed sidecar actions without carrying endpoint strings on every flush.

The previous FFE metrics path accepted an OTLP endpoint alongside each metrics batch. That diverged from the normal Endpoint flow, made test-session-token propagation easy to miss, and forced PHP to keep fetching endpoint configuration during request flushes.

Design context: https://docs.google.com/document/d/1NvMfTpZWLBlFmEFNjdnlMyeVpy5l7KD8qujGFco6w2w/edit?tab=t.0

Companion PHP PR: DataDog/dd-trace-php#3911

Changes

  • Add an optional OTLP metrics Endpoint to sidecar session configuration.
  • Thread that endpoint through ddog_sidecar_session_set_config and session state.
  • Dispatch SidecarAction::FfeEvaluationMetrics through the session's configured OTLP metrics endpoint instead of an endpoint passed with each metrics batch.
  • Preserve request-replayer test-session routing by inheriting the agent endpoint test_token when the OTLP metrics endpoint does not already have one.
  • Update runtime test-token changes so the configured OTLP metrics endpoint is updated with the rest of the session endpoints.
  • Extend the sidecar FFE metrics dispatch test to assert the outgoing OTLP request carries x-datadog-test-session-token through Endpoint standard headers.

Decisions

FFE metrics remain caller-driven. This PR does not make shared evaluator calls auto-emit metrics, so SDKs that still log metrics in host language implementations can continue to coexist without double counting.

The sidecar owns metrics transport, aggregation, OTLP/protobuf encoding, and endpoint/header handling. PHP remains responsible for constructing the appropriate OTLP metrics endpoint from its existing configuration resolver and supplying it once during sidecar session setup.

The test-session-token behavior is intentionally implemented through Endpoint, not by manually adding FFE-specific headers. Production endpoints normally have no test token, so production behavior is unchanged.

Validation

Local targeted test:

cargo test -p datadog-sidecar ffe_metric_actions_dispatch_without_registered_application

Result: passed locally.

PHP integration smoke test:

Companion PHP PR: DataDog/dd-trace-php#3911

The local PHP branch backing that PR was updated to this libdatadog PR head (091c98df6): PHP passes the OTLP metrics Endpoint once during sidecar session setup, and FFE metric flushes send only context plus metric payloads.

docker-compose run --no-deps 8.3-bookworm bash -lc \
  'export PATH=/rust/cargo/bin:$PATH TMPDIR=/home/circleci/app/tmp TMP=/home/circleci/app/tmp TEMP=/home/circleci/app/tmp; make MAX_TEST_PARALLELISM=1 test_c TESTS=tests/ext/ffe/evaluation_metrics_unix_agent_endpoint.phpt'

Result: tests/ext/ffe/evaluation_metrics_unix_agent_endpoint.phpt passed locally under Docker on PHP 8.3/Linux aarch64: Tests passed 1 (100.0%).

That PHPT validates the PHP sidecar integration over DD_TRACE_AGENT_URL=unix://...: the request reaches /v1/metrics, uses application/x-protobuf, and carries the request-replayer test-session token through the configured endpoint path.

Negative validation during review:

  • Removing only the mock header matcher still allows the request through, but stops proving token propagation.
  • Keeping the matcher and removing Endpoint.test_token fails with a missing x-datadog-test-session-token header.
  • Restoring both passes, confirming the request uses Endpoint standard headers.

CI on commit 091c98d reports all tests passed and no new flaky tests.

End-to-End Evidence

The companion PHP branch for DataDog/dd-trace-php#3911 was validated locally against this libdatadog PR head (091c98df6). That path exercises the full PHP metrics flow:

PHP API -> tracer request buffer -> sidecar session-configured OTLP metrics Endpoint -> sidecar FFE metrics action -> request-replayer UDS intake

Docker command used from the PHP repo:

docker-compose run --no-deps 8.3-bookworm bash -lc \
  'export PATH=/rust/cargo/bin:$PATH TMPDIR=/home/circleci/app/tmp TMP=/home/circleci/app/tmp TEMP=/home/circleci/app/tmp; make MAX_TEST_PARALLELISM=1 test_c TESTS=tests/ext/ffe/evaluation_metrics_unix_agent_endpoint.phpt'

Observed result:

Number of tests :     1                 1
Tests failed    :     0 (  0.0%) (  0.0%)
Tests passed    :     1 (100.0%) (100.0%)

The passing PHPT confirms PHP can record and flush an FFE evaluation metric through the sidecar over DD_TRACE_AGENT_URL=unix://..., and that the captured request reaches /v1/metrics with Content-Type: application/x-protobuf plus the request-replayer test-session token propagated through the configured Endpoint.

@datadog-prod-us1-6
Copy link
Copy Markdown

datadog-prod-us1-6 Bot commented Jun 3, 2026

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 87.06%
Overall Coverage: 73.60% (+0.16%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 5e86692 | Docs | Datadog PR Page | Give us feedback!

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jun 3, 2026

Clippy Allow Annotation Report

Comparing clippy allow annotations between branches:

  • Base Branch: origin/main
  • PR Branch: origin/leo.romanovsky/ffe-metrics-test-token

Summary by Rule

Rule Base Branch PR Branch Change
expect_used 2 2 No change (0%)
unwrap_used 5 5 No change (0%)
Total 7 7 No change (0%)

Annotation Counts by File

File Base Branch PR Branch Change
datadog-sidecar/src/service/session_info.rs 1 1 No change (0%)
datadog-sidecar/src/service/sidecar_server.rs 6 6 No change (0%)

Annotation Stats by Crate

Crate Base Branch PR Branch Change
clippy-annotation-reporter 5 5 No change (0%)
datadog-ffe-ffi 1 1 No change (0%)
datadog-ipc 21 21 No change (0%)
datadog-live-debugger 6 6 No change (0%)
datadog-live-debugger-ffi 10 10 No change (0%)
datadog-profiling-replayer 4 4 No change (0%)
datadog-remote-config 3 3 No change (0%)
datadog-sidecar 57 57 No change (0%)
libdd-common 13 13 No change (0%)
libdd-common-ffi 12 12 No change (0%)
libdd-data-pipeline 5 5 No change (0%)
libdd-ddsketch 2 2 No change (0%)
libdd-dogstatsd-client 1 1 No change (0%)
libdd-profiling 13 13 No change (0%)
libdd-telemetry 20 20 No change (0%)
libdd-tinybytes 4 4 No change (0%)
libdd-trace-normalization 2 2 No change (0%)
libdd-trace-obfuscation 3 3 No change (0%)
libdd-trace-stats 1 1 No change (0%)
libdd-trace-utils 13 13 No change (0%)
Total 196 196 No change (0%)

About This Report

This report tracks Clippy allow annotations for specific rules, showing how they've changed in this PR. Decreasing the number of these annotations generally improves code quality.

@leoromanovsky leoromanovsky changed the title Preserve FFE metrics test token fix(sidecar): preserve FFE metrics test token Jun 3, 2026
@dd-octo-sts
Copy link
Copy Markdown
Contributor

dd-octo-sts Bot commented Jun 3, 2026

Artifact Size Benchmark Report

aarch64-alpine-linux-musl
Artifact Baseline Commit Change
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.a 82.85 MB 82.85 MB 0% (0 B) 👌
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.so 7.63 MB 7.63 MB 0% (0 B) 👌
aarch64-unknown-linux-gnu
Artifact Baseline Commit Change
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.a 93.92 MB 93.92 MB 0% (0 B) 👌
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.so 10.25 MB 10.25 MB 0% (0 B) 👌
libdatadog-x64-windows
Artifact Baseline Commit Change
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.dll 24.53 MB 24.53 MB 0% (0 B) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.lib 83.96 KB 83.96 KB 0% (0 B) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.pdb 178.05 MB 178.05 MB -0% (-8.00 KB) 👌
/libdatadog-x64-windows/debug/static/datadog_profiling_ffi.lib 913.44 MB 913.44 MB 0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.dll 8.02 MB 8.02 MB 0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.lib 83.96 KB 83.96 KB 0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.pdb 23.75 MB 23.75 MB 0% (0 B) 👌
/libdatadog-x64-windows/release/static/datadog_profiling_ffi.lib 47.38 MB 47.38 MB 0% (0 B) 👌
libdatadog-x86-windows
Artifact Baseline Commit Change
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.dll 21.25 MB 21.25 MB 0% (0 B) 👌
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.lib 85.29 KB 85.29 KB 0% (0 B) 👌
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.pdb 181.98 MB 181.98 MB 0% (0 B) 👌
/libdatadog-x86-windows/debug/static/datadog_profiling_ffi.lib 906.06 MB 906.06 MB 0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.dll 6.19 MB 6.19 MB 0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.lib 85.29 KB 85.29 KB 0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.pdb 25.45 MB 25.45 MB 0% (0 B) 👌
/libdatadog-x86-windows/release/static/datadog_profiling_ffi.lib 45.03 MB 45.03 MB 0% (0 B) 👌
x86_64-alpine-linux-musl
Artifact Baseline Commit Change
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.a 73.86 MB 73.86 MB 0% (0 B) 👌
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.so 8.52 MB 8.52 MB 0% (0 B) 👌
x86_64-unknown-linux-gnu
Artifact Baseline Commit Change
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.a 89.27 MB 89.27 MB 0% (0 B) 👌
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.so 10.35 MB 10.35 MB 0% (0 B) 👌

@leoromanovsky leoromanovsky marked this pull request as ready for review June 3, 2026 18:26
@leoromanovsky leoromanovsky requested review from a team as code owners June 3, 2026 18:26
@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Jun 3, 2026

Codecov Report

❌ Patch coverage is 87.05882% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.60%. Comparing base (b2ef19f) to head (5e86692).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2076      +/-   ##
==========================================
+ Coverage   73.43%   73.60%   +0.16%     
==========================================
  Files         470      470              
  Lines       78340    78449     +109     
==========================================
+ Hits        57532    57741     +209     
+ Misses      20808    20708     -100     
Components Coverage Δ
libdd-crashtracker 65.44% <ø> (-0.02%) ⬇️
libdd-crashtracker-ffi 37.68% <ø> (ø)
libdd-agent-client 83.79% <ø> (ø)
libdd-alloc 98.77% <ø> (ø)
libdd-data-pipeline 87.13% <ø> (-0.02%) ⬇️
libdd-data-pipeline-ffi 77.03% <ø> (ø)
libdd-common 79.93% <ø> (+0.04%) ⬆️
libdd-common-ffi 74.41% <ø> (ø)
libdd-telemetry 73.34% <ø> (ø)
libdd-telemetry-ffi 31.36% <ø> (ø)
libdd-dogstatsd-client 82.64% <ø> (ø)
datadog-ipc 76.22% <ø> (+1.46%) ⬆️
libdd-profiling 81.69% <ø> (ø)
libdd-profiling-ffi 64.79% <ø> (ø)
libdd-sampling 97.41% <ø> (ø)
datadog-sidecar 35.99% <87.05%> (+1.38%) ⬆️
datdog-sidecar-ffi 15.89% <95.45%> (+7.27%) ⬆️
spawn-worker 48.86% <ø> (ø)
libdd-tinybytes 93.80% <ø> (ø)
libdd-trace-normalization 81.71% <ø> (ø)
libdd-trace-obfuscation 87.30% <ø> (ø)
libdd-trace-protobuf 68.25% <ø> (ø)
libdd-trace-utils 89.29% <ø> (+0.08%) ⬆️
libdd-tracer-flare 86.88% <ø> (ø)
libdd-log 74.83% <ø> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@leoromanovsky leoromanovsky requested a review from a team as a code owner June 3, 2026 19:22
@leoromanovsky leoromanovsky requested review from dd-oleksii, sameerank and typotter and removed request for a team and typotter June 3, 2026 19:22
@leoromanovsky leoromanovsky changed the title fix(sidecar): preserve FFE metrics test token fix(sidecar): configure OTLP endpoint for FFE metrics Jun 3, 2026
Comment thread datadog-sidecar-ffi/src/lib.rs Outdated
parent_session_id: ffi::CharSlice,
) -> MaybeError {
let session_id_str: String = session_id.to_utf8_lossy().into();
let mut otlp_metrics_endpoint = unsafe { otlp_metrics_endpoint.as_ref().cloned() };
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Was a bit confused by this variable shadowing approach, though I guess it's not uncommon.

id: *const ConfigInvariants,
target: *const Arc<Target>,
) -> *mut c_char {
let id = unsafe { &*id };
let target = unsafe { &*target };

Maybe you could add a : Option<Endpoint> type hint to differentiate this from *const Endpoint ?

..Default::default()
},
&Endpoint::default(),
null(),
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An observation: no non-null case means that this code is not tested

if let Some(endpoint) = &mut otlp_metrics_endpoint {
if endpoint.test_token.is_none() {
endpoint.test_token = agent_endpoint.test_token.clone();
}
}

Not sure if there are any challenges to testing that path, so up to you if it's worth the trouble.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the careful review!

Copy link
Copy Markdown
Contributor

@yannham yannham left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't say about the PHP/sidecar's side of things, but the Rust part LGTM

Copy link
Copy Markdown
Contributor

@bwoebi bwoebi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks reasonable, approving!

@gh-worker-dd-mergequeue-cf854d gh-worker-dd-mergequeue-cf854d Bot merged commit 582bee1 into main Jun 4, 2026
89 checks passed
@gh-worker-dd-mergequeue-cf854d gh-worker-dd-mergequeue-cf854d Bot deleted the leo.romanovsky/ffe-metrics-test-token branch June 4, 2026 13:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants