Skip to content

enhancement(core): Add efficient trace payload (aka v1) encoder#1794

Draft
ajgajg1134 wants to merge 13 commits into
mainfrom
andrew.glaude/moreV1
Draft

enhancement(core): Add efficient trace payload (aka v1) encoder#1794
ajgajg1134 wants to merge 13 commits into
mainfrom
andrew.glaude/moreV1

Conversation

@ajgajg1134

Copy link
Copy Markdown
Contributor

pls no look tobz

Summary

Change Type

  • Bug fix
  • New feature
  • Non-functional (chore, refactoring, docs)
  • Performance

How did you test this PR?

References

ajgajg1134 and others added 2 commits May 29, 2026 16:19
Replaces the `add_tracer_payloads` (protobuf field 42) call with
`add_idx_tracer_payloads` (protobuf field 90), which uses a string
interning table for deduplication. New fields now carried: 128-bit
trace ID on the chunk, sampling mechanism, origin, language version,
runtime ID, and per-span env/version/component/kind.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@dd-octo-sts dd-octo-sts Bot added area/io General I/O and networking. area/components Sources, transforms, and destinations. encoder/datadog-traces Datadog Traces encoder. labels Jun 1, 2026
@datadog-prod-us1-5

This comment has been minimized.

@pr-commenter

pr-commenter Bot commented Jun 1, 2026

Copy link
Copy Markdown

Regression Detector (Agent Data Plane)

Run ID: 54a45479-295c-40d5-a9d6-7b2e2fb28b8d
Baseline: 7bb563e4 · Comparison: 1233f58a · diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment (5)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment goal Δ mean % links
quality_gates_rss_dsd_medium memory ⚪ +0.00 metrics profiles logs
quality_gates_rss_dsd_ultraheavy memory ⚪ -0.02 metrics profiles logs
quality_gates_rss_dsd_heavy memory ⚪ -0.14 metrics profiles logs
quality_gates_rss_idle memory ⚪ -0.39 metrics profiles logs
quality_gates_rss_dsd_low memory ⚪ -0.59 metrics profiles logs
Bounds Checks: ✅ Passed (5)
experiment check replicates observed links
quality_gates_rss_dsd_heavy memory_usage 10/10 ✅ 131 MiB ≤ 140 MiB metrics profiles logs
quality_gates_rss_dsd_low memory_usage 10/10 ✅ 42.6 MiB ≤ 50 MiB metrics profiles logs
quality_gates_rss_dsd_medium memory_usage 10/10 ✅ 64.7 MiB ≤ 75 MiB metrics profiles logs
quality_gates_rss_dsd_ultraheavy memory_usage 10/10 ✅ 191 MiB ≤ 200 MiB metrics profiles logs
quality_gates_rss_idle memory_usage 10/10 ✅ 28.5 MiB ≤ 40 MiB metrics profiles logs
Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression.

Comment thread bin/correctness/datadog-intake/src/app/traces/state.rs
Comment thread lib/protos/datadog/src/lib.rs
Comment thread lib/protos/datadog/build.rs Outdated
Comment thread lib/saluki-components/src/encoders/datadog/traces/mod.rs Outdated
@dd-octo-sts dd-octo-sts Bot added the area/core Core functionality, event model, etc. label Jun 1, 2026
@ajgajg1134 ajgajg1134 changed the title Andrew.glaude/more v1 enhancement(core): Add efficient trace payload (aka v1) encoder Jun 1, 2026
@pr-commenter

pr-commenter Bot commented Jun 1, 2026

Copy link
Copy Markdown

Binary Size Analysis (Agent Data Plane)

Baseline: 7bb563e · Comparison: 1233f58 · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 40.87 MiB (baseline) vs 40.77 MiB (comparison)
Size Change: -95.36 KiB (-0.23%)

✅ Binary size difference within threshold

Changes by Module
Module File Size Symbols
otlp_protos::otlp_include::opentelemetry -46.94 KiB 110
prost +45.08 KiB 100
tokio -41.99 KiB 707
figment -35.41 KiB 148
serde_core +22.35 KiB 112
core -21.88 KiB 1545
&mut serde_json -15.97 KiB 27
axum -11.72 KiB 87
quick_cache +9.63 KiB 40
saluki_components::common::datadog +9.39 KiB 107
tonic +8.93 KiB 51
saluki_common::cache::CacheBuilder<K,V,W,H> -8.14 KiB 3
saluki_components::destinations::dsd_stats +8.12 KiB 2
saluki_common::cache::Cache<K,V,W,H> -7.52 KiB 8
piecemeal +7.18 KiB 18
saluki_components::forwarders::cluster_agent +7.07 KiB 6
anon.1ba911e98cbb868a3910d01d81cd64e3.695.llvm.2067643073951026959 +6.63 KiB 1
anon.261bb94b8d6de96de010fe9a3c608bbb.4.llvm.7628180700126258038 -6.63 KiB 1
saluki_components::transforms::trace_sampler -6.60 KiB 11
saluki_components::sources::otlp +6.21 KiB 21
Detailed Symbol Changes
    FILE SIZE        VM SIZE    
 --------------  -------------- 
  +189% +19.7Ki  +191% +19.7Ki    saluki_components::encoders::datadog::metrics::encode_v3_metrics_batch::h183af91d6a7ed4a5
 +36e2% +17.6Ki +42e2% +17.6Ki    prost::message::Message::encode::h033f747c7fcca8bd
 +26e2% +13.8Ki +44e2% +13.8Ki    _<saluki_components::transforms::trace_obfuscation::TraceObfuscation as saluki_core::components::transforms::SynchronousTransform>::transform_buffer::h8a1ac133c92b7e0f
  [NEW] +13.3Ki  [NEW] +13.1Ki    saluki_components::common::datadog::apm::_::_<impl serde_core::de::Deserialize for saluki_components::common::datadog::apm::ApmConfiguration>::deserialize::h81d7f8a7cda2edc6
 +20e2% +12.7Ki +22e2% +12.7Ki    prost::message::Message::encode_to_vec::hb5ea4dcf442722e6
  +624% +11.7Ki  +691% +11.7Ki    _<saluki_components::sources::otlp::logs::translator::OtlpLogsTranslator as core::iter::traits::iterator::Iterator>::next::h2d6fd7f0b3ed8983
  [NEW] +9.02Ki  [NEW] +8.87Ki    _<serde_json::de::SeqAccess<R> as serde_core::de::SeqAccess>::next_element_seed::h04599995f9f7becf
  [NEW] +8.23Ki  [NEW] +8.09Ki    _<tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::h6650231af0cdb8f5
  [NEW] +7.99Ki  [NEW] +7.81Ki    _<saluki_components::destinations::dsd_stats::DogStatsDStatsAPIHandler as saluki_api::APIHandler>::generate_routes::h166352a1b6d1824d
  [NEW] +7.57Ki  [NEW] +7.37Ki    _<saluki_components::encoders::datadog::traces::DatadogTrace as saluki_core::components::encoders::Encoder>::run::_{{closure}}::hb6e71c7eb3e17af9
  [NEW] +7.51Ki  [NEW] +7.36Ki    _<core::marker::PhantomData<T> as serde_core::de::DeserializeSeed>::deserialize::h3674312e8f38a035
  [DEL] -7.36Ki  [DEL] -7.20Ki    _<&mut serde_json::de::Deserializer<R> as serde_core::de::Deserializer>::deserialize_struct::h4d04809f2fddc209
  [DEL] -7.57Ki  [DEL] -7.37Ki    _<saluki_components::encoders::datadog::traces::DatadogTrace as saluki_core::components::encoders::Encoder>::run::_{{closure}}::h1889b8e699e3b054
  [DEL] -7.77Ki  [DEL] -7.62Ki    _<tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::h7a3f66a8ac330560
  [DEL] -8.02Ki  [DEL] -7.86Ki    _<&mut serde_json::de::Deserializer<R> as serde_core::de::Deserializer>::deserialize_struct::hca5756677c477512
  [DEL] -9.87Ki  [DEL] -9.70Ki    _<&mut serde_json::de::Deserializer<R> as serde_core::de::Deserializer>::deserialize_struct::he770aa8f5094e2d9
  [DEL] -12.1Ki  [DEL] -12.0Ki    saluki_components::sources::otlp::logs::transform::transform_log_record::hea2a036923af6631
  [DEL] -14.1Ki  [DEL] -14.0Ki    saluki_components::transforms::trace_obfuscation::TraceObfuscation::obfuscate_span::h9eaa4de856be9025
  [DEL] -14.8Ki  [DEL] -14.7Ki    figment::figment::Figment::extract::h962b28a07c04a29f
 -71.9% -15.4Ki -72.2% -15.4Ki    h2::proto::connection::DynConnection<B>::recv_frame::h2860dcc6f1c3655c
  -1.5%  -127Ki  -0.6% -38.6Ki    [9497 Others]
  -0.2% -95.4Ki  -0.0% -6.42Ki    TOTAL

Comment thread lib/saluki-components/src/encoders/datadog/traces/mod.rs Outdated
Comment thread lib/saluki-components/src/encoders/datadog/traces/mod.rs Outdated
ajgajg1134 and others added 8 commits June 4, 2026 14:07
The CI runners had 1.0.6 cached from main, but this branch's Makefile
still specified 1.0.3. Since cargo-install skips reinstalling when a
binary already exists, CI was effectively running 1.0.6 against a
LICENSE-3rdparty.csv generated by 1.0.3, causing ~70 packages to be
reported as missing.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/components Sources, transforms, and destinations. area/core Core functionality, event model, etc. area/io General I/O and networking. encoder/datadog-traces Datadog Traces encoder.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants