feat(data-pipeline): move the async boundary up#2064
Conversation
Clippy Allow Annotation ReportComparing clippy allow annotations between branches:
Summary by Rule
Annotation Counts by File
Annotation Stats by Crate
About This ReportThis report tracks Clippy allow annotations for specific rules, showing how they've changed in this PR. Decreasing the number of these annotations generally improves code quality. |
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: 4c79a65 | Docs | Datadog PR Page | Give us feedback! |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #2064 +/- ##
==========================================
+ Coverage 73.40% 73.43% +0.02%
==========================================
Files 464 464
Lines 77970 77934 -36
==========================================
- Hits 57236 57230 -6
+ Misses 20734 20704 -30
🚀 New features to boost your workflow:
|
Artifact Size Benchmark Reportaarch64-alpine-linux-musl
aarch64-unknown-linux-gnu
libdatadog-x64-windows
libdatadog-x86-windows
x86_64-alpine-linux-musl
x86_64-unknown-linux-gnu
|
|
/merge |
|
View all feedbacks in Devflow UI.
This pull request is not mergeable according to GitHub. Common reasons include pending required checks, missing approvals, or merge conflicts — but it could also be blocked by other repository rules or settings.
The expected merge time in
Tests failed on this commit 5d789af: What to do next?
|
Aaalibaba42
force-pushed
the
jwiritah/move-async-up
branch
from
c0c430d to
4c79a65
Compare
|
/merge |
|
View all feedbacks in Devflow UI.
This pull request is not mergeable according to GitHub. Common reasons include pending required checks, missing approvals, or merge conflicts — but it could also be blocked by other repository rules or settings.
The expected merge time in
Tests failed on this commit f2bb5f4: What to do next?
|
|
#test Windows latest is too slow for all_checks so failing merge queue :c |
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
|
4 participants
Summary
Today
SharedRuntime::block_onis buried deep inside the data pipeline (incheck_agent_info,stop_stats_computation, telemetry start-up andsend_deser).That forces every internal path to block on a runtime even when the caller is
already async, and makes it impossible for an async host to drive libdatadog
without nested-runtime hazards.
This PR pushes the async/sync boundary up to the top-level public entry
points. The internals are now async all the way down, and the synchronous
methods are thin
block_onfacades over their_asynccounterparts. This is aprerequisite for letting async consumers (e.g. a Rust service embedding
dd-trace-rs) call the_asyncAPI directly.The change is self-contained to
libdd-data-pipelineand does not touchlibdd-shared-runtime; the sync facades keep using the existing owned-modeSharedRuntime::block_on.What changed
trace_exporter/mod.rssend,send_trace_chunks,shutdownare now thinblock_onfacades oversend_async,send_trace_chunks_asyncand the newshutdown_async.check_agent_infois nowasync(native + wasm); it awaits the stats pathand takes an owned status snapshot via
load_full()to avoid holding a!SendArcSwapguard across.await.send_deser— its deserialize-and-send logic already lives insend_async.trace_exporter/builder.rsbuild_async(the real builder body);buildis now ablock_onfacade that materializes the
SharedRuntimeup front so both share a singleinstance.
client_tel.start().await) instead ofblock_on.!SendHandle::enter()guard is scoped to a block so it drops before any.await, keeping the futureSend.trace_exporter/stats.rsstop_stats_computationandhandle_stats_enabledare nowasyncand awaitthe stats-worker shutdown directly instead of routing through
block_on.Notes / compatibility
send,send_trace_chunks,shutdown,build) areunchanged, so FFI callers are unaffected.
buildfacade is gated to native (#[cfg(not(target_arch = "wasm32"))]),matching the existing convention where
send/send_trace_chunksarenative-only and wasm consumers use the
_asyncvariants (block_ondoes notexist on wasm).
build_asyncis cross-platform.