Optionally emit Dag tags as structured log context on task logs - #71793
Draft
hanxdatadog wants to merge 2 commits into
Draft
Optionally emit Dag tags as structured log context on task logs#71793hanxdatadog wants to merge 2 commits into
hanxdatadog wants to merge 2 commits into
Conversation
Dag tags are already surfaced in metrics via [metrics] dag_tags_in_metrics, but users who want the same team/environment attribution in structured task logs currently have to write a custom Airflow listener plugin to bind that context themselves. This adds an equivalent, opt-in [logging] dag_tags_in_logs option, reusing the same tag-parsing helper so both surfaces share one convention.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Optionally emit Dag tags as structured log context (logging counterpart to
dag_tags_in_metrics)What does this PR do?
[logging] dag_tags_in_logsconfig option (boolean, defaultFalse).task_runner.startup()binds Dag-tag-derived fields into structlog contextvars right after the Dag is parsed, so every subsequent log line for that task attempt carries them.build_dag_metric_tags()(the same helper backing[metrics] dag_tags_in_metrics, added in Optionally emit Dag tags as metric tags #68568) so tags of the formkey:valuesplit into a key/value field; plain tags map to an empty-string value — keeping the same tag-parsing convention across metrics and logs.ti_id,dag_id,task_id,run_id,try_number,map_index) always win on collision with a same-named Dag tag.Motivation
Dag tags (e.g.
team:data-eng,env:prod) are already surfaced in metrics viadag_tags_in_metrics, but not in structured logs — users who want this today have to write a custom Airflow listener plugin to bind the context themselves. This closes that gap with a small, opt-in, native option, mirroring the metrics precedent.Testing
task-sdk/tests/task_sdk/execution_time/test_task_runner.pycovering: flag off (no leak into log context), flag on with mixedkey:value/bare tags, flag on with no tags, and the reserved-key collision guard.breeze testing core-tests: 7/7 pass (4 new + 3 pre-existingdag_tags_in_metrics/stats_tagstests, confirming no regression).ruff format,ruff check,mypy-task-sdk, YAML lint, newsfragment validation — all pass viaprek.Additional Notes
task_runner.py+ config + tests only — no changes to the Listener Plugin API, executors, or Celery. Dag tags are already available in-process at this point (the Dag is parsed just above), so no new DB/ORM access is introduced.71793.feature.rst.Who will it impact?
Opt-in only (default
False) — no behavior change for existing deployments unless explicitly enabled.Was generative AI tooling used to co-author this PR?