Skip to content

Optionally emit Dag tags as structured log context on task logs - #71793

Draft
hanxdatadog wants to merge 2 commits into
apache:mainfrom
hanxdatadog:hanx/dag-tags-in-logs
Draft

Optionally emit Dag tags as structured log context on task logs#71793
hanxdatadog wants to merge 2 commits into
apache:mainfrom
hanxdatadog:hanx/dag-tags-in-logs

Conversation

@hanxdatadog

@hanxdatadog hanxdatadog commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Optionally emit Dag tags as structured log context (logging counterpart to dag_tags_in_metrics)

What does this PR do?

  • Adds a new [logging] dag_tags_in_logs config option (boolean, default False).
  • When enabled, task_runner.startup() binds Dag-tag-derived fields into structlog contextvars right after the Dag is parsed, so every subsequent log line for that task attempt carries them.
  • Reuses build_dag_metric_tags() (the same helper backing [metrics] dag_tags_in_metrics, added in Optionally emit Dag tags as metric tags #68568) so tags of the form key:value split into a key/value field; plain tags map to an empty-string value — keeping the same tag-parsing convention across metrics and logs.
  • Built-in TI identifier keys (ti_id, dag_id, task_id, run_id, try_number, map_index) always win on collision with a same-named Dag tag.

Motivation

Dag tags (e.g. team:data-eng, env:prod) are already surfaced in metrics via dag_tags_in_metrics, but not in structured logs — users who want this today have to write a custom Airflow listener plugin to bind the context themselves. This closes that gap with a small, opt-in, native option, mirroring the metrics precedent.

Testing

  • Unit tests: 4 new tests added to task-sdk/tests/task_sdk/execution_time/test_task_runner.py covering: flag off (no leak into log context), flag on with mixed key:value/bare tags, flag on with no tags, and the reserved-key collision guard.
  • Ran via breeze testing core-tests: 7/7 pass (4 new + 3 pre-existing dag_tags_in_metrics/stats_tags tests, confirming no regression).
  • Static checks: ruff format, ruff check, mypy-task-sdk, YAML lint, newsfragment validation — all pass via prek.

Additional Notes

  • Intentionally scoped to task_runner.py + config + tests only — no changes to the Listener Plugin API, executors, or Celery. Dag tags are already available in-process at this point (the Dag is parsed just above), so no new DB/ORM access is introduced.
  • Newsfragment filename: 71793.feature.rst.
  • Related: Propagate DAG tags to Sentry for alert routing #37901 (Dag tags → observability tooling, currently Sentry-scoped) and Optionally emit Dag tags as metric tags #68568 (the metrics counterpart of this change).

Who will it impact?

Opt-in only (default False) — no behavior change for existing deployments unless explicitly enabled.


Was generative AI tooling used to co-author this PR?
  • Yes (Claude Code)

Dag tags are already surfaced in metrics via [metrics] dag_tags_in_metrics,
but users who want the same team/environment attribution in structured
task logs currently have to write a custom Airflow listener plugin to
bind that context themselves. This adds an equivalent, opt-in
[logging] dag_tags_in_logs option, reusing the same tag-parsing helper
so both surfaces share one convention.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant