Skip to content

Render map_index_template during task running state - #57208

Merged
kaxil merged 1 commit into
apache:mainfrom
astronomer:render-template
Oct 26, 2025
Merged

Render map_index_template during task running state#57208
kaxil merged 1 commit into
apache:mainfrom
astronomer:render-template

Conversation

@kaxil

@kaxil kaxil commented Oct 24, 2025

Copy link
Copy Markdown
Member

Currently, custom labels defined via map_index_template for mapped tasks are only rendered and displayed in the UI after task completion. This means users see only numeric indices (0, 1, 2...) while tasks are running, making it difficult to identify which specific mapped task instance is executing.

For example, if a mapped task processes different customers, users want to see "customer_alice" or "customer_bob" in the UI while the task is running, not just "0" or "1".

The map_index_template may reference context
variables set during task execution (e.g., XCom values, return values from execute(), etc.). These variables are only available after the task's execute method completes, which is why rendering was originally delayed until task completion.

Example problematic template:

"Processing {{ ti.xcom_pull(key='customer_name') }}"

or one that access Task Attributes:

"Processing {{ task.value }}"

This PR implements a two-phase rendering approach:

Phase 1:

  • Renders map_index_template immediately after template rendering, before task execution
  • Uses available context variables (task, dag, run_id, logical_date, etc.)
  • Sends the rendered label to the API server for immediate UI display
  • If rendering fails (e.g., due to missing variables), error is suppressed and logged at debug level
  • Provides immediate feedback for templates using pre-execution context

Phase 2:

  • Re-renders map_index_template with full context after task execution
  • Includes all execution-time variables (XCom, Task attributes, return values, etc.)
  • Only sends update if the value changed from Phase 1
  • Handles both success and failure cases
  • For failures, suppresses rendering errors to avoid masking task errors

For templates with execution-time variables, the label updates after execution completes with the final rendered value.

Fixes #39118
Fixes #56722

Oct-24-2025 17-55-37

image

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@kaxil kaxil added this to the Airflow 3.2.0 milestone Oct 24, 2025
@boring-cyborg boring-cyborg Bot added area:API Airflow's REST/HTTP API area:task-sdk labels Oct 24, 2025
@kaxil
kaxil force-pushed the render-template branch 6 times, most recently from 3f78d7e to ad77ee6 Compare October 25, 2025 23:29
@kaxil
kaxil marked this pull request as ready for review October 26, 2025 00:39
Currently, custom labels defined via `map_index_template` for mapped tasks
are only rendered and displayed in the UI after task completion. This means
users see only numeric indices (0, 1, 2...) while tasks are running, making
it difficult to identify which specific mapped task instance is executing.

For example, if a mapped task processes different customers, users want to
see "customer_alice" or "customer_bob" in the UI while the task is running,
not just "0" or "1".

The `map_index_template` may reference context
variables set during task execution (e.g., XCom values, return values from
`execute()`, etc.). These variables are only available after the task's
execute method completes, which is why rendering was originally delayed
until task completion.

Example problematic template:
```
"Processing {{ ti.xcom_pull(key='customer_name') }}"
```

or one that access Task Attributes:

```
"Processing {{ task.value }}"
```

This PR implements a two-phase rendering approach:

Phase 1:
- Renders `map_index_template` immediately after template rendering, before
  task execution
- Uses available context variables (task, dag, run_id, logical_date, etc.)
- Sends the rendered label to the API server for immediate UI display
- If rendering fails (e.g., due to missing variables), error is suppressed
  and logged at debug level
- Provides immediate feedback for templates using pre-execution context

Phase 2:
- Re-renders `map_index_template` with full context after task execution
- Includes all execution-time variables (XCom, Task attributes, return values, etc.)
- Only sends update if the value changed from Phase 1
- Handles both success and failure cases
- For failures, suppresses rendering errors to avoid masking task errors

For templates with execution-time variables, the label updates after
execution completes with the final rendered value.

@gopidesupavan gopidesupavan left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM :)

@gopidesupavan

Copy link
Copy Markdown
Member

may be we need to exclude this new type from the trigger supervisor type tests?

@kaxil
kaxil merged commit 5a5dc4f into apache:main Oct 26, 2025
114 checks passed
@kaxil
kaxil deleted the render-template branch October 26, 2025 14:48
@kaxil

kaxil commented Oct 26, 2025

Copy link
Copy Markdown
Member Author

may be we need to exclude this new type from the trigger supervisor type tests?

Yeah, excluded

@gygabyte017

Copy link
Copy Markdown

Hi, is this working with latest version 3.2.0?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:API Airflow's REST/HTTP API area:task-sdk

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dynamic Task Mapping – map_index_template value is not rendered in the UI during running state Rendering custom map index before task is run.

4 participants