Skip to content

Mask list-shaped Variable values on deserialization - #70891

Merged
vatsrahul1001 merged 2 commits into
apache:mainfrom
potiuk:mask-list-shaped-variable-values
Aug 4, 2026
Merged

Mask list-shaped Variable values on deserialization#70891
vatsrahul1001 merged 2 commits into
apache:mainfrom
potiuk:mask-list-shaped-variable-values

Conversation

@potiuk

@potiuk potiuk commented Aug 1, 2026

Copy link
Copy Markdown
Member

_mask_and_deserialize_variable dispatched on the top-level type of the
deserialized value and handled only str and dict. A Variable whose JSON is a
list therefore matched no branch and was returned with nothing inside it masked.

add_mask walks dicts and iterables itself, so the same list nested one level
inside a dict was already masked -- the top-level dispatch was the only thing
deciding whether the nested values were reached at all.

Approach

Add the list branch, passing the value under the variable's key rather
than anonymously.

That distinction is the whole design of the change. Elements of a list have no
key names of their own, so they follow the variable key's sensitivity -- the same
rule already applied to a plain string value. Passing the list anonymously would
mask every element unconditionally and add each to the global pattern set, so a
Variable holding ["us-east-1", "eu-west-1"] would cause us-east-1 to be
redacted everywhere it appeared in any log. test_var_json_list_value_does_not_over_mask
already guarded that, and this keeps it true. Dicts inside the list are still
masked by their own key names.

Verified against the real masker:

variable key result
[{"password": "…"}] db_configs [{"password": "***"}]
["us-east-1", "eu-west-1"] aws_regions unchanged
["…"] my_password ***
{"password": "…"} db_config {"password": "***"} (unchanged path)

test_var_json_list_value_does_not_over_mask is updated: it asserted
mask_secret was called exactly once, and there are now two calls. The property
it guards is unchanged and the assertion is now stated directly -- the list is
passed under the variable's key, and never anonymously.

Test plan

  • test_var_json_masks_list_values -- list handed to the masker under the
    variable's key
  • test_var_json_list_value_does_not_over_mask -- updated to assert the
    no-over-masking property rather than the call count
  • test_var_json_scalar_values_pass_through -- int/bool/null/float returned
    unchanged
  • Verified against unmodified code: both list tests fail
  • test_context.py -- 170 passed
  • ruff check / ruff format clean
Was generative AI tooling used to co-author this PR?
  • Yes — Claude Opus 5 (1M context)

Generated-by: Claude Opus 5 (1M context) following the guidelines at
https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions

@potiuk potiuk added this to the Airflow 3.3.1 milestone Aug 1, 2026
@potiuk potiuk added the backport-to-v3-3-test Backport to v3-3-test label Aug 1, 2026

@SameerMesiah97 SameerMesiah97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Just a few stylistic nits (for e.g. I found one comment a bit hard to understand).

Comment thread task-sdk/src/airflow/sdk/execution_time/context.py Outdated
Comment thread task-sdk/tests/task_sdk/execution_time/test_context.py Outdated
Comment thread task-sdk/tests/task_sdk/execution_time/test_context.py Outdated
Comment thread task-sdk/tests/task_sdk/execution_time/test_context.py Outdated
Comment thread task-sdk/src/airflow/sdk/execution_time/context.py Outdated
Comment thread task-sdk/tests/task_sdk/execution_time/test_context.py Outdated
Comment thread task-sdk/src/airflow/sdk/execution_time/context.py Outdated
potiuk added 2 commits August 4, 2026 01:14
_mask_and_deserialize_variable dispatched on the top-level type of the
deserialized value and handled only str and dict, so a Variable whose JSON is a
list was returned with no masking applied to anything inside it. add_mask walks
iterables itself, so the same list nested one level inside a dict was already
masked -- only a top-level one was skipped.

The list is passed under the variable's key rather than anonymously: elements
have no key names of their own, so they follow the variable key's sensitivity,
and a list of ordinary values such as region names is not added to the global
pattern set. Dicts inside the list are still masked by their own key names.
The rationale was spread across three sites saying the same thing; keep it at
the branch where the decision is made.
@potiuk
potiuk force-pushed the mask-list-shaped-variable-values branch from 07d2d16 to c7e188e Compare August 3, 2026 23:16
@potiuk

potiuk commented Aug 4, 2026

Copy link
Copy Markdown
Member Author

All threads addressed and resolved. Comments trimmed as suggested.

I did not take the branch collapse — the equivalence relies on add_mask ignoring the name it is handed for a dict, so the key would be passed and silently ignored, and the dict branch is pre-existing behaviour I would rather not rewrite inside a 3.3.1 backport. Reasoning is in that thread if you disagree.

@amoghrajesh @SameerMesiah97


Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

@amoghrajesh amoghrajesh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, simple enough now.

@vatsrahul1001 vatsrahul1001 added the type:misc/internal Changelog: Misc changes that should appear in change log label Aug 4, 2026
@vatsrahul1001
vatsrahul1001 merged commit b968192 into apache:main Aug 4, 2026
107 checks passed
@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Backport successfully created: v3-3-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status Branch Result
v3-3-test PR Link

github-actions Bot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request Aug 4, 2026
…he#70891)

* Mask list-shaped Variable values on deserialization

_mask_and_deserialize_variable dispatched on the top-level type of the
deserialized value and handled only str and dict, so a Variable whose JSON is a
list was returned with no masking applied to anything inside it. add_mask walks
iterables itself, so the same list nested one level inside a dict was already
masked -- only a top-level one was skipped.

The list is passed under the variable's key rather than anonymously: elements
have no key names of their own, so they follow the variable key's sensitivity,
and a list of ordinary values such as region names is not added to the global
pattern set. Dicts inside the list are still masked by their own key names.

* Tighten the comments around Variable masking dispatch

The rationale was spread across three sites saying the same thing; keep it at
the branch where the decision is made.
(cherry picked from commit b968192)

Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
aws-airflow-bot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request Aug 4, 2026
…he#70891)

* Mask list-shaped Variable values on deserialization

_mask_and_deserialize_variable dispatched on the top-level type of the
deserialized value and handled only str and dict, so a Variable whose JSON is a
list was returned with no masking applied to anything inside it. add_mask walks
iterables itself, so the same list nested one level inside a dict was already
masked -- only a top-level one was skipped.

The list is passed under the variable's key rather than anonymously: elements
have no key names of their own, so they follow the variable key's sensitivity,
and a list of ordinary values such as region names is not added to the global
pattern set. Dicts inside the list are still masked by their own key names.

* Tighten the comments around Variable masking dispatch

The rationale was spread across three sites saying the same thing; keep it at
the branch where the decision is made.
(cherry picked from commit b968192)

Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
vatsrahul1001 pushed a commit that referenced this pull request Aug 4, 2026
…) (#71069)

* Mask list-shaped Variable values on deserialization

_mask_and_deserialize_variable dispatched on the top-level type of the
deserialized value and handled only str and dict, so a Variable whose JSON is a
list was returned with no masking applied to anything inside it. add_mask walks
iterables itself, so the same list nested one level inside a dict was already
masked -- only a top-level one was skipped.

The list is passed under the variable's key rather than anonymously: elements
have no key names of their own, so they follow the variable key's sensitivity,
and a list of ordinary values such as region names is not added to the global
pattern set. Dicts inside the list are still masked by their own key names.

* Tighten the comments around Variable masking dispatch

The rationale was spread across three sites saying the same thing; keep it at
the branch where the decision is made.
(cherry picked from commit b968192)

Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
vatsrahul1001 pushed a commit that referenced this pull request Aug 5, 2026
…) (#71069)

* Mask list-shaped Variable values on deserialization

_mask_and_deserialize_variable dispatched on the top-level type of the
deserialized value and handled only str and dict, so a Variable whose JSON is a
list was returned with no masking applied to anything inside it. add_mask walks
iterables itself, so the same list nested one level inside a dict was already
masked -- only a top-level one was skipped.

The list is passed under the variable's key rather than anonymously: elements
have no key names of their own, so they follow the variable key's sensitivity,
and a list of ordinary values such as region names is not added to the global
pattern set. Dicts inside the list are still masked by their own key names.

* Tighten the comments around Variable masking dispatch

The rationale was spread across three sites saying the same thing; keep it at
the branch where the decision is made.
(cherry picked from commit b968192)

Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
dabla pushed a commit to dabla/airflow that referenced this pull request Aug 14, 2026
* Mask list-shaped Variable values on deserialization

_mask_and_deserialize_variable dispatched on the top-level type of the
deserialized value and handled only str and dict, so a Variable whose JSON is a
list was returned with no masking applied to anything inside it. add_mask walks
iterables itself, so the same list nested one level inside a dict was already
masked -- only a top-level one was skipped.

The list is passed under the variable's key rather than anonymously: elements
have no key names of their own, so they follow the variable key's sensitivity,
and a list of ordinary values such as region names is not added to the global
pattern set. Dicts inside the list are still masked by their own key names.

* Tighten the comments around Variable masking dispatch

The rationale was spread across three sites saying the same thing; keep it at
the branch where the decision is made.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:task-sdk backport-to-v3-3-test Backport to v3-3-test type:misc/internal Changelog: Misc changes that should appear in change log

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants