Use is not None for GCSToS3Operator match_glob provision check - #71591
Open
DhanushAnegondi wants to merge 1 commit into
Open
Use is not None for GCSToS3Operator match_glob provision check#71591DhanushAnegondi wants to merge 1 commit into
DhanushAnegondi wants to merge 1 commit into
Conversation
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
|
DhanushAnegondi
force-pushed
the
use-is-not-none-for-match-glob-provision-check
branch
from
August 14, 2026 22:16
a9f0132 to
71c4728
Compare
match_glob is a template field, so its value is not available until Jinja rendering happens on the worker, well after __init__ has run. The guard that rejects match_glob on an unsupported google provider tested it for truthiness, which the validate-operators-init prek hook flags. The check only asks whether the argument was supplied; it never inspects the value. Per the false-positive guidance on apache#70296 that makes it a provision check, which is fixed in place rather than moved to execute(). Moving it would break the check under render_template_as_native_obj=True, where a supplied field can render to None and so becomes indistinguishable from an omitted one, and would defer a static authoring mistake from Dag parse time to every task instance and retry. Switching to an explicit is not None comparison changes one case: match_glob="" is now rejected on a google provider older than 10.3.0, where truthiness previously read the supplied empty string as absent. An empty glob is not a valid pattern and was never honoured on those versions. Narrow the raised exception from AirflowException to ValueError, which is what an invalid argument warrants, and drop the file from generated/known_airflow_exceptions.txt accordingly. This follows the merged apache#70359 precedent for S3DeleteObjectsOperator, which made the same one-for-one change to the same four kinds of file. Also remove the class from the prek exemption list, since the hook fails on stale entries, and add tests covering supplied, empty-string and omitted match_glob. Related: apache#70296
DhanushAnegondi
force-pushed
the
use-is-not-none-for-match-glob-provision-check
branch
from
August 14, 2026 22:30
71c4728 to
c1d5460
Compare
DhanushAnegondi
marked this pull request as ready for review
August 14, 2026 22:33
DhanushAnegondi
requested review from
amoghrajesh,
ashb,
bugraoz93,
gopidesupavan,
jason810496,
jscheffl,
o-nikolas and
potiuk
as code owners
August 14, 2026 22:33
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
match_globis inGCSToS3Operator.template_fields, so the constructor only ever sees the un-rendered Jinja expression, never the value the DAG author meant. The guard that rejectsmatch_globon an unsupported google provider tested it for truthiness:which is what
validate-operators-initflags, and why the class is on the exemption list.The check stays in
__init__. It is a provision check, not a value check: it asks whethermatch_globwas supplied, combined with an environment capability the constructor can already answer for itself. It never inspects the value. Per the false-positives section of #70296 those belong in the constructor and get rewritten in place rather than moved — withrender_template_as_native_obj=Truea provided field can render toNone, so the same check inexecute()would report a supplied argument as missing; and raising at construction surfaces a static authoring mistake as a Dag import error rather than once per task instance and per retry.Since #70505 narrowed the hook to sanction
is None/is not Nonereads, the in-place rewrite passes and the exemption entry still goes.What this changes
match_glob is not NoneGCSToS3Operatorfromscripts/ci/prek/validate_operators_init_exemptions.txt— the hook fails on stale exemptions, so this has to land in the same commit as the fixBehavioural impact
One cell changes, on google providers older than 10.3.0:
match_globNone)"**/*.csv"""match_glob=""was supplied by the user and truthiness read it as absent. An empty glob is not a valid pattern, and on a provider below 10.3.0 it was never going to be honoured anyway — it would have been passed to aGCSHook.list()call that cannot accept it. This is the stricter direction that05_pull_requests.rstsanctions for provision checks.Relationship to #70723
Flagging this up front: #70723 is open against the same entry and reaches the same one-line fix. I worked this independently and only found that PR afterwards, so this is not a deliberate competing implementation — I'm raising it because the two diffs are close enough that a reviewer deserves to know rather than discover it.
Where they differ:
AirflowExceptiontoValueErrorand drops the now-stalegcs_to_s3.py::1entry fromgenerated/known_airflow_exceptions.txt, per the Keep S3DeleteObjectsOperator validation in __init__, narrow to ValueError #70359 precedent. The production change is now equivalent to Check GCSToS3Operator match_glob support after template rendering #70723's.If maintainers prefer #70723 as the base — which is reasonable, it is older and more complete on the exception-type question — I am happy for this to be closed and will offer the missing test case there instead. I would rather that than have two near-identical PRs consuming review time.
Deliberately not changed
raise AirflowException— kept as-is, for the reason above.except ImportError.gcs_to_s3.pyalready does a module-levelfrom airflow.providers.google.cloud.hooks.gcs import GCSHook, so theexcept ImportErroraround the function-levelfrom airflow.providers.google import __version__in__init__cannot fire — the module import would have failed first. Real, but a separate concern.flatten_structure/keep_directory_structurewarning. Reads two fields that are not template fields, so the rule does not apply and the hook does not flag it.Verification
Run in a WSL checkout, from
providers/amazon:No newsfragment:
providers/amazonhas nonewsfragments/directory, andproviders/AGENTS.mdsays never to use them for providers.related: #70296
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5) following the guidelines