Skip to content

Fix MetastoreHivePartitionSensor failing due to duplicate aliases - #45001

Merged
potiuk merged 3 commits into
apache:mainfrom
CYarros10:metastore-sensor-fix
Dec 18, 2024
Merged

Fix MetastoreHivePartitionSensor failing due to duplicate aliases#45001
potiuk merged 3 commits into
apache:mainfrom
CYarros10:metastore-sensor-fix

Conversation

@CYarros10

@CYarros10 CYarros10 commented Dec 17, 2024

Copy link
Copy Markdown
Contributor

MetastoreHivePartitionSensor is not working.

The query it uses to list partitions

query = f"""
                SELECT *
                FROM PARTITIONS
                INNER JOIN TBLS
                ON PARTITIONS.TBL_ID = TBLS.TBL_ID
                WHERE
                    TBLS.TBL_NAME = '{table}'"""

fails, due to Duplicate Aliases (https://cloud.google.com/spanner/docs/reference/standard-sql/query-syntax#duplicate_aliases)

This query fixes the issue:

query = f"""
            SELECT PARTITIONS.*, TBLS.TBL_TYPE, TBLS.TBL_NAME
            FROM PARTITIONS
            INNER JOIN TBLS ON PARTITIONS.TBL_ID = TBLS.TBL_ID
            WHERE TBLS.TBL_NAME = '{table}'"""

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg boring-cyborg Bot added area:providers provider:google Google (including GCP) related issues labels Dec 17, 2024
@potiuk
potiuk merged commit 553ddf3 into apache:main Dec 18, 2024
got686-yandex pushed a commit to got686-yandex/airflow that referenced this pull request Jan 30, 2025
…ache#45001)

* fix MetastoreHivePartitionSensor with updated query to avoid duplicate aliases

* Update TestMetastoreHivePartitionSensor

* Fix MetastoreHivePartitionSensor pytests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants