Fix location on cloud build operators by tnk-ysk · Pull Request #29937 · apache/airflow

tnk-ysk · 2023-03-06T13:29:39Z

Fixed an issue where cloud build operators did not work for private pools or a non-global default pools.
In this case the CloudBuildClient must set the location with api_endpoint.

Document

    # If you're using Private Pools or a non-global default pool, add a regional
    # `api_endpoint` to `CloudBuildClient()`
    # For example, '<YOUR_POOL_REGION>-cloudbuild.googleapis.com'
    #
    # from google.api_core import client_options
    # client_options = client_options.ClientOptions(
    #     api_endpoint="us-central1-cloudbuild.googleapis.com"
    # )
    # client = cloudbuild_v1.services.cloud_build.CloudBuildClient(client_options=client_options)

Related #29689

Target operators

CloudBuildCancelBuildOperator
CloudBuildGetBuildOperator
CloudBuildListBuildsOperator
CloudBuildRetryBuildOperator
CloudBuildListBuildTriggersOperator
CloudBuildCreateBuildOperator
CloudBuildCreateBuildTriggerOperator
CloudBuildDeleteBuildTriggerOperator
CloudBuildGetBuildTriggerOperator
CloudBuildRunBuildTriggerOperator
CloudBuildUpdateBuildTriggerOperator
CloudBuildDeleteBuildTriggerOperator

Reference

https://cloud.google.com/build/docs/private-pools/private-pools-overview

potiuk · 2023-03-07T21:32:46Z

LGTM. @VladaZakharova - WDYT?

potiuk · 2023-03-07T22:09:35Z

tests needs fixing though

tnk-ysk · 2023-03-08T01:53:47Z

@potiuk

tests needs fixing though

I fixed it.
23cd526

VladaZakharova · 2023-03-08T08:33:06Z

Hi!
Did you check if, for example, BigQuery service is working okay with this changes without providing location to the operator?

tnk-ysk · 2023-03-08T14:06:46Z

@VladaZakharova
I checked and fixed some issues.

Fixed some bugs.
Other CloudBuildTriggerOperator had a same issue, fixed.
Fixed CloudBuild links.

Test dag

from __future__ import print_function

import datetime
import os

from airflow import models
from airflow.providers.google.cloud.operators import cloud_build
from airflow.providers.google.cloud.utils import credentials_provider

PROJECT_ID = "xxxxxxx"
REPO_NAME = "sample"

gcp_conn = credentials_provider.build_gcp_conn(
    "/files/key.json",
    ["https://www.googleapis.com/auth/cloud-platform"],
    PROJECT_ID,
)
os.environ["AIRFLOW_CONN_GOOGLE_CLOUD_DEFAULT"] = gcp_conn

default_dag_args = {
    'start_date': datetime.datetime(2022, 9, 28),
}

with models.DAG(
    'CloudBuildTest',
    schedule_interval=None,
    default_args=default_dag_args,
    catchup=False
) as dag:
    build = {
        "steps": [{
            "name": "alpine",
            "script": "sleep 10 && echo hello",
        }],
    }
    trigger = {
        "name": "trigger",
        "trigger_template": {
            "repo_name": REPO_NAME,
            "branch_name": "main",
        },
        "filename": "cloudbuild.yaml",
    }

    # global
    create_build_global = cloud_build.CloudBuildCreateBuildOperator(
        task_id="create_build_global",
        project_id=PROJECT_ID,
        build=build,
    )
    create_build_global_deferrable = cloud_build.CloudBuildCreateBuildOperator(
        task_id="create_build_global_deferrable",
        project_id=PROJECT_ID,
        build=build,
        deferrable=True,
    )
    list_build_global = cloud_build.CloudBuildListBuildsOperator(
        task_id="list_build_global",
        project_id=PROJECT_ID,
        page_size=1,
    )
    retry_build_global = cloud_build.CloudBuildRetryBuildOperator(
        task_id="retry_build_global",
        project_id=PROJECT_ID,
        id_='{{ ti.xcom_pull(task_ids="create_build_global", key="id") }}',
    )
    get_build_global = cloud_build.CloudBuildGetBuildOperator(
        task_id="get_build_global",
        project_id=PROJECT_ID,
        id_='{{ ti.xcom_pull(task_ids="retry_build_global", key="id") }}',
    )
    list_build_triggers_global = cloud_build.CloudBuildListBuildTriggersOperator(
        task_id="list_build_triggers_global",
        project_id=PROJECT_ID,
    )
    create_build_global_for_cancel = cloud_build.CloudBuildCreateBuildOperator(
        task_id="create_build_global_for_cancel",
        project_id=PROJECT_ID,
        build=build,
        wait=False,
    )
    cancel_build_global = cloud_build.CloudBuildCancelBuildOperator(
        task_id="cancel_build_global",
        project_id=PROJECT_ID,
        id_='{{ ti.xcom_pull(task_ids="create_build_global_for_cancel", key="id") }}',
    )
    create_build_trigger_global = cloud_build.CloudBuildCreateBuildTriggerOperator(
        task_id="create_build_trigger_global",
        project_id=PROJECT_ID,
        trigger=trigger,
    )
    get_build_trigger_global = cloud_build.CloudBuildGetBuildTriggerOperator(
        task_id="get_build_trigger_global",
        project_id=PROJECT_ID,
        trigger_id='{{ ti.xcom_pull(task_ids="create_build_trigger_global", key="id") }}',
    )
    update_build_trigger_global = cloud_build.CloudBuildUpdateBuildTriggerOperator(
        task_id="update_build_trigger_global",
        project_id=PROJECT_ID,
        trigger_id='{{ ti.xcom_pull(task_ids="create_build_trigger_global", key="id") }}',
        trigger=trigger,
    )
    run_build_trigger_global = cloud_build.CloudBuildRunBuildTriggerOperator(
        task_id="run_build_trigger_global",
        project_id=PROJECT_ID,
        trigger_id='{{ ti.xcom_pull(task_ids="create_build_trigger_global", key="id") }}',
        source={
            "branch_name": "main"
        }
    )
    delete_build_trigger_global = cloud_build.CloudBuildDeleteBuildTriggerOperator(
        task_id="delete_build_trigger_global",
        project_id=PROJECT_ID,
        trigger_id='{{ ti.xcom_pull(task_ids="create_build_trigger_global", key="id") }}',
    )

    # us-centrarl
    create_build_us_central1 = cloud_build.CloudBuildCreateBuildOperator(
        task_id="create_build_us_central1",
        project_id=PROJECT_ID,
        build=build,
        location="us-central1",
    )
    create_build_us_central1_deferrable = cloud_build.CloudBuildCreateBuildOperator(
        task_id="create_build_us_central1_deferrable",
        project_id=PROJECT_ID,
        build=build,
        deferrable=True,
        location="us-central1",
    )
    list_build_us_central1 = cloud_build.CloudBuildListBuildsOperator(
        task_id="list_build_us_central1",
        project_id=PROJECT_ID,
        page_size=1,
        location="us-central1",
    )
    retry_build_us_central1 = cloud_build.CloudBuildRetryBuildOperator(
        task_id="retry_build_us_central1",
        project_id=PROJECT_ID,
        id_='{{ ti.xcom_pull(task_ids="create_build_us_central1", key="id") }}',
        location="us-central1",
    )
    get_build_us_central1 = cloud_build.CloudBuildGetBuildOperator(
        task_id="get_build_us_central1",
        project_id=PROJECT_ID,
        id_='{{ ti.xcom_pull(task_ids="retry_build_us_central1", key="id") }}',
        location="us-central1",
    )
    list_build_triggers_us_central1 = cloud_build.CloudBuildListBuildTriggersOperator(
        task_id="list_build_triggers_us_central1",
        project_id=PROJECT_ID,
        location="us-central1",
    )
    create_build_us_central1_for_cancel = cloud_build.CloudBuildCreateBuildOperator(
        task_id="create_build_us_central1_for_cancel",
        project_id=PROJECT_ID,
        build=build,
        wait=False,
        location="us-central1",
    )
    cancel_build_us_central1 = cloud_build.CloudBuildCancelBuildOperator(
        task_id="cancel_build_us_central1",
        project_id=PROJECT_ID,
        id_='{{ ti.xcom_pull(task_ids="create_build_us_central1_for_cancel", key="id") }}',
        location="us-central1",
    )
    create_build_trigger_us_central1 = cloud_build.CloudBuildCreateBuildTriggerOperator(
        task_id="create_build_trigger_us_central1",
        project_id=PROJECT_ID,
        trigger=trigger,
        location="us-central1",
    )
    get_build_trigger_us_central1 = cloud_build.CloudBuildGetBuildTriggerOperator(
        task_id="get_build_trigger_us_central1",
        project_id=PROJECT_ID,
        trigger_id='{{ ti.xcom_pull(task_ids="create_build_trigger_us_central1", key="id") }}',
        location="us-central1",
    )
    update_build_trigger_us_central1 = cloud_build.CloudBuildUpdateBuildTriggerOperator(
        task_id="update_build_trigger_us_central1",
        project_id=PROJECT_ID,
        trigger_id='{{ ti.xcom_pull(task_ids="create_build_trigger_us_central1", key="id") }}',
        trigger=trigger,
        location="us-central1",
    )
    run_build_trigger_us_central1 = cloud_build.CloudBuildRunBuildTriggerOperator(
        task_id="run_build_trigger_us_central1",
        project_id=PROJECT_ID,
        trigger_id='{{ ti.xcom_pull(task_ids="create_build_trigger_us_central1", key="id") }}',
        source={
            "branch_name": "main"
        },
        location="us-central1",
    )
    delete_build_trigger_us_central1 = cloud_build.CloudBuildDeleteBuildTriggerOperator(
        task_id="delete_build_trigger_us_central1",
        project_id=PROJECT_ID,
        trigger_id='{{ ti.xcom_pull(task_ids="create_build_trigger_us_central1", key="id") }}',
        location="us-central1",
    )


[
        # global
        create_build_global >> retry_build_global >> get_build_global,
        create_build_global_deferrable,
        list_build_global >> list_build_triggers_global,
        create_build_global_for_cancel >> cancel_build_global,
        create_build_trigger_global >> get_build_trigger_global >> update_build_trigger_global
        >> run_build_trigger_global >> delete_build_trigger_global,

        # us-central1
        create_build_us_central1 >> retry_build_us_central1 >> get_build_us_central1,
        create_build_us_central1_deferrable,
        list_build_us_central1 >> list_build_triggers_us_central1,
        create_build_us_central1_for_cancel >> cancel_build_us_central1,
        create_build_trigger_us_central1 >> get_build_trigger_us_central1 >> update_build_trigger_us_central1
        >> run_build_trigger_us_central1 >> delete_build_trigger_us_central1,
    ]

VladaZakharova · 2023-03-08T14:09:34Z

Cool! Thank you for the fix :)
@potiuk
LGTM

boring-cyborg · 2023-03-10T12:09:47Z

Awesome work, congrats on your first merged pull request!

potiuk · 2023-03-10T12:09:55Z

Cool. Thank @VladaZakharova !

potiuk · 2023-03-10T12:10:11Z

And @tnk-ysk of course :)

Yusuke Tanaka and others added 7 commits March 2, 2023 20:43

Add parent to request in create build

d961e93

Fixed assertion on create_build_without_waiting_for_result

d314de2

Fix doc on create_build_without_waiting_for_result

1a10aa0

Merge branch 'main' into add-parent-to-request-in-create-build

9479802

Fix location on cloud build client

8a0b539

Add location to template_fields

b265a75

Fixed location on cloud build

838c58f

boring-cyborg Bot added area:providers provider:google Google (including GCP) related issues labels Mar 6, 2023

tnk-ysk changed the title ~~Fixed location on cloud build operators~~ Fix location on cloud build operators Mar 6, 2023

Yusuke Tanaka and others added 3 commits March 7, 2023 00:37

Fix CloudBuildListBuildTriggersOperator

fbae5bb

Fix deferrable

0f273c2

Merge branch 'main' into fix-location-on-cloud-build

526be4d

tnk-ysk mentioned this pull request Mar 6, 2023

Add location to CloudBuildCreateBuildOperator #29689

Closed

tnk-ysk and others added 3 commits March 7, 2023 17:12

Merge branch 'main' into fix-location-on-cloud-build

b54d294

Fix code format

ca3ae36

Add port to api_endpoint

8232278

potiuk approved these changes Mar 7, 2023

View reviewed changes

Fix test_cloud_build_service_client_creation

23cd526

tnk-ysk force-pushed the fix-location-on-cloud-build branch from b3758ab to 23cd526 Compare March 8, 2023 01:47

Yusuke Tanaka added 4 commits March 8, 2023 21:36

Fix bugs

a0e1b37

Fix location on CloudBuildTriggerOperations

f22e9d7

Fix CloudBuild links

35f1091

Fix run_build_trigger

d7d396f

Yusuke Tanaka and others added 3 commits March 8, 2023 23:34

Add location to template_fields

cdb88d3

Fix format

5e963e5

Merge branch 'main' into fix-location-on-cloud-build

8d1aac9

potiuk merged commit def1f89 into apache:main Mar 10, 2023

tnk-ysk deleted the fix-location-on-cloud-build branch March 10, 2023 13:13

eladkal mentioned this pull request Apr 2, 2023

Status of testing Providers that were prepared on April 02, 2023 #30427

Closed

64 tasks

tnk-ysk mentioned this pull request Apr 3, 2023

Fix cloud build async credentials #30441

Merged

potiuk mentioned this pull request May 13, 2023

unable to specify location in arguments to CloudBuildRunBuildTriggerOperator #31272

Closed

2 tasks

Navigation Menu

Uh oh!

Fix location on cloud build operators - #29937

Fix location on cloud build operators#29937
potiuk merged 21 commits into
apache:mainfrom
tnk-ysk:fix-location-on-cloud-build

tnk-ysk commented Mar 6, 2023 •

edited

Loading

Uh oh!

potiuk commented Mar 7, 2023

Uh oh!

potiuk commented Mar 7, 2023

Uh oh!

tnk-ysk commented Mar 8, 2023

Uh oh!

VladaZakharova commented Mar 8, 2023

Uh oh!

tnk-ysk commented Mar 8, 2023 •

edited

Loading

Uh oh!

VladaZakharova commented Mar 8, 2023 •

edited

Loading

Uh oh!

boring-cyborg Bot commented Mar 10, 2023

Uh oh!

potiuk commented Mar 10, 2023

Uh oh!

potiuk commented Mar 10, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

tnk-ysk commented Mar 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reference

Uh oh!

potiuk commented Mar 7, 2023

Uh oh!

potiuk commented Mar 7, 2023

Uh oh!

tnk-ysk commented Mar 8, 2023

Uh oh!

VladaZakharova commented Mar 8, 2023

Uh oh!

tnk-ysk commented Mar 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

VladaZakharova commented Mar 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

boring-cyborg Bot commented Mar 10, 2023

Uh oh!

potiuk commented Mar 10, 2023

Uh oh!

potiuk commented Mar 10, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tnk-ysk commented Mar 6, 2023 •

edited

Loading

tnk-ysk commented Mar 8, 2023 •

edited

Loading

VladaZakharova commented Mar 8, 2023 •

edited

Loading