Apache Airflow version
main (development)
What happened
The provider operator for Google Cloud Dataproc Batches has two bugs:
- The running operator returns successful even if the job transitions to State.CANCELLED or State.CANCELLING
- It attempts to 'reattach' to a potentially running job if it AlreadyExists, but it sends the wrong type since 'result' is a Batch and needs Operation
What you think should happen instead
A new hook that polls for batch job completion. There is precedent for it in traditional dataproc with 'wait_for_job'.
How to reproduce
Use the Breeze environment and a DAG that runs DataprocCreateBatchOperator. Allow the first instance to start.
Use the gcloud CLI to cancel the job.
gcloud dataproc batches cancel <batch_id> --project <project_id> --region <region>
Observe that the task completes successfully after a 3-5 minute timeout, even though the job was cancelled.
Run the task again with the same batch_id. Observe the ValueError where it expects Operation but receives Batch
Operating System
Darwin 5806 21.6.0 Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64 x86_64
Versions of Apache Airflow Providers
Same as dev (main) version.
Deployment
Other Docker-based deployment
Deployment details
Observable in the Breeze environment, when running against real Google Infrastructure.
Anything else
Every time.
Are you willing to submit PR?
Code of Conduct
Apache Airflow version
main (development)
What happened
The provider operator for Google Cloud Dataproc Batches has two bugs:
What you think should happen instead
A new hook that polls for batch job completion. There is precedent for it in traditional dataproc with 'wait_for_job'.
How to reproduce
Use the Breeze environment and a DAG that runs DataprocCreateBatchOperator. Allow the first instance to start.
Use the gcloud CLI to cancel the job.
gcloud dataproc batches cancel <batch_id> --project <project_id> --region <region>Observe that the task completes successfully after a 3-5 minute timeout, even though the job was cancelled.
Run the task again with the same batch_id. Observe the ValueError where it expects Operation but receives Batch
Operating System
Darwin 5806 21.6.0 Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64 x86_64
Versions of Apache Airflow Providers
Same as dev (main) version.
Deployment
Other Docker-based deployment
Deployment details
Observable in the Breeze environment, when running against real Google Infrastructure.
Anything else
Every time.
Are you willing to submit PR?
Code of Conduct