Azure Batch hook "wait_for_job_tasks_to_complete" method does not return failed tasks

[ArvidMartensRenson]

Apache Airflow Provider(s)

microsoft-azure

Versions of Apache Airflow Providers

apache-airflow-providers-microsoft-azure==8.4.0

Apache Airflow version

2.8.0

Operating System

Ubuntu 22.04

Deployment

Other Docker-based deployment

Deployment details

No response

What happened

I have a task that has failed in azure batch account that returns as successful in the airflow overview. When running this code locally I noticed that the wait_for_job_tasks_to_complete method of the azure batch hook returned no failed batch tasks, hence the airflow task is not set to failed. When using the the azure SDK directly on this task it returns as failed. As such I believe the somewhere in the hook the information on the batch task is lost.

What you think should happen instead

No response

How to reproduce

Create a task in azure batch that fails and wait for it with the airflow batch hook (wait_for_job_tasks_to_complete). You will notice that the this method suggests that the task has succeeded.

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[dirrao]

Can you share the sample DAG with screenshots of DAG run and task logs?

[eladkal]

Fixed in #36785