Same task instance attempt is detected as a zombie multiple times and fails without retries

[wolfier]

Apache Airflow version

Other Airflow 2 version (please specify below)

If "Other Airflow 2 version" selected, which one?

2.9.3

What happened?

A task instance's LocalTaskJobRunner exited without addressing the task instance state which is expected. The task instance will be identified as a zombie.

The issue is that the task instance is identified as a zombie three times and each time a TaskCallbackRequest is created. The reason why three callback requests were sent is because the DagFileProcessorProcess did not parse the file in a timely manner. There were about 600 seconds between each parse which allowed the zombie detection operation to find the same task instance multiple times.

Eventually when the the source file is parsed, the DagFileProcessorProcess executed all the TaskCallbackRequests and moved the task from running to up_to_retry then to failed.

This was previously reported in #31212

What you think should happen instead?

Ideally, each task instance attempt should only be identified once regardless of how it is done.

I think the following options are viable:

• deduplicating the TaskCallbackRequest / DbCallbackRequest
• tie TaskCallbackRequest to the task instance key and check before it is sent to the DatabaseCallbackSink

How to reproduce

1. Increase min_file_process_interval to 600 seconds
2. Create zombies by causing the LocalTaskJobRunner to terminate (delete the PgBouncer)
3. Confirm the same task instance attempt is identified as zombies between source file parsings
4. Confirm the DagFileProcessorProcess parses the file and that the on_failure_callback runs multiple times

Operating System

n/a

Versions of Apache Airflow Providers

No response

Deployment

Astronomer

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[github-actions]

This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.