Task stuck in upstream_failed

[Tonkonozhenko]

Apache Airflow version

2.1.3 (latest released)

Operating System

Debian

Versions of Apache Airflow Providers

No response

Deployment

Other Docker-based deployment

Deployment details

No response

What happened

1. task prepare_timestamps failed with unknown k8s-related issue (most likely pod was killed)
2. task prepare_timestamps succeeded on the second try
3. Downstream task create_athena_partition stuck in upstream_failed state. Also, we experienced the same issue in 2.1.2 and then the task stuck in the queued state.

What you expected to happen

The task should not be stuck

How to reproduce

No response

Anything else

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[rpfernandezjr]

I just recently upgraded to 2.1.3, and I'm seeing the same thing.

on some cases, upstream tasks succeeds, but the downstream tasks are stuck in an upstream_failed state.

[ephraimbuddy]

Related: #16625
PR trying to address this: #17819

[ephraimbuddy]

Following the discussions at #17819 , you applied the fix then in your DAG above, prepare_timestamps failed and retried but create_athena_partition is stuck in up_for_retry?

[ephraimbuddy]

@Tonkonozhenko Do you have wait_for_downstream args set?

[WattsInABox]

We're also seeing this. Most of our DAGs have well under 100 tasks, a few just under 200 tasks, 673 active DAGs, 179 paused DAGs. We do not use wait_for_downstream anywhere.

We started seeing this after upgrading to 2.1.3 which we upgraded to specifically get the bug fix PR #16301, not sure if that bug might be related since we seem to be having weird status issues all over Airflow...

We see this in all manner of DAGs, some with a very linear path, some that branch into 100 tasks and then back to 1, others with 2 pre-requisite tasks into the final task.

Behavior:

• upstream tasks all successful
• downstream task(s) marked as upstream_failed
• sometimes an upstream task will have a previous run marked as failed but then it retries as successful, almost as if the downstream tasks get marked as upstream_failed on that run but then don't get cleared for the subsequent retry. But this does not always happen: we have seen multiple dagruns a night have upstream_failed tasks where all tasks prior worked on their first attempt (or at least only have logs for 1 attempt).

Please advise on what other information we can provide.

[ephraimbuddy]

@WattsInABox. If you can get scheduler logs when this happens, that would be very helpful.

[Tonkonozhenko]

@ephraimbuddy, @WattsInABox perfectly explained what happens. We have the completely same situation.

[ephraimbuddy]

@Tonkonozhenko @WattsInABox Do you see FATAL: sorry, too many clients already. in your scheduler logs when this happens?

If there’s reproducible step please share

[Tonkonozhenko]

@ephraimbuddy unfortunately, I don't have 2.1.3 logs now, but for 2.1.2 no such error and no fatal errors at all