AirflowException: Celery command failed on host: [...]

[maxcountryman]

Apache Airflow version: 2.1.0

Kubernetes version (if you are using kubernetes) (use kubectl version): N/A (ECS)

Environment: Production

• Cloud provider or hardware configuration: AWS ECS Fargate
• OS (e.g. from /etc/os-release): Ubuntu
• Kernel (e.g. uname -a):
• Install tools: poetry
• Others:

What happened:

Since Airflow 2.x we've seen Celery-related failures with exceptions like this being the most prominent: AirflowException: Celery command failed on host: ip-10-0-10-110.us-west-2.compute.internal.

What you expected to happen:

No Celery command failures during normal operation.

How to reproduce it:

Deploy an Airflow cluster with a Celery + Redis executor to an ECS Fargate cluster.

Anything else we need to know:

How often does this problem occur? Once? Every time etc?

~30 times per hour.

[7yl4r]

I'm dealing with this too (but mine is 100% of the time on one system only).

After researching it looks like this is an old issue that seems to keep popping up after being "fixed".
These resources haven't fixed my problem but there are some ideas that might be relevant to your setup:

• some general ideas on astronomer forum. The most important seems to be insufficient RAM.
• thread that suggests hostname could be changing.

I suspect you are also seeing Airflow JIRA issue If a task crashes, hostname not committed to db so logs not in the UI which is a symptom of the Celery command failed on host? I am.

Do you have one worker failing consistenly among other workers suceesding or are your failures on a worker which is also succeeding sometimes?

[potiuk]

@maxcountryman (and possibly @7yl4r ) - it's next to impossible to help without more information.

I think what would be extremely helpful to investigate that one is to take a look at the logs of your worker. I think what would give more information is to setup container insights in your ECS Fargate cluster, including Firelens sending logs to CloudWatch. This way you could see more details for your failing instances - metrics, logs etc.

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/deploy-container-insights-ECS.html

Then - with more information we could probably help to pin-point the problems.

[maxcountryman]

@potiuk Container insights are already enabled so happy to go grab these and share.

Is there any other info that would be helpful?

[potiuk]

@potiuk Container insights are already enabled so happy to go grab these and share.

Is there any other info that would be helpful?

Not really - I think if you look at logs and metrics of the failing containers, there should be some warning/error messages or other indicators (memory usage spiking ?) of the root cause of the problem. But there might be many different reasons so it's hard to tell what exactly you should look for.

[maxcountryman]

@potiuk Container Insight metrics look well below any resource starvation, looking back several weeks.

I'm not seeing anything too obvious in the logs, but will continue looking.

[maxcountryman]

@potiuk An important correction: the containers are not failing. It's the Celery commands that seem to be falling over for some unknown reason (some messages claim memory issues, but we don't see that in the metrics as I mentioned).

[potiuk]

Yeah. I understand that - but I would expect to have some logs generated by the celery failures. They might be either in the logs of airflow - explaining what happened, or in case the celery commands failed at some "resource exhaustion" (memory, disk, sockets .... you name it) celery might simply not be able to print out the reasons (often because of this resource exhaution) that will be seen in airflow logs - but then there might be some indication of problems in container logs or metrics. You do mention some memory issues - which is most likely. I wonder what indication of memory problems you mention?

Unfortunately. sometimes you will not see the resource starvation in the container. For example If a process requests big chunk of memory that the container does not have, you will not see the starvation at all - you will simply see the process failing. But then maybe you will see some indication of that in some of the container logs (like information that memory request could not be satisfied or something like that). And memory is just one - most likely - reason.

[maxcountryman]

@potiuk Yes, understood--unfortunately there's very little additional context to share. There are three errors which all seem to be related to each other. One mentions the Celery task being killed, i.e. SIGTERM, and says it may be due to memory. That's all.

Even more strange is most of the time things work just fine. We see a spike in these errors around deployment and then see a consistent number of them during normal execution. Actual Airflow tasks run as expected, seemingly unaffected (altho I suspect there's some thrashing behind the scenes that slows things down unnecessarily when the Celery commands fail in this way).

I'll look through the logs more, but I suspect anything we see there won't be much different that the errors Sentry is surfacing.

[potiuk]

Can you copy the SIGTERM message ?.

[El-Carverino]

We have the same (or similar) setup, using Airflow on AWS ECS Fargate containers, and I had a very similar ambiguous issue, with the webserver container just exiting/crashing, without really providing any useful info in the logs. Specifically, it started happening after upgrading from Airflow v1.13 (where this setup worked fine) to Airflow v2.1. And the logs seemed to suggest that it was related to the webserver's gunicorn worker processes (somehow).

I discovered that the problem was directly tied to the resources allocated to the container(s), along with the Airflow configuration values for these worker processes. In our case, the webserver was on a relatively small container and ECS task (1GB memory, 0.25 vCPU), because we try to run things pretty lean (you know, $$). It's now been upped to 0.5 vCPU, but the same problem persisted, until I found this formula from someone else on the internet. The # desired workers should correlate to the CPU cores of the instance/container: 2 * num_cpu_cores + 1 (Airflow's default is 4).

Since this Fargate task/container isn't even allocated a single CPU, I updated this value AIRFLOW__WEBSERVER__WORKERS=2, and now the webserver container is stable! While this may not be the exact same problem, it might be worth increasing the resources you have allocated to the container, just to see if it's related, or maybe see if there are similar Airflow configuration settings that might be the culprit.

[github-actions]

This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.