Apache Airflow Provider(s)
docker
Versions of Apache Airflow Providers
apache-airflow-providers-celery==3.1.0
apache-airflow-providers-docker==3.3.0
Apache Airflow version
2.5.0
Operating System
centos 7
Deployment
Other Docker-based deployment
Deployment details
Running an a docker-swarm cluster deployed locally.
What happened
Same issue as #13675
With logging_enabled=True the DAG never completes and stays in running.
When using DockerSwarmOperator together with the default enable_logging=True option, tasks do not succeed and stay in state running. When checking the docker service logs I can clearly see that the container ran and ended successfully. Airflow however does not recognize that the container finished and keeps the tasks in state running.
What you think should happen instead
DAG should complete.
How to reproduce
Docker-compose deployment:
curl -LfO 'https://airflow.apache.org/docs/apache-airflow/2.5.0/docker-compose.yaml'
docker compose up airflow-init
docker compose up -d
DAG code:
from airflow import DAG
from docker.types import Mount, SecretReference
from airflow.providers.docker.operators.docker_swarm import DockerSwarmOperator
from datetime import timedelta
from airflow.utils.dates import days_ago
from airflow.models import Variable
# Setup default args for the job
default_args = {
'owner': 'airflow',
'start_date': days_ago(2),
'retries': 0
}
# Create the DAG
dag = DAG(
'test_dag', # DAG ID
default_args=default_args,
schedule_interval='0 0 * * *',
catchup=False
)
# # Create the DAG object
with dag as dag:
docker_swarm_task = DockerSwarmOperator(
task_id="job_run",
image="<any image>",
execution_timeout=timedelta(minutes=5),
command="<specific code>",
api_version='auto',
tty=True,
enable_logging=True
)Anything else
No response
Are you willing to submit PR?
Code of Conduct
Apache Airflow Provider(s)
docker
Versions of Apache Airflow Providers
apache-airflow-providers-celery==3.1.0
apache-airflow-providers-docker==3.3.0
Apache Airflow version
2.5.0
Operating System
centos 7
Deployment
Other Docker-based deployment
Deployment details
Running an a docker-swarm cluster deployed locally.
What happened
Same issue as #13675
With logging_enabled=True the DAG never completes and stays in running.
When using DockerSwarmOperator together with the default enable_logging=True option, tasks do not succeed and stay in state running. When checking the docker service logs I can clearly see that the container ran and ended successfully. Airflow however does not recognize that the container finished and keeps the tasks in state running.
What you think should happen instead
DAG should complete.
How to reproduce
Docker-compose deployment:
DAG code:
Anything else
No response
Are you willing to submit PR?
Code of Conduct