Description
Dags folder in celery workers can be kept at different location as kept in Airflow scheduler node. Airflow task run can use dag full file path based on airflow home configuration on their node given dag file relative path to dags folder.
Use case / motivation
Currently scheduler uses absolute path for dags file to schedule task in task run command (-sd arg). This command makes hard to keep dags folder at different locations in celery workers which may be running in different environment. So using Airflow with celery with cross environment setup is difficult.
This feature will help to keep dags folder at any location at worker nodes and tasks command can be configured for dag file path using home configuration.
Description
Dags folder in celery workers can be kept at different location as kept in Airflow scheduler node. Airflow task run can use dag full file path based on airflow home configuration on their node given dag file relative path to dags folder.
Use case / motivation
Currently scheduler uses absolute path for dags file to schedule task in task run command (
-sdarg). This command makes hard to keep dags folder at different locations in celery workers which may be running in different environment. So using Airflow with celery with cross environment setup is difficult.This feature will help to keep dags folder at any location at worker nodes and tasks command can be configured for dag file path using home configuration.