No pidfile created when running in daemon mode

[mdhont]

Apache Airflow version

2.8.1

If "Other Airflow 2 version" selected, which one?

No response

What happened?

Currently, when executing airflow scheduler in daemon mode, a pidfile is generated in the designated AIRFLOW_HOME directory. However, even when specifying the --pid option with an absolute path to an alternative directory, the pidfile continues to be created in the default AIRFLOW_HOME directory.

Also, when running the airflow celery worker command, specifying an alternative pidfile path via the --pid option functions only if the --daemon option is not set.

What you think should happen instead?

For airflow scheduler:

When running in daemon mode and specifying an alternative path for the pidfile using the --pid option, the pidfile should be created in the specified directory rather than the default AIRFLOW_HOME directory.

For airflow celery worker:

When both the --pid and --daemon options are provided, the pidfile should be created in the directory specified by the --pid option.

How to reproduce

For airflow scheduler:

Run airflow scheduler in daemon mode with the --pid option set to an absolute path pointing to an alternative folder.
Note that the pidfile is still generated in the default AIRFLOW_HOME directory instead of the specified alternative folder.

For airflow celery worker:

Execute airflow celery worker with both the --pid and --daemon options specified.
Note that the pidfile is created in the default directory instead of the location specified by the --pid option.

Operating System

Linux x64 / Debian 11

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==8.16.0
apache-airflow-providers-apache-cassandra==3.4.1
apache-airflow-providers-apache-drill==2.6.0
apache-airflow-providers-apache-druid==3.7.0
apache-airflow-providers-apache-hdfs==4.3.2
apache-airflow-providers-apache-hive==6.4.1
apache-airflow-providers-apache-impala==1.3.0
apache-airflow-providers-apache-pinot==4.3.0
apache-airflow-providers-arangodb==2.4.1
apache-airflow-providers-celery==3.5.1
apache-airflow-providers-cloudant==3.4.1
apache-airflow-providers-cncf-kubernetes==7.13.0
apache-airflow-providers-common-io==1.2.0
apache-airflow-providers-common-sql==1.10.0
apache-airflow-providers-databricks==6.0.0
apache-airflow-providers-docker==3.9.1
apache-airflow-providers-elasticsearch==5.3.1
apache-airflow-providers-exasol==4.4.1
apache-airflow-providers-ftp==3.7.0
apache-airflow-providers-google==10.13.1
apache-airflow-providers-grpc==3.4.1
apache-airflow-providers-hashicorp==3.6.1
apache-airflow-providers-http==4.8.0
apache-airflow-providers-imap==3.5.0
apache-airflow-providers-influxdb==2.4.0
apache-airflow-providers-microsoft-azure==8.5.1
apache-airflow-providers-microsoft-mssql==3.6.0
apache-airflow-providers-mongo==3.5.0
apache-airflow-providers-mysql==5.5.1
apache-airflow-providers-neo4j==3.5.0
apache-airflow-providers-postgres==5.10.0
apache-airflow-providers-presto==5.4.0
apache-airflow-providers-redis==3.6.0
apache-airflow-providers-sendgrid==3.4.0
apache-airflow-providers-sftp==4.8.1
apache-airflow-providers-slack==8.5.1
apache-airflow-providers-sqlite==3.7.0
apache-airflow-providers-ssh==3.10.0
apache-airflow-providers-trino==5.6.0
apache-airflow-providers-vertica==3.7.0

Deployment

Other

Deployment details

Bitnami Airflow Singlevm 2.8.1 is deployed with custom startup scripts using gonit as a service manager. Python version is 3.11 and Node version is 18.

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[mdhont]

Ok, I think this might be related to this issue:

#36541

[eladkal]

Fixed by #36794