Apple M1 Docker-compose up airflow-init produces airflow command error

[oursy]

Apache Airflow version: 2.0.1

Environment:

• OS (e.g. from /etc/os-release): macOS 11.2.2 M1
• Kernel (e.g. uname -a): Darwin Darwin Kernel Version 20.3.0
• Install tools: docker-compose

What happened:

Following the steps in the Quick start > Running Airflow in Docker page (https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html#initializing-environment):

1. Pulled down the pre-made docker-compose.yaml file (https://airflow.apache.org/docs/apache-airflow/2.0.1/docker-compose.yaml)
2. Ran docker-compose up airflow-init within the same directory as the docker-compose file
3. This command failed with: airflow command error: argument GROUP_OR_COMMAND: airflow upgradedb command, has been removed, please use airflow db upgrade, see help above.

What you expected to happen:

The latest stable official airflow docker image to be pulled, and airflow-init to complete successfully.

How to reproduce it:

Follow the instructions per the quick start docs (https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html) to pull down the docker-compose.yaml file, then attempt to run docker-compose up airflow-init.

1. mkdir ./dags ./logs ./plugins echo -e "AIRFLOW_UID=$(id -u)\nAIRFLOW_GID=0" > .env
2. docker-compose up airflow-init

Anything else we need to know:

Below is the log information

$ docker-compose up airflow-init         

Building with native build. Learn about native build in Compose here: https://docs.docker.com/go/compose-native-build/
Creating network "airflow_default" with the default driver
Creating volume "airflow_postgres-db-volume" with default driver
Pulling postgres (postgres:13)...
13: Pulling from library/postgres
706ee5d0a6b5: Pull complete
f0d0c8ce025c: Pull complete
d65206854bd5: Pull complete
e02ab81971e4: Pull complete
3e4d9f964bbd: Pull complete
22d34800f722: Pull complete
b580fcf09f2e: Pull complete
927098918ec9: Pull complete
135acccac104: Pull complete
e83b2f02fcec: Pull complete
5a4f5dd074b7: Pull complete
a7c009ac8442: Pull complete
f27c1fa930aa: Pull complete
97e124b88a98: Pull complete
Digest: sha256:c83014a2b46834ef6d17f64c8e4a70089901a8c0dee158f1ca5ccae032ea32e5
Status: Downloaded newer image for postgres:13
Pulling redis (redis:latest)...
latest: Pulling from library/redis
706ee5d0a6b5: Already exists
ca1c5a94560f: Pull complete
36348e11b8f8: Pull complete
b3e4d340a8f5: Pull complete
556088962a68: Pull complete
461f118c9c37: Pull complete
Digest: sha256:e97d506be34a39fa69f45eea846080d6985c2c9ee338c0d408c7ea4347f014a5
Status: Downloaded newer image for redis:latest
Pulling airflow-init (apache/airflow:2.0.1)...
2.0.1: Pulling from apache/airflow
45b42c59be33: Pull complete
f875e16ab19c: Pull complete
a4fb13de3215: Pull complete
f3dd9aaf319e: Pull complete
451a8f7ee895: Pull complete
e2d0adf49357: Pull complete
52856765f525: Pull complete
fd428150aaa6: Pull complete
b4fa7f6b9775: Pull complete
b34f0a9234cb: Pull complete
537882d8c1ac: Pull complete
dffdc965d748: Pull complete
a97e5a67cb46: Pull complete
c401f5284516: Pull complete
af49f5db1642: Pull complete
c61449868181: Pull complete
0b574505c97e: Pull complete
a65349ef32b2: Pull complete
Digest: sha256:2bf4fe559e9e699487a50d4e587358bb0b3c90b59c8fefeccc8a4fb2e83f3d09
Status: Downloaded newer image for apache/airflow:2.0.1
Creating airflow_postgres_1 ... done
Creating airflow_redis_1    ... done
Creating airflow_airflow-init_1 ... done
Attaching to airflow_airflow-init_1
airflow-init_1       | BACKEND=postgresql+psycopg2
airflow-init_1       | DB_HOST=postgres
airflow-init_1       | DB_PORT=5432
airflow-init_1       | 
airflow-init_1       | [2021-03-15 09:24:55,179] {cli_action_loggers.py:105} WARNING - Failed to log action with (psycopg2.errors.UndefinedTable) relation "log" does not exist
airflow-init_1       | LINE 1: INSERT INTO log (dttm, dag_id, task_id, event, execution_dat...
airflow-init_1       |                     ^
airflow-init_1       | 
airflow-init_1       | [SQL: INSERT INTO log (dttm, dag_id, task_id, event, execution_date, owner, extra) VALUES (%(dttm)s, %(dag_id)s, %(task_id)s, %(event)s, %(execution_date)s, %(owner)s, %(extra)s) RETURNING log.id]
airflow-init_1       | [parameters: {'dttm': datetime.datetime(2021, 3, 15, 9, 24, 55, 145656, tzinfo=Timezone('UTC')), 'dag_id': None, 'task_id': None, 'event': 'cli_upgradedb', 'execution_date': None, 'owner': 'default', 'extra': '{"host_name": "1c70efb17222", "full_command": "[\'/home/airflow/.local/bin/airflow\', \'db\', \'upgrade\']"}'}]
airflow-init_1       | (Background on this error at: http://sqlalche.me/e/13/f405)
airflow-init_1       | DB: postgresql+psycopg2://airflow:***@postgres/airflow
airflow-init_1       | [2021-03-15 09:24:56,100] {db.py:674} INFO - Creating tables
airflow-init_1       | INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
airflow-init_1       | INFO  [alembic.runtime.migration] Will assume transactional DDL.
airflow-init_1       | WARNI [airflow.providers_manager] Exception when importing 'airflow.providers.microsoft.azure.hooks.wasb.WasbHook' from 'apache-airflow-providers-microsoft-azure' package: No module named 'azure.storage.blob'
airflow-init_1       | qemu: uncaught target signal 4 (Illegal instruction) - core dumped
airflow-init_1       | /entrypoint: line 192:    26 Illegal instruction     airflow db upgrade
airflow-init_1       | 
airflow-init_1       | airflow command error: argument GROUP_OR_COMMAND: `airflow upgradedb` command, has been removed, please use `airflow db upgrade`, see help above.
airflow-init_1       | usage: airflow [-h] GROUP_OR_COMMAND ...
airflow-init_1       | 
airflow-init_1       | positional arguments:
airflow-init_1       |   GROUP_OR_COMMAND
airflow-init_1       | 
airflow-init_1       |     Groups:
airflow-init_1       |       celery         Celery components
airflow-init_1       |       config         View configuration
airflow-init_1       |       connections    Manage connections
airflow-init_1       |       dags           Manage DAGs
airflow-init_1       |       db             Database operations
airflow-init_1       |       kubernetes     Tools to help run the KubernetesExecutor
airflow-init_1       |       pools          Manage pools
airflow-init_1       |       providers      Display providers
airflow-init_1       |       roles          Manage roles
airflow-init_1       |       tasks          Manage tasks
airflow-init_1       |       users          Manage users
airflow-init_1       |       variables      Manage variables
airflow-init_1       | 
airflow-init_1       |     Commands:
airflow-init_1       |       cheat-sheet    Display cheat sheet
airflow-init_1       |       info           Show information about current Airflow and environment
airflow-init_1       |       kerberos       Start a kerberos ticket renewer
airflow-init_1       |       plugins        Dump information about loaded plugins
airflow-init_1       |       rotate-fernet-key
airflow-init_1       |                      Rotate encrypted connection credentials and variables
airflow-init_1       |       scheduler      Start a scheduler instance
airflow-init_1       |       sync-perm      Update permissions for existing roles and DAGs
airflow-init_1       |       version        Show the version
airflow-init_1       |       webserver      Start a Airflow webserver instance
airflow-init_1       | 
airflow-init_1       | optional arguments:
airflow-init_1       |   -h, --help         show this help message and exit
airflow-init_1       | [2021-03-15 09:25:25,537] {providers_manager.py:299} WARNING - Exception when importing 'airflow.providers.microsoft.azure.hooks.wasb.WasbHook' from 'apache-airflow-providers-microsoft-azure' package: No module named 'azure.storage.blob'
airflow-init_1       | qemu: uncaught target signal 4 (Illegal instruction) - core dumped
airflow-init_1       | /entrypoint: line 129:    53 Illegal instruction     airflow users create --username "${_AIRFLOW_WWW_USER_USERNAME="admin"}" --firstname "${_AIRFLOW_WWW_USER_FIRSTNAME="Airflow"}" --lastname "${_AIRFLOW_WWW_USER_LASTNME="Admin"}" --email "${_AIRFLOW_WWW_USER_EMAIL="airflowadmin@example.com"}" --role "${_AIRFLOW_WWW_USER_ROLE="Admin"}" --password "${local_password}"
airflow-init_1       | 
airflow-init_1       | airflow command error: argument GROUP_OR_COMMAND: `airflow create_user` command, has been removed, please use `airflow users create`, see help above.
airflow-init_1       | usage: airflow [-h] GROUP_OR_COMMAND ...
airflow-init_1       | 
airflow-init_1       | positional arguments:
airflow-init_1       |   GROUP_OR_COMMAND
airflow-init_1       | 
airflow-init_1       |     Groups:
airflow-init_1       |       celery         Celery components
airflow-init_1       |       config         View configuration
airflow-init_1       |       connections    Manage connections
airflow-init_1       |       dags           Manage DAGs
airflow-init_1       |       db             Database operations
airflow-init_1       |       kubernetes     Tools to help run the KubernetesExecutor
airflow-init_1       |       pools          Manage pools
airflow-init_1       |       providers      Display providers
airflow-init_1       |       roles          Manage roles
airflow-init_1       |       tasks          Manage tasks
airflow-init_1       |       users          Manage users
airflow-init_1       |       variables      Manage variables
airflow-init_1       | 
airflow-init_1       |     Commands:
airflow-init_1       |       cheat-sheet    Display cheat sheet
airflow-init_1       |       info           Show information about current Airflow and environment
airflow-init_1       |       kerberos       Start a kerberos ticket renewer
airflow-init_1       |       plugins        Dump information about loaded plugins
airflow-init_1       |       rotate-fernet-key
airflow-init_1       |                      Rotate encrypted connection credentials and variables
airflow-init_1       |       scheduler      Start a scheduler instance
airflow-init_1       |       sync-perm      Update permissions for existing roles and DAGs
airflow-init_1       |       version        Show the version
airflow-init_1       |       webserver      Start a Airflow webserver instance
airflow-init_1       | 
airflow-init_1       | optional arguments:
airflow-init_1       |   -h, --help         show this help message and exit
airflow-init_1       | 2.0.1
airflow_airflow-init_1 exited with code 0

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[mik-laj]

mkdir ./dags ./logs ./plugins echo -e "AIRFLOW_UID=$(id -u)\nAIRFLOW_GID=0" > .env

This command should only be run on Linux.

[potiuk]

Airflow has no support for M1 - this M1 is an ARM device and we do not support ARM in docker images. There are some anecdotal evidences of people who were able to install Airflow on M1 device but it is neither tested nor supported.

If you would like to use Airflow with M1. I suggest to open a feature request for that.