Decouple remote logging configuration from core airflow

[amoghrajesh]

Body

airflow_local_settings.py hardcodes imports and configuration for each remote logging provider (S3, GCS, OpenSearch, etc). Example: https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/config_templates/airflow_local_settings.py#L165-L181

Right now this means that:

• Core must know provider implementation details
• Provider changes require core updates
• New logging providers must modify core code

Find a way to have providers register their remote logging config themselves - either via provider.yml or entrypoints. More details to follow in a PR once I have completed some investigation here.

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[amoghrajesh]

cc @ashb @jason810496 FYI

[eladkal]

I may be oversimplifying it but... users are able to specify any executor which exists in providers or even roll their own executor. This just work without any need to make modifications to core airflow. Can't we use the same approach?

[jason810496]

The existing airflow_logging_settings is hard-coded some providers instead of importing via ProvidersManager as user interface and we will even make the provider level to expose the corresponding remote IO logging configs to fully decouple from core.

The executor did respect the ProvidersManager, so we're actually moving toward the executor approach you mentioned.

[amoghrajesh]

No bandwidth in the couple 2 -3 weeks, unassigning