Skip to content

[Compatibility] Official Helm Chart 1.16.0, Apache Airflow 2.10.5: opensearch remote_logging broken.  #52763

Description

@VergeDX

Official Helm Chart version

1.16.0

Apache Airflow version

2.10.5

Kubernetes Version

v1.29.15

Helm Chart configuration

opensearch:
  enabled: true
  connection:
    scheme: http
    host: opensearch-cluster-master.opensearch.svc
    port: 9200

Docker Image customizations

# FROM docker.io/apache/airflow:2.10.5-python3.10
FROM registry.cn-chengdu.aliyuncs.com/d-i/apache_airflow:2.10.5-python3.10

# https://mirrors.bfsu.edu.cn/help/pypi/
RUN pip config set global.index-url https://mirrors.bfsu.edu.cn/pypi/web/simple
RUN pip install --no-cache-dir "apache-airflow[opensearch]==2.10.5" toml rocketchat_API

USER root
RUN apt update && apt install -y --no-install-recommends git && \
    apt autoremove -yqq --purge && apt clean && rm -rf /var/lib/apt/lists/*
USER airflow

Image registry.cn-chengdu.aliyuncs.com/d-i/apache_airflow:2.10.5-python3.10 is just a mirror of docker.io/apache/airflow:2.10.5-python3.10.

What happened

Incorrect remote log configuration. Please check the configuration of option 'host' in section 'elasticsearch' if you are using Elasticsearch. In the other case, 'remote_base_log_folder' option in the 'logging' section.

What you think should happen instead

Remote logging + OpenSearch backend should work as well.

How to reproduce

https://airflow.apache.org/docs/helm-chart/1.16.0/index.html

Anything else

PR #45082 adds opensearch helm values and first appears at the Official Helm Chart version 1.16.0;
Official Helm Chart version 1.16.0 corresponds Apache Airflow version 2.10.5 (defaultAirflowTag).
But the opensearch backend implementation (#41799) first appeared at the Apache Airflow version 3.0.0.a1.

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions