Skip to content

opensearch: generalize extra parameters - #65613

Merged
potiuk merged 4 commits into
apache:mainfrom
DonHaul:opensearchhook-extend
Jul 3, 2026
Merged

opensearch: generalize extra parameters#65613
potiuk merged 4 commits into
apache:mainfrom
DonHaul:opensearchhook-extend

Conversation

@DonHaul

@DonHaul DonHaul commented Apr 21, 2026

Copy link
Copy Markdown
Contributor

The current OpenSearch hook, does not include the possibility to set arguments for the opensearch client that may be important for some users such as http_compress and url_prefix etc. opensearchpy library supports many more arguments, including kwargs - see here
As such I've generalized the hook so all the parameters in extra are passed to the opensearch client


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)
  • Codex GPT-5.4

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@eladkal

eladkal commented Apr 22, 2026

Copy link
Copy Markdown
Contributor

cc @Owen-CH-Leung

@DonHaul
DonHaul force-pushed the opensearchhook-extend branch from 1d3ef41 to 2af56cd Compare April 22, 2026 14:25
@potiuk potiuk added the ready for maintainer review Set after triaging when all criteria pass. label Apr 22, 2026

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is too generic as it might update things we do not want to update via extras. Extras are potentially more sensitive than Hook parameters, because they are configurable by "connection editing" users and not "Dag Authors" - we never pass extras directly to hoo params/args we are always explicit which extras should be passed.

@DonHaul

DonHaul commented Apr 29, 2026

Copy link
Copy Markdown
Contributor Author

okay I will specifically define all the parameters in https://opensearch-project.github.io/opensearch-py/_modules/opensearchpy/connection/base.html#Connection . I see that the kwargs is actually only scheme so I will add it as an arg as well

@DonHaul
DonHaul force-pushed the opensearchhook-extend branch 4 times, most recently from 1ff244d to 3ba6c52 Compare April 30, 2026 14:54
@DonHaul

DonHaul commented Apr 30, 2026

Copy link
Copy Markdown
Contributor Author

@potiuk comments addressed

@DonHaul
DonHaul force-pushed the opensearchhook-extend branch from 3ba6c52 to 47ee0a7 Compare May 8, 2026 09:00
@DonHaul
DonHaul force-pushed the opensearchhook-extend branch from 47ee0a7 to db7bfe8 Compare May 15, 2026 08:28
@DonHaul

DonHaul commented Jun 2, 2026

Copy link
Copy Markdown
Contributor Author

@potiuk could you check whenever you have the time? thanks!

@potiuk potiuk removed the ready for maintainer review Set after triaging when all criteria pass. label Jun 9, 2026
@potiuk

potiuk commented Jun 9, 2026

Copy link
Copy Markdown
Member

@DonHaul — A reviewer (@potiuk) has requested changes on this PR, so I've removed the ready for maintainer review label — the next step is on your side. Could you address the review comments (push a fix, or reply in-thread explaining why the feedback doesn't apply)? Once addressed, re-request review from @potiuk or re-mark the PR ready and it returns to the maintainer queue. Thank you.

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

@DonHaul
DonHaul requested a review from potiuk June 10, 2026 06:46
@potiuk

potiuk commented Jun 25, 2026

Copy link
Copy Markdown
Member

@DonHaul — you've pushed commits since the changes-requested review, but the review threads are still open. If you've addressed the feedback, reply to confirm and nudge the reviewer for another look; otherwise let us know what's still outstanding. Thanks!

See the PR quality criteria.

Automated first-pass triage note drafted by an AI-assisted tool — may get things wrong; once addressed, a real Apache Airflow maintainer takes the next look. (why automated)


Drafted-by: Claude Code (Opus 4.8); reviewed by @potiuk before posting

@potiuk potiuk added the ready for maintainer review Set after triaging when all criteria pass. label Jul 2, 2026

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This resolves my objection cleanly. Instead of forwarding connection extras
generically, you've moved to an explicit, named allowlist — url_prefix,
timeout, headers, http_compress, opaque_id, scheme — each declared in
OpenSearchClientArguments, read by name in __init__, and conditionally passed
to the client only when set. That's exactly the "be explicit about which extras
reach the hook arguments" pattern; there's no arbitrary passthrough of
connection-editor-controlled values, and every forwarded key maps to a known
opensearchpy client argument. test_hook_extra_params pins the mapping, and
CI is green. Thanks for reworking it — LGTM.


This review was drafted by an AI-assisted tool and confirmed by an Airflow
maintainer. The maintainer approving this PR has read the findings and signed
off. If something feels off, please reply on the PR and a maintainer will
follow up.

More on how Airflow handles maintainer review:
contributing-docs/05_pull_requests.rst.

@potiuk
potiuk merged commit dc4742b into apache:main Jul 3, 2026
104 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants