Skip to content

Add WasbRemoteLogIO.from_config and register wasb remote logging scheme - #70301

Merged
potiuk merged 1 commit into
apache:mainfrom
Andrushika:wasb-remote-log-from-config
Aug 1, 2026
Merged

Add WasbRemoteLogIO.from_config and register wasb remote logging scheme#70301
potiuk merged 1 commit into
apache:mainfrom
Andrushika:wasb-remote-log-from-config

Conversation

@Andrushika

@Andrushika Andrushika commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Add WasbRemoteLogIO.from_config and register wasb remote logging scheme

What

Core resolves remote log handlers by URL scheme through ProvidersManager dispatch, added in #67056.This PR adds WasbRemoteLogIO.from_config() and registers the wasb scheme, so Azure Blob logging is built by the provider instead of the hardcoded branch in airflow_local_settings.py. Existing wasb:// configs resolve to an equivalent handler, and a from_config error falls back to the legacy path, so behaviour does not change.

Tested with a real task using remote_base_log_folder = wasb://logs against a local Azurite. The log uploaded to the airflow-logs container and read back. ProvidersManager().remote_logging_handler_by_scheme("wasb") returns WasbRemoteLogIO, so the resolver uses provider dispatch, not the legacy branch. Unit tests mirror TestS3RemoteLogIOFromConfig.

related: #70265
closes: #70268

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Claude Opus 4.8 following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Faithful port. All four constructor arguments match the legacy wasb branch — base_log_folder, remote_base with the wasb:// prefix stripped, delete_local_copy, wasb_container — and applying | io_kwargs is right here, since wasb is one of the object-storage backends where legacy merges. (I reviewed this alongside the sibling ports; the two search backends deliberately do not merge, and #70295 currently gets that wrong.)

The io/fth split test is the one that earns its keep, and I checked it isn't passing vacuously:

("logging", "remote_task_handler_kwargs"): '{"delete_local_copy": true, "max_bytes": 1024}'
...
assert subject.delete_local_copy is True
assert not hasattr(subject, "max_bytes")

max_bytes is a real FileTaskHandler.__init__ parameter, so this exercises both sides with genuine names — the IO kwarg reaches the class, the file-handler kwarg is filtered. If the split logic broke, max_bytes would be forwarded and construction would raise, so the test guards the mechanism and not just the outcome.

One inherited oddity, not something to fix here: conf.get_mandatory_value(..., fallback="airflow-logs") is contradictory — mandatory with a fallback. It comes across verbatim from airflow_local_settings.py, so the port is accurate; worth someone untangling in core at some point.

Rebased onto main before merging.


Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

Core resolves remote log handlers by URL scheme through ProvidersManager dispatch (apache#67056); s3 and cloudwatch already migrated. This moves wasb onto the same path, so Azure Blob remote logging is built by the provider's from_config() instead of the hardcoded branch in airflow_local_settings.py. Existing wasb:// configs resolve to an equivalent handler, and a from_config failure falls back to the legacy path, so behaviour is unchanged.

Part of apache#70265. closes apache#70268.
@potiuk
potiuk force-pushed the wasb-remote-log-from-config branch from e21ec57 to 1036583 Compare August 1, 2026 03:26
@Andrushika

Andrushika commented Aug 1, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for reviewing! The static check failure looks unrelated, could you please rerun it?

@potiuk
potiuk merged commit a472c83 into apache:main Aug 1, 2026
158 of 159 checks passed
@Andrushika
Andrushika deleted the wasb-remote-log-from-config branch August 1, 2026 10:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add WasbRemoteLogIO.from_config and register wasb remote logging scheme

2 participants