Add WasbRemoteLogIO.from_config and register wasb remote logging scheme - #70301
Conversation
potiuk
left a comment
There was a problem hiding this comment.
Faithful port. All four constructor arguments match the legacy wasb branch — base_log_folder, remote_base with the wasb:// prefix stripped, delete_local_copy, wasb_container — and applying | io_kwargs is right here, since wasb is one of the object-storage backends where legacy merges. (I reviewed this alongside the sibling ports; the two search backends deliberately do not merge, and #70295 currently gets that wrong.)
The io/fth split test is the one that earns its keep, and I checked it isn't passing vacuously:
("logging", "remote_task_handler_kwargs"): '{"delete_local_copy": true, "max_bytes": 1024}'
...
assert subject.delete_local_copy is True
assert not hasattr(subject, "max_bytes")max_bytes is a real FileTaskHandler.__init__ parameter, so this exercises both sides with genuine names — the IO kwarg reaches the class, the file-handler kwarg is filtered. If the split logic broke, max_bytes would be forwarded and construction would raise, so the test guards the mechanism and not just the outcome.
One inherited oddity, not something to fix here: conf.get_mandatory_value(..., fallback="airflow-logs") is contradictory — mandatory with a fallback. It comes across verbatim from airflow_local_settings.py, so the port is accurate; worth someone untangling in core at some point.
Rebased onto main before merging.
Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting
Core resolves remote log handlers by URL scheme through ProvidersManager dispatch (apache#67056); s3 and cloudwatch already migrated. This moves wasb onto the same path, so Azure Blob remote logging is built by the provider's from_config() instead of the hardcoded branch in airflow_local_settings.py. Existing wasb:// configs resolve to an equivalent handler, and a from_config failure falls back to the legacy path, so behaviour is unchanged. Part of apache#70265. closes apache#70268.
e21ec57 to
1036583
Compare
|
Thanks for reviewing! The static check failure looks unrelated, could you please rerun it? |
Add WasbRemoteLogIO.from_config and register wasb remote logging scheme
What
Core resolves remote log handlers by URL scheme through ProvidersManager dispatch, added in #67056.This PR adds
WasbRemoteLogIO.from_config()and registers thewasbscheme, so Azure Blob logging is built by the provider instead of the hardcoded branch inairflow_local_settings.py. Existingwasb://configs resolve to an equivalent handler, and afrom_configerror falls back to the legacy path, so behaviour does not change.Tested with a real task using
remote_base_log_folder = wasb://logsagainst a local Azurite. The log uploaded to theairflow-logscontainer and read back.ProvidersManager().remote_logging_handler_by_scheme("wasb")returnsWasbRemoteLogIO, so the resolver uses provider dispatch, not the legacy branch. Unit tests mirrorTestS3RemoteLogIOFromConfig.related: #70265
closes: #70268
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Opus 4.8 following the guidelines
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.