Add support for operators that achieve durable execution manually in registry - #70291
Merged
amoghrajesh merged 5 commits intoJul 24, 2026
Merged
Conversation
amoghrajesh
requested review from
Lee-W,
ashb,
bugraoz93,
choo121600,
ephraimbuddy,
gopidesupavan,
hussein-awala,
jason810496,
jedcunningham,
jscheffl,
kaxil,
potiuk and
vatsrahul1001
as code owners
July 23, 2026 09:37
Lee-W
reviewed
Jul 23, 2026
1 task
…etect-manual-durable-marker
Lee-W
approved these changes
Jul 24, 2026
Contributor
Backport successfully created: v3-3-testNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Was generative AI tooling used to co-author this PR?
What
Follow-up to #70289, which added
__supports_durable_execution: ClassVar[bool] = Truedirectly toKubernetesPodOperatorandAgentOperatorboth of which implement crash-safe execution againsttask_state_storewithout inheritingResumableJobMixin. That attribute wasn't read by anything yet; the Registry'sis_durable_capable()only checked forResumableJobMixinin the MRO, so both operators stayed invisible to the badge despite genuinely qualifying.Current behaviour
is_durable_capable()indev/registry/extract_parameters.pyonly qualifies a class via theResumableJobMixinstructural check (inherits it, all abstract methods overridden,execute_resumablereachable fromexecute()). Classes that implement durable execution some other way have no path to qualify.Proposed change
Adds a second, independent qualifying path:
is_durable_capable()now also checks for a class level__supports_durable_execution = Truedeclaration. Because the attribute name uses a double leading underscore, Python name mangles it to_{ClassName}__supports_durable_executionon whichever class declares it. The check computes that mangled name per class being inspected (f"_{cls.__name__}__supports_durable_execution") rather than a fixed string, so it only matches the exact class that wrote the declaration in its own body -- not any subclass that merely inherits from it.Changes of Note
This distinction matters concretely:
KubernetesPodOperatorhas real subclasses (SparkKubernetesOperator,KubernetesJobOperator,KubernetesStartKueueJobOperator) that overrideexecute()themselves. Nothing verifies those overrides preserve thetask_state_storereconnect behavior yet, so the marker must not propagate to them automatically.Testing
Manual validation after generating registry data shows:
{ "id": "common-ai-agent-AgentOperator", "name": "AgentOperator", "type": "operator", "import_path": "airflow.providers.common.ai.operators.agent.AgentOperator", "module_path": "airflow.providers.common.ai.operators.agent", "short_description": "Run a pydantic-ai Agent with tools and multi-turn reasoning.", "docs_url": "https://airflow.apache.org/docs/apache-airflow-providers-common-ai/stable/_api/airflow/providers/common/ai/operators/agent/index.html#airflow.providers.common.ai.operators.agent.AgentOperator", "source_url": "https://github.com/apache/airflow/blob/providers-common-ai/0.7.0/providers/common/ai/src/airflow/providers/common/ai/operators/agent.py#L120", "category": "common-ai", "provider_id": "common-ai", "provider_name": "Common AI", "supports_durable_execution": true },And
{ "id": "cncf-kubernetes-pod-KubernetesPodOperator", "name": "KubernetesPodOperator", "type": "operator", "import_path": "airflow.providers.cncf.kubernetes.operators.pod.KubernetesPodOperator", "module_path": "airflow.providers.cncf.kubernetes.operators.pod", "short_description": "Execute a task in a Kubernetes Pod.", "docs_url": "https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/_api/airflow/providers/cncf/kubernetes/operators/pod/index.html#airflow.providers.cncf.kubernetes.operators.pod.KubernetesPodOperator", "source_url": "https://github.com/apache/airflow/blob/providers-cncf-kubernetes/10.20.0/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/operators/pod.py#L142", "category": "kubernetes", "provider_id": "cncf-kubernetes", "provider_name": "Kubernetes", "supports_durable_execution": true },{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.