Make RedshiftDeleteClusterOperator delete reliably during cluster transitions

[seanghaeli]

Why

RedshiftDeleteClusterOperator retries on InvalidClusterStateFault when the cluster is mid-transition (pausing/resuming/resizing), but the retry budget was hardcoded to 10 attempts * 15s = 2.5 minutes. Transitions routinely take longer, so the loop exhausts and re-raises before the cluster becomes deletable.

The error seen mid-transition:

InvalidClusterState: Unable to delete the cluster ... You can only delete clusters with
ACTIVE, INCOMPATIBLE_HSM, INSUFFICIENT_VPCE, HARDWARE_FAILURE, INCOMPATIBLE_NETWORK,
PAUSED, INACCESSIBLE_KMS_KEY, INSUFFICIENT_CAPACITY, INCOMPATIBLE_RESTORE lifecycles.

What

Non-deferrable mode: raise the synchronous busy-retry budget to 60 * 15s = ~15 min so it outlasts a transition. Still throw failure once timed out.

Deferrable mode: previously the synchronous retry loop ran before the defer, blocking the worker for the whole budget (defeating the purpose of deferrable mode). Now the operator attempts the delete once; on InvalidClusterStateFault it defers to a new RedshiftClusterSettledTrigger that treats transitional states as retry and fires once the cluster reaches a deletable lifecycle. A callback then re-issues the delete and defers to the existing RedshiftDeleteClusterTrigger (cluster_deleted); on a race it re-defers to the settle trigger. No worker is blocked at any point.

Mirrors the deferrable re-defer pattern established for EKS in #32355

Testing

End-to-end before/after

Non-deferrable — BEFORE: FAILS, cluster leaks:

20:32:04  Cluster in resizing state, unable to delete. 9 attempts remaining.
   ... (15s apart, 9→1) ...
20:34:22  botocore.errorfactory.InvalidClusterStateFault: Unable to delete the cluster rs-e2e-sync ...
          DagRun ... state=failed   TASK_EXIT=1

Non-deferrable — AFTER: SUCCEEDS, cluster deleted:

20:58:33  Cluster in resizing state, unable to delete. 49 attempts remaining.   <-- past stock's 10-cap
21:01:07  Cluster in resizing state, unable to delete. 39 attempts remaining.
21:01:22  new_state=success   DagRun ... state=success   TASK_EXIT=0

Deferrable — BEFORE: blocks worker 2.5 min then FAILS (never defers):

20:51:53  Cluster in resizing state, unable to delete. 9 attempts remaining.
   ... 9→1 ...
20:54:11  new_state=failed   DagRun ... state=failed   TASK_EXIT=1

Deferrable — AFTER: async wait via cluster_deletable waiter, then delete. SUCCEEDS:

21:02:10  Cluster rs-e2e-async is busy; deferring until it settles into a deletable state.
21:02:10  Pausing task as DEFERRED.
21:02:11  Waiting for redshift cluster to settle into a deletable state: ['resizing']
   ... waiter treats 'resizing' as retry for ~10 min, task stays DEFERRED (worker not blocked) ...
21:12:38  (settled) delete re-issued and accepted -> defer to cluster_deleted
21:15:10  Redshift Cluster deletion in progress: ['deleting']
21:16:10  Cluster deleted successfully   DagRun ... state=success   TASK_EXIT=0

[potiuk]

@seanghaeli This PR has been converted to draft because it does not yet meet our Pull Request quality criteria.

Issues found:

• ❌ Merge conflicts: This PR has merge conflicts with main and cannot be merged as-is. Rebase locally and resolve them: git fetch upstream main && git rebase upstream/main, resolve the conflicts, then force-push.
• ❌ Unresolved review comments: There are 1 unresolved review thread(s) on this PR from @vincbeck. For each one: either apply the suggestion in a follow-up commit, or reply in-thread with a brief explanation of why the feedback doesn't apply, or resolve the thread if the feedback is no longer relevant. Once you believe a thread is resolved — whether by pushing a fix or by explaining why the suggestion doesn't apply — it is the author's responsibility to mark it as resolved by clicking the 'Resolve conversation' button at the bottom of the thread. Reviewers don't auto-close their own threads, so unresolved threads read as 'still waiting on the author' and block the PR from moving forward.

Note: Your branch is 237 commits behind main. Some check failures may be caused by changes in the base branch rather than by your PR. Please rebase your branch and push again to get up-to-date CI results.

What to do next:

• Fix each issue listed above.
• Make sure static checks pass locally (prek run --from-ref main --stage pre-commit).
• Mark the PR as "Ready for review" when you're done.

Converting a PR to draft is not a rejection — it is an invitation to bring the PR up to the project's standards so that maintainer review time is spent productively. There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates. If you have questions, feel free to ask on the Airflow Slack.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[vincbeck]

Thanks for doing the changes