Skip to content

Add Cloud SQL no-operation sensor - #68146

Closed
anmolxlight wants to merge 4 commits into
apache:mainfrom
anmolxlight:fix/68040-cloudsql-idle-sensor
Closed

Add Cloud SQL no-operation sensor#68146
anmolxlight wants to merge 4 commits into
apache:mainfrom
anmolxlight:fix/68040-cloudsql-idle-sensor

Conversation

@anmolxlight

Copy link
Copy Markdown
Contributor

Add a deferrable CloudSQLNoOperationInProgressSensor for waiting until a Cloud SQL instance has no non-terminal operation in progress.

This gives DAG authors an explicit provider primitive to avoid operationInProgress 409s before submitting import, export, clone, patch, or other Cloud SQL administrative operations.

Changes:

  • add sync and async Cloud SQL operation-list helpers
  • add CloudSQLNoOperationInProgressSensor
  • add CloudSQLNoOperationInProgressTrigger
  • register the new sensor in the Google provider metadata
  • document the sensor in the Cloud SQL how-to guide and system test example
  • add unit coverage for the hook helper, sensor, and trigger

Closes: #68040

Tests:

  • uv run pytest --with-db-init providers/google/tests/unit/google/cloud/sensors/test_cloud_sql.py providers/google/tests/unit/google/cloud/triggers/test_cloud_sql_no_operation.py providers/google/tests/unit/google/cloud/hooks/test_cloud_sql.py -q
  • uv run ruff check providers/google/src/airflow/providers/google/cloud/hooks/cloud_sql.py providers/google/src/airflow/providers/google/cloud/triggers/cloud_sql.py providers/google/src/airflow/providers/google/cloud/sensors/cloud_sql.py providers/google/tests/system/google/cloud/cloud_sql/example_cloud_sql.py providers/google/tests/unit/google/cloud/hooks/test_cloud_sql.py providers/google/tests/unit/google/cloud/sensors/test_cloud_sql.py providers/google/tests/unit/google/cloud/triggers/test_cloud_sql_no_operation.py
  • uv run ruff format --check providers/google/src/airflow/providers/google/cloud/hooks/cloud_sql.py providers/google/src/airflow/providers/google/cloud/triggers/cloud_sql.py providers/google/src/airflow/providers/google/cloud/sensors/cloud_sql.py providers/google/tests/system/google/cloud/cloud_sql/example_cloud_sql.py providers/google/tests/unit/google/cloud/hooks/test_cloud_sql.py providers/google/tests/unit/google/cloud/sensors/test_cloud_sql.py providers/google/tests/unit/google/cloud/triggers/test_cloud_sql_no_operation.py
  • PYTHONPATH=scripts/ci/prek uv run --script scripts/ci/prek/check_trigger_serialize_init.py providers/google/src/airflow/providers/google/cloud/triggers/cloud_sql.py
  • uv run --script scripts/ci/prek/check_deferrable_default.py
  • uv run prek run --config .pre-commit-config.yaml --files generated/provider_dependencies.json.sha256sum providers/google/docs/operators/cloud/cloud_sql.rst providers/google/provider.yaml providers/google/src/airflow/providers/google/cloud/hooks/cloud_sql.py providers/google/src/airflow/providers/google/cloud/triggers/cloud_sql.py providers/google/src/airflow/providers/google/cloud/sensors/cloud_sql.py providers/google/tests/system/google/cloud/cloud_sql/example_cloud_sql.py providers/google/tests/unit/google/cloud/hooks/test_cloud_sql.py providers/google/tests/unit/google/cloud/sensors/test_cloud_sql.py providers/google/tests/unit/google/cloud/triggers/test_cloud_sql_no_operation.py

@anmolxlight
anmolxlight requested a review from shahar1 as a code owner June 6, 2026 22:29
@boring-cyborg boring-cyborg Bot added area:providers kind:documentation provider:google Google (including GCP) related issues labels Jun 6, 2026
@anmolxlight
anmolxlight force-pushed the fix/68040-cloudsql-idle-sensor branch from 6a83bc7 to cfeb9f1 Compare June 7, 2026 07:45

@shahar1 shahar1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please resolve conflicts

@potiuk

potiuk commented Jul 31, 2026

Copy link
Copy Markdown
Member

Thank you for this, and I'm sorry it's taken eight weeks to get you an answer.

#68151 proposes the same feature — same CloudSQLNoOperationInProgressSensor, same CloudSQLNoOperationInProgressTrigger, near-identical file set — and was opened a day after yours. Two people built the same primitive in parallel and neither got reviewed, which is a failure of our process rather than of either contribution. I've merged #68151 and am closing this one.

To be straight about why, since "the other one won" is not a useful answer:

Your version was ahead on one point I want on record: it introduces no AirflowException usage at all. #68151 defines a CloudSQLOperationError(AirflowException) subclass — acceptable under the project's guidance, but yours avoided the base class entirely.

One thing to check on your side for future PRs: this branch modifies generated/provider_dependencies.json.sha256sum, a generated checksum that shouldn't normally be hand-edited in a feature PR — worth confirming that wasn't an accidental inclusion.

None of this reflects on the quality of your work. If you'd like to stay involved, a review on #68151 would be genuinely welcome — you know this problem space as well as anyone — and checking gh pr list --search "<feature>" before starting will stop this happening again.


Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

@potiuk potiuk closed this Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers kind:documentation provider:google Google (including GCP) related issues ready for maintainer review Set after triaging when all criteria pass.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cloud SQL — CloudSQLImportInstanceOperator / CloudSQLExportInstanceOperator 409 operationInProgress on parallel tasks against the same instanc

3 participants