Skip to content

Add SQLBulkLoadOperator to Common SQL Provider - #69362

Merged
shahar1 merged 2 commits into
apache:mainfrom
SameerMesiah97:SQLBulkLoadOperator
Jul 26, 2026
Merged

Add SQLBulkLoadOperator to Common SQL Provider#69362
shahar1 merged 2 commits into
apache:mainfrom
SameerMesiah97:SQLBulkLoadOperator

Conversation

@SameerMesiah97

Copy link
Copy Markdown
Contributor

Description

This change introduces a new operator, SQLBulkLoadOperator, to bulk load tab-delimited files into SQL database tables.

The operator delegates to the underlying hook's bulk_load implementation, allowing each SQL provider to use its native bulk loading mechanism through a common interface. It also supports optional preoperator and postoperator SQL statements to perform common setup and cleanup tasks as part of the bulk load workflow.

Rationale

DbApiHook already exposes a common bulk_load API, which is implemented by multiple SQL providers. However, unlike run() and insert_rows(), there is currently no corresponding operator that exposes this functionality to DAG authors.

While the operator provides a convenient wrapper around the existing hook API, it also simplifies common bulk loading workflows by allowing setup and cleanup SQL to be executed as part of the same task. This avoids requiring users to implement custom PythonOperators solely to orchestrate bulk loading operations.

Tests

Added unit tests verifying that:

  • SQLBulkLoadOperator correctly delegates to the underlying hook's bulk_load method.
  • Optional preoperator and postoperator SQL statements are executed before and after the bulk load operation, respectively.
  • Template fields are correctly rendered before invoking the underlying hook.

Example DAG

Added a new example DAG, example_sql_bulk_load, demonstrating how to bulk load a tab-delimited file into a SQL table using SQLBulkLoadOperator. The example also shows how to use preoperator and postoperator to prepare and clean up the destination table as part of the bulk load workflow.

Documentation

Added documentation for SQLBulkLoadOperator, including its supported parameters and a usage example.

Backwards Compatibility

This is a new operator and does not modify existing functionality. No breaking changes are introduced.

@SameerMesiah97
SameerMesiah97 marked this pull request as ready for review July 3, 2026 21:12
@SameerMesiah97
SameerMesiah97 marked this pull request as draft July 3, 2026 21:12
@SameerMesiah97
SameerMesiah97 marked this pull request as ready for review July 6, 2026 16:33
@potiuk potiuk added the ready for maintainer review Set after triaging when all criteria pass. label Jul 8, 2026
@SameerMesiah97

Copy link
Copy Markdown
Contributor Author

Requesting review for this.

@shahar1 shahar1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good overall!
Few minor/nit comments to address before merging.

Comment thread providers/common/sql/tests/unit/common/sql/operators/test_sql.py Outdated
Comment thread providers/common/sql/src/airflow/providers/common/sql/operators/sql.py Outdated
Sameer Mesiah added 2 commits July 26, 2026 20:05
…les using the underlying hook's native bulk loading implementation. The operator supports optional pre- and post-load SQL statements for common setup and cleanup tasks. Includes unit tests, documentation, and an example DAG.
@shahar1
shahar1 merged commit c4a5f19 into apache:main Jul 26, 2026
106 checks passed
Kunal8954 pushed a commit to Kunal8954/airflow that referenced this pull request Jul 27, 2026
Add SQLBulkLoadOperator to bulk load tab-delimited files into SQL tables using the underlying hook's native bulk loading implementation. The operator supports optional pre- and post-load SQL statements for common setup and cleanup tasks. Includes unit tests, documentation, and an example DAG.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants