Add SQLBulkLoadOperator to Common SQL Provider - #69362
Merged
Merged
Conversation
SameerMesiah97
marked this pull request as ready for review
July 3, 2026 21:12
SameerMesiah97
marked this pull request as draft
July 3, 2026 21:12
SameerMesiah97
force-pushed
the
SQLBulkLoadOperator
branch
from
July 3, 2026 21:44
af26e74 to
b5d34d2
Compare
SameerMesiah97
marked this pull request as ready for review
July 6, 2026 16:33
Contributor
Author
|
Requesting review for this. |
shahar1
approved these changes
Jul 25, 2026
added 2 commits
July 26, 2026 20:05
…les using the underlying hook's native bulk loading implementation. The operator supports optional pre- and post-load SQL statements for common setup and cleanup tasks. Includes unit tests, documentation, and an example DAG.
SameerMesiah97
force-pushed
the
SQLBulkLoadOperator
branch
from
July 26, 2026 19:05
b5d34d2 to
2d01210
Compare
Kunal8954
pushed a commit
to Kunal8954/airflow
that referenced
this pull request
Jul 27, 2026
Add SQLBulkLoadOperator to bulk load tab-delimited files into SQL tables using the underlying hook's native bulk loading implementation. The operator supports optional pre- and post-load SQL statements for common setup and cleanup tasks. Includes unit tests, documentation, and an example DAG.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This change introduces a new operator,
SQLBulkLoadOperator, to bulk load tab-delimited files into SQL database tables.The operator delegates to the underlying hook's
bulk_loadimplementation, allowing each SQL provider to use its native bulk loading mechanism through a common interface. It also supports optionalpreoperatorandpostoperatorSQL statements to perform common setup and cleanup tasks as part of the bulk load workflow.Rationale
DbApiHookalready exposes a commonbulk_loadAPI, which is implemented by multiple SQL providers. However, unlikerun()andinsert_rows(), there is currently no corresponding operator that exposes this functionality to DAG authors.While the operator provides a convenient wrapper around the existing hook API, it also simplifies common bulk loading workflows by allowing setup and cleanup SQL to be executed as part of the same task. This avoids requiring users to implement custom
PythonOperators solely to orchestrate bulk loading operations.Tests
Added unit tests verifying that:
SQLBulkLoadOperatorcorrectly delegates to the underlying hook'sbulk_loadmethod.preoperatorandpostoperatorSQL statements are executed before and after the bulk load operation, respectively.Example DAG
Added a new example DAG,
example_sql_bulk_load, demonstrating how to bulk load a tab-delimited file into a SQL table usingSQLBulkLoadOperator. The example also shows how to usepreoperatorandpostoperatorto prepare and clean up the destination table as part of the bulk load workflow.Documentation
Added documentation for
SQLBulkLoadOperator, including its supported parameters and a usage example.Backwards Compatibility
This is a new operator and does not modify existing functionality. No breaking changes are introduced.