Skip to content

Add AWS DMS replication task operators - #15850

Merged
potiuk merged 7 commits into
apache:mainfrom
Visya:add-aws-dms
Jun 18, 2021
Merged

Add AWS DMS replication task operators#15850
potiuk merged 7 commits into
apache:mainfrom
Visya:add-aws-dms

Conversation

@Visya

@Visya Visya commented May 14, 2021

Copy link
Copy Markdown
Contributor

This PR adds ability to work with AWS DMS replication tasks in Airflow:

  • AWS DMS base hook.
  • AWS DMS replication task base and completion sensor.
  • Operators for creating, starting, stopping, deleting and describing AWS DMS replication tasks.
  • Example of a DAG for creating, waiting for completion and deleting a DMS replication task.

Addresses a JIRA issue: AIRFLOW-2334


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

@Visya
Visya requested review from mik-laj and vikramkoka as code owners May 14, 2021 14:06
@Visya
Visya force-pushed the add-aws-dms branch 4 times, most recently from d9d9a4e to 8255981 Compare May 14, 2021 20:44
@github-actions

Copy link
Copy Markdown
Contributor

The Workflow run is cancelling this PR. Building images for the PR has failed. Follow the workflow link to check the reason.

@eladkal
eladkal requested a review from feluelle May 21, 2021 10:28
Comment thread airflow/providers/amazon/aws/hooks/dms.py Outdated
Comment thread airflow/providers/amazon/aws/hooks/dms.py Outdated
Comment thread airflow/providers/amazon/aws/hooks/dms.py Outdated
Comment thread airflow/providers/amazon/aws/hooks/dms.py Outdated
@Visya
Visya force-pushed the add-aws-dms branch 6 times, most recently from b51c969 to 7f126d1 Compare May 25, 2021 20:15
@Visya
Visya force-pushed the add-aws-dms branch 4 times, most recently from 191c13d to 5c3e50e Compare June 2, 2021 15:09
@Visya

Visya commented Jun 3, 2021

Copy link
Copy Markdown
Contributor Author

The pylint CI times out for some reason, but it works when I run it locally, and all the tests pass. From what I see, most of the CI builds have failed in the past couple of days, so I don't think it's specific to my PR.

This is ready to merge unless you have any more comments. 🙂

@potiuk potiuk closed this Jun 3, 2021
@potiuk potiuk reopened this Jun 3, 2021
@potiuk

potiuk commented Jun 3, 2021

Copy link
Copy Markdown
Member

There was a GitHub failure. Closing/reopening to rebuild

@Visya

Visya commented Jun 4, 2021

Copy link
Copy Markdown
Contributor Author

Still timed out at pylint, I've updated my fork to see if this would pass.

Update: Static linting failed on unrelated to the PR files.

@Visya
Visya force-pushed the add-aws-dms branch 2 times, most recently from a2dc09f to 45a71ab Compare June 7, 2021 06:54
@Visya

Visya commented Jun 7, 2021

Copy link
Copy Markdown
Contributor Author

I'm still unable to make this pass, it times out at pylint.
I have run pre-commit locally, and it succeeds.

@Visya
Visya force-pushed the add-aws-dms branch 3 times, most recently from 826c13d to a10668e Compare June 10, 2021 11:14
@Visya

Visya commented Jun 11, 2021

Copy link
Copy Markdown
Contributor Author

The pipeline finally succeeded 🙂

@o-nikolas o-nikolas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My requested changes were addressed and the build is now passing, approving!

@potiuk
potiuk merged commit db10c68 into apache:main Jun 18, 2021
@tooptoop4

Copy link
Copy Markdown
Contributor

@potiuk this doesn't seem to be in any release?

@potiuk

potiuk commented Jul 31, 2021

Copy link
Copy Markdown
Member

https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/index.html#id1 It's in the 2.0.0 providers according to the changelog. Don't you see it there?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants