Skip to content

Provider Airbyte: breaking change, update provider to use Airbyte API Python SDK - #41122

Merged
eladkal merged 28 commits into
apache:mainfrom
marcosmarxm:marcos/airbyte-update-api-auth-method
Aug 28, 2024
Merged

Provider Airbyte: breaking change, update provider to use Airbyte API Python SDK#41122
eladkal merged 28 commits into
apache:mainfrom
marcosmarxm:marcos/airbyte-update-api-auth-method

Conversation

@marcosmarxm

@marcosmarxm marcosmarxm commented Jul 30, 2024

Copy link
Copy Markdown
Contributor

Hello 👋, Airbyte has finally consolidated API access across the open-source, cloud, and enterprise versions.
This means all of them have a single method client_id and client_secret to authenticate to the API now.
This pull request remove the dependency of HttpHook and uses the airbyte-api Python SDK library to execute all operations (create job, cancel job, etc).
The change is a breaking change to users as it removed the api_type parameter as it isn't required anymore.
The new authentication method will become soon the unique method to auth to the API so this is a required changed.
Unit tests were updated to run using the new code.


Comment thread airflow/providers/airbyte/hooks/airbyte.py Outdated
Comment thread airflow/providers/airbyte/sensors/airbyte.py Outdated
Comment thread docs/apache-airflow-providers-airbyte/operators/airbyte.rst Outdated
Comment thread airflow/providers/airbyte/provider.yaml Outdated
Comment thread airflow/providers/airbyte/provider.yaml Outdated
Comment thread airflow/providers/airbyte/hooks/airbyte.py Outdated
@marcosmarxm
marcosmarxm requested a review from eladkal July 31, 2024 17:10
@marcosmarxm

Copy link
Copy Markdown
Contributor Author

@eladkal, can you take another look? Some tests are failing, but I can't determine if it’s due to my changes.

@potiuk

potiuk commented Aug 5, 2024

Copy link
Copy Markdown
Member

If in doubt - rebase:

Screenshot 2024-08-05 at 15 20 56

@marcosmarxm
marcosmarxm force-pushed the marcos/airbyte-update-api-auth-method branch from 9338ffd to 06961bb Compare August 5, 2024 13:52
Comment thread airflow/providers/airbyte/hooks/airbyte.py Outdated
Comment thread airflow/providers/airbyte/hooks/airbyte.py Outdated
Comment thread airflow/providers/airbyte/operators/airbyte.py Outdated
@eladkal eladkal linked an issue Aug 10, 2024 that may be closed by this pull request
2 tasks
Comment thread airflow/providers/airbyte/CHANGELOG.rst Outdated
@marcosmarxm

Copy link
Copy Markdown
Contributor Author

Before merging this change I'll run a final test with Airbye cloud and open source.

@eladkal
eladkal force-pushed the marcos/airbyte-update-api-auth-method branch from 9dd3a18 to 6c40cb1 Compare August 10, 2024 19:15
@eladkal

eladkal commented Aug 10, 2024

Copy link
Copy Markdown
Contributor

Before merging this change I'll run a final test with Airbye cloud and open source.

Please do. If no further comments we are good to go

@potiuk

potiuk commented Aug 16, 2024

Copy link
Copy Markdown
Member

You need to rebase/run pre-commit

@potiuk
potiuk requested a review from eladkal August 19, 2024 17:49
@marcosmarxm

Copy link
Copy Markdown
Contributor Author

Running final test with local deployment and cloud. Gonna 💚 🚀 after finish these tests.

@marcosmarxm
marcosmarxm requested a review from o-nikolas as a code owner August 19, 2024 23:09
@marcosmarxm

Copy link
Copy Markdown
Contributor Author

🟢 Tests are working
🟢 I executed DAGs with a local deployment and cloud, both are working.

Need rebase and 🚀

@marcosmarxm
marcosmarxm force-pushed the marcos/airbyte-update-api-auth-method branch 2 times, most recently from 5bf3e2f to ae2825a Compare August 21, 2024 14:39
@eladkal
eladkal force-pushed the marcos/airbyte-update-api-auth-method branch from c094c21 to b3962a3 Compare August 28, 2024 06:51
@marcosmarxm

Copy link
Copy Markdown
Contributor Author

Wooohooo 🎉 !!!

@eladkal
eladkal merged commit c2d5ef4 into apache:main Aug 28, 2024
@baugarcia

Copy link
Copy Markdown

Great! any ETA on when will the new version be deployed in the python repository? Version is still 3.9.0 there:
https://pypi.org/project/apache-airflow-providers-airbyte/

@marcosmarxm

Copy link
Copy Markdown
Contributor Author

@baugarcia probably the next window of releases is in 3~4 weeks. But keep this to the maintainers to provide a better answer.

@marcosmarxm
marcosmarxm deleted the marcos/airbyte-update-api-auth-method branch September 26, 2024 13:48
@eladkal eladkal mentioned this pull request Feb 27, 2025
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Providers] Airbyte new deployment version changed API endpoints

5 participants