AWS DMS Serverless Operators - #43986
Closed
ellisms wants to merge 802 commits into
Closed
Conversation
This table was previously only used in Task Duraton Legacy view. Follow-up of apache#38339 (review)
This reverts commit c3254a3.
* Add type hints to DAG test utils * Remove unused fixture usage * First type hint user
* Run only ui tests for PR with new UI only changes. * Revert test change to UI folder.
* Add latest dag run info to dags list and dag details * Add tooltip offset
* Migrate public endpoint Get DAG Source to FastAPI, with main resynced Co-authored-by: pierrejeambrun <pierrejbrun@gmail.com> * Re-run static checks --------- Co-authored-by: pierrejeambrun <pierrejbrun@gmail.com>
closes apache#43011 By "definition code" we mean anything needed at definition/parse time, leaving anything to do with scheduling time decisions in Airflow's core. Also in this PR I have _attempted_ to keep it to only porting defintiion code for simple DAGs, leaving anything to do with mapped tasks or execution time in core for now, but a few things "leaked" across. And as the goal of this PR is to go from working state to working state some of the code in Task SDK still imports from "core" (various types, enums or helpers) that will need to be resolved before 3.0 release, but it is fine for now. I'm also aware that the class hierarchy with airflow.models.baseoperator.BaseOperator (and to a lesser extend with DAG) in particular is very messy right now, and we will need to think how we want to add on the scheduling-time functions etc, as I'm not yet sold that having Core Airflow depend upon the Task-SDK classes/import the code is the right structure, but we can address that later We will also need to addresses the rendered docs for the Task SDK in a future PR -- the goal is that "anything" exposed on `airflow.sdk` directly is part of the public API, but right now the renedered docs show DAG as `airflow.sdk.definitions.dag.DAG` which is certainly not what we want users to see. Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
…eeded. (apache#43512) * Run only ui tests for PR with new UI only changes. * Revert test change to UI folder. * Skip integration tests for new UI changes only PR. * Build docs only when needed. Move new UI files only test to a separate attribute for more enhancements. * Fix attribute. * Add docs-build to pass it to build. * Add docs-build in ci.yml to pass it. * Fix outputs.
Fixing one of the TODOs: apache#43076 This was already addressed.
* Keycloak Breeze integration Work is starting on multi-team Airflow, and this project has many dependencies on a auth manager that can support authn and authz and also support the changes to the auth manager api (upcoming). An option for this is Keycloak. This PR adds a Breeze integartion for Keycloak which creates a container running keycloak, which uses the existing Postgres container as the DB and disables the requirements for ssl/https/certificates to use the Keycloak admin console (since this is only for development purposes not production).
…apache#43526) It was only used in S3 To Hive module. One less thing in the `utils` directory.
The code that used it was removed in apache#37988 already. It was used in FAB views.
This was missed in apache#43490
…3530) These functions have been deprecated from a long time now.
Not sure why this was deferred in https://github.com/apache/airflow/pull/39336/files#r1594440137 but I can't find any usages of this in the codebase.
* ignore spell check Uncompress * change to lowercase
…ontrol.rst` (apache#43495) * fix doc error for provider fab * Update docs/apache-airflow-providers-fab/auth-manager/access-control.rst --------- Co-authored-by: Elad Kalif <45845474+eladkal@users.noreply.github.com>
When we generated list of commits for provider's documentation, we only checked for commits from the source folders of the providers, however there were some commits that only changed documentation for the provider and such commits were not included. This PR adds documentation folder to be used while checkign for the commit list, and next time when documentation will be generated for providers, the doc-only commits should also be included for all historical versions of providers.
…43529) When providers have been moved in apache#42505 broke passing parameters when provider tests were passed as extra args of "testing tests" command of breeze. Previously all tests were under "tests" folder and there was an exclusion to disable "All" tests when any test was passed as parameter. But after moving tests to "providers" this stopped working. Additional exclusion needs to be added for "providers/tests" and "providers/tests_sdk/". This PR also adds autocompletion for tests passed this way by setting the click type to Path for the extra args (but without the need for the Path to exist). Also during this check it turned out that "All" tests are not working in the intended way - but this should not impact our CI only local runs. Appropriate comment has been added and it's captured in apache#42632
* AIP-84 Get Task Instance * Add TaskInstanceCollectionResponse * Add more tests * Update following code review
* Prepare docs for Oct 2nd wave of providers RC2 * fix
* add core.time provider * add source-date-epoch * change core to essentials * revert external task sensor location * add provider to airflow_providers_bug_report list * change new provider name to standard * add integration * revert hatch_build * move examples back to airflow core * change sensors example dags paths * remove init * revert howto docs * change provider as not-ready * Move python operator to standard provider * fix test failures * move python operator inside standard/operators * rebase and update python imports * keep standard provider in hatch_build * update selective checks and dependencies * update imports and dependency file * fix compat import errors * fix selective_checks and imports from compat * remove imports from compat for pythonoperator * update run_tests.py python branching test paths * add setup method in python operator tests * check imports in python operator tests * fix imports and move python virtual env scripts inside standard provider * update selective checks tests and imports in test_python * move python_virtualenv tests inside standard provider * fix pre-commit static checks * use compat provider for standard provider imports * fix _SERIALIZERS ModuleNotFoundError and static checks * import _SERIALIZERS for AIRFLOW_2_10 or higher versions * fix ruff format * fix static checks in test_dag * fix test_provide_context_does_not_fail failure: add TypeError to pytest assert --------- Co-authored-by: romsharon98 <rom.sharon98@gmail.com>
* Add logging to the migration commands This PR enhances logging for the migration commands and ensures that alembic doesn't change the logger when logging. A PR was raised to address the issue of reset db command changing logging level; see f166467, but that did not address it for other commands. This PR addresses it for all the migration commands * fixup! Add logging to the migration commands * Apply suggestions from code review Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> --------- Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
`parse_execution_date` was just calling `timezone.parse`, there isn't a need for it to be a separate function. `round_time` wasn't used anywhere
…apache#43954) * Fixed seeking issue with none utf-8 signs * Changed version and changelog * Removed hard utf-8 decoding --------- Co-authored-by: Marco Küttelwesch <marco.kuettelwesch@de.bosch.com>
The Pinecone partners team asked if it would be possible to add an attribution tag to the provider to track how much SDK usage they get through it.
ellisms
requested review from
Fokko,
ashb,
bolkedebruin,
dstandish,
eladkal,
ephraimbuddy,
hussein-awala,
jedcunningham,
josh-fell,
jscheffl,
kaxil,
mobuchowski,
o-nikolas,
pierrejeambrun,
potiuk and
uranusjr
as code owners
November 13, 2024 21:06
boring-cyborg
Bot
added
the
area:production-image
ellisms
requested review from
XD-DENG,
bbovenzi and
ryanahamilton
as code owners
November 13, 2024 21:06
Contributor
Author
|
Looks like something went bad in my rebase. I did not change 5000+ files. I need to look into it. |
Contributor
|
Something is "wrong" with this PR :-D |
Contributor
|
New PR: #43988 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adding Operators for DMS Serverless replications. Added required triggers and waiters.
closes #39954