Status of testing Providers that were prepared on September 14, 2023

[eladkal]

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

The guidelines on how to test providers can be found in

Verify providers by contributors

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 8.7.1rc1

• Bugfix: Fix RDS triggers parameters so that they handle serialization/deserialization (#34222): @vandonr-amz
• Use a AwsBaseWaiterTrigger-based trigger in EmrAddStepsOperator deferred mode (#34216): @yermalov-here
• Remove unused parameter cluster_role_arn from EksPodOperator's docstring (#34300): @Taragolis
• Correct parameter names in docstring for S3CreateObjectOperator (#34263): @mjsqu
• Simplify 'True if ... else False' to bool(...) (#34258): @pankajkoti

Provider cncf.kubernetes: 7.6.0rc1

• Add progress_callback parameter to KubernetesPodOperator (#34153): @zeotuan
• Move definition of Pod*Exceptions to pod_generator (#34346): @potiuk
• Push to xcom before KubernetesPodOperator deferral (#34209): @vchiapaikeo

Provider google: 10.9.0rc1

• Stream (realtime) pipelines for CloudDataFusionStartPipelineOperator (#34271): @moiseenkov
• Add expected_terminal_state parameter to Dataflow operators (#34217): @shahar1
• Fix ComputeEngineInsertInstanceOperator doesn't respect jinja-templated instance name when given in body argument (#34171): @aryehgigi
• fix: BigQuery job error message (#34208): @nathadfield
• fix(providers-google): gcp_conn_id parameter ignored in GKEStartPodOperator hook property (#34194): @mikaeld
• Bump min common-sql provider version for Google provider (#34257): @pankajkoti
• Remove unnecessary call to keys() method on dictionaries (#34260): @pankajkoti
• Deprecate AutoMLTrainModelOperator for NL (#34212): @VladaZakharova
• Simplify 'True if ... else False' to bool(...) (#34258): @pankajkoti
• Google Dataform operators: Update template_fields (#34187): @retsilla

All users involved in the PRs:
@potiuk @shahar1 @vandonr-amz @pankajkoti @Taragolis @retsilla @zeotuan @aryehgigi @VladaZakharova @mikaeld @vchiapaikeo @nathadfield @mjsqu @moiseenkov @yermalov-here

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[mjsqu]

#34263 - just a change to docstrings - no functional impact

[yermalov-here]

#34216 works as expected

[vchiapaikeo]

#34209 continues to work as expected. Thank you @eladkal!

[retsilla]

#34187 - works as expected.

[potiuk]

All good for the cncf.k8s fix.

[pankajkoti]

Checked that #34258, #34260 are included in the RCs. They are minor refactoring and no functional changes.

While testing the sources for the Google RC wrt PR #34257 , I see the dependences apache-airflow-providers-common-sql>=1.7.2.dev0, apache-airflow>=2.4.0.dev0 have a dev0 suffix in the setup.cfg file. Is that expected for RCs? I see apache-airflow has a dev0 version suffix in the Amazon RC sources too in its setup.cfg file.
Whereas for earlier released packages e.g. for the Google provider 10.8.0 it does NOT have a dev0 suffix on the Airflow version in its setup.cfg file.

setup.cfg for Google RC 10.9.0rc1

setup.cfg for earlier released Google provider 10.8.0

[potiuk]

While testing the sources for the Google RC, I see the dependences apache-airflow-providers-common-sql>=1.7.2.dev0, apache-airflow>=2.4.0.dev0 have a dev0 suffix in the setup.cfg file. Is that expected for RCs? I see apache-airflow has a dev0 version suffix in the Amazon RC sources too in its setup.cfg file.
Whereas for earlier released packages e.g. for the Google provider 10.8.0 it does NOT have a dev0 suffix on the Airflow version in its setup.cfg file.

Yes. This is all expected (providing that your observations are about the RC packages you downloaded from PyPI).

I guess you downloaded the packages from PyPI - but those are not the packages that PMC members verify and vote on. They are "Convenience" to test and install the packages from PyPI but they are NOT the same as packges we vote on.

The RC packages in PyPI are different than the one we have in https://dist.apache.org/repos/dist/dev/airflow/providers/ - they differ by the RC* suffix and .dev0 dependencies. The packages in https://dist.apache.org/repos/dist/dev/airflow/providers/ are the very same files we will eventually upload as "x.y.z" packages to PyPI though.

There are several different reasons for that process/differences:

1. In PyPI you cannot delete and re-upload package with the same version - so for PyPI testing we need to submit RC packages. It is impossible to have same package uploaded as RC and final version. So we need to slightly (dynamically) modify the packages when we are preparing them for PyPI (but only for RC candidates).

2. PIP does not recognize x.y.z.dev0 as >=x.y.z. This means that if we have two packages released at the same time and one depends on the other (happens) we need to dynamically add x.y.z.dev0 -> this will make sure that we can install the new packages from PyPI (otherwise they would comply that there is no >=x.y.z package. That's why for all RCs we add .dev0 dynamically for every dependency where we depend on any apache-airflow package. Otherwise we would have to first relase (vote and publish) the dependent package and only after the the package that depends on it. Ading .dev0 in PyPI RCS allows to vote and release (and install from PyPI) such cross-dependent packages together.

3. Finally the packages we are voting on (because we do not vote on PyPI packages - we vote on the packages in https://dist.apache.org/repos/dist/dev/airflow/providers/ ) do not contain neither RC nor .dev0, because those are the very packages that we will move to https://downloads.apache.org/airflow/providers/ . They are THE SAME packages - with checksums and signatures etc. So the packages we VOTE on already have final versions, and final dependencies, because those are the packages that will become x.y.z if they pass the voting.

So yeah - good points, but everything is perfectly as expected.

[vandonr-amz]

I'm go.

[zeotuan]

#34153 - Works as expected

[moiseenkov]

#34271 and #34212 work as expected

[nathadfield]

Hello, Confirmed that #34208<#34208> works as expected. Cheers, Nathan