AIP-56 - FAB AM - Move FAB auth manager to new provider

[vincbeck]

It has been decided to keep this task for the end. The purpose of this task is to:

• Decide whether is it feasible and a good idea to move the FAB auth manager from core Airflow to a new provider
• If yes, create a new provider and move the auth manager in this newly created provider
• Make it a dependency of Airflow since it will be used by default

[vincbeck]

I would like to start this discussion. I think everyone agrees that (if not, please call it out!),if possible, we should move the FAB auth manager to a new provider. If I remember well @jedcunningham had some concerns about this move, especially regarding the migrations right? Could you please elaborate a bit more on the challenge? I would like to see if there is no solution to go around that, 'cause I would really love to move this FAB auth manager to providers.

@potiuk

[jedcunningham]

Yeah, my concern was around the db. Things like this and this. Today providers have no way to hook into the db creation/migration process.

[potiuk]

I think it should not be too difficult to add though as a provider capability. I'v done that for configuration it took me few days, and DB migration seems like way simpler problem to solve - especially that we have building blocks in Airflow for that. Migration might be a big tricky - but ultimately we might only benefit from it.

[potiuk]

It literally looks like a hook wher you can discover "if" a given provider supports migration and hook it in, and a way to query providers to see if they "need" migration (to account for independent upgrades/downgrades of providers from core) + handle back-compatibility - we aalredy have the DB quite separated - we know which tables belong to FAB and which not,.

Alembic alredy supports separate "migration environments' I believe (I am not too experienced with Alembic though). It would be a bit tricky to move before/ after migration but since FAB provider will likely come pre-installed, we might not worry too much about back-compat (i.e. airflow >= 2.8 for provider and mandtory FAB provider for Airflow should solve the back-compatibility problem nicely).

[jedcunningham]

Unfortunately I think there is a bit more complexity here. The db cli is very "core" centric right now. We'd have to support downgrade --from-version, --to-version and friends, for not only core but all providers that do db things. Sure, a bare db upgrade is easy but it's also the simplest case. It's definitely solvable though.

If the FAB provider will be pre-installed, does moving this stuff to a provider really buy us anything? Even if you didn't use it, you'd still get the db stuff and code anyways (unless you went off the beaten path of course).

[potiuk]

Unfortunately I think there is a bit more complexity here. The db cli is very "core" centric right now. We'd have to support downgrade --from-version, --to-version and friends, for not only core but all providers that do db things. Sure, a bare db upgrade is easy but it's also the simplest case. It's definitely solvable though.

If the FAB provider will be pre-installed, does moving this stuff to a provider really buy us anything? Even if you didn't use it, you'd still get the db stuff and code anyways (unless you went off the beaten path of course).

Yes. There are multiple benefits of having FAB as provider even if it is preinstalled.

For example it would be possible to remove it more easly in next versions - we might make NOT preinstalled in 2.9 for example.

Right now when we moved daskexecutor to provider, we KNOW we can move it out. Also when FAB is in provider we can potentially suspend it's development, tests and releases when we decide that we do not want to maintain it any more. This might mean for example that we stop developing it main and it does not hold us back (people might simply continue using the version that is released already.

But there is way more. It will NOT keep airflow tied to specific version of FAB - we might have 5 versions of FAB provider tied to specific FAB version - each of them linked to a different FAB version by pinning. Which might unblock some scenarios. For example people might want to upgrade to latest FAB version because of security issue while something might still be keeping them with older Airflow version.

Or the opposite - for some reasons (changed mechanism of OAUT authentication as happened in the past) people might want to stick to old FAB while upgrading to newer Airflow. We never really told that to people - but in Airflow 2 we actually HAD breaking changes becasue FAB changed the way how you configure OAUTH integration. It broke a number of people integratiosn in 2.2 or 2.3. Having FAB provider separately we could make a breaking release of provider while allowing people to stay with the old version of provider and still upgrade to newer version of Airflow.

So yeah. There are plenty of benefits of splitting when it comes to a number of different scenarios for our users (most of them happened in the past so well it's not theorethical).

And really whether we eventually make FAB truly optional is another story but we once we do separate it to provider, it just opens up all kinds of options we might have and know what it brings. If we won't do it, we cannot even start a discussion about it becase just not knowing what it involves (like this migration for example) will hold us back from having discussion on whether we want to do it at all or not and what benefits it will bring.

[potiuk]

Also. purely from the speed of delivering security fixes (which we had plenty of in FAB). We could also handle bugfixes faster - we would not have to wait with releasing a security release for the next airflow version - it could just come in the next provider's wave.

[potiuk]

(I just added it to "security improvements" project because of the last point cc: @hussein-awala @pierrejeambrun @eladkal)

[potiuk]

Unfortunately I think there is a bit more complexity here. The db cli is very "core" centric right now. We'd have to support downgrade --from-version, --to-version and friends, for not only core but all providers that do db things. Sure, a bare db upgrade is easy but it's also the simplest case. It's definitely solvable though.

Yeah. I think we would have to have a separate airflow providers db CLI + I think only the default migration should automically call the "provider's" one, the "from/to" migration would only do the "core" or "specific provider" migration (for airflow providers db <provider>) This would have to be added (like the few things I added for config) but mostly mimicking what the main db command is doing. By building blocks I mean we know already what are the use cases and options, and we have the code, so implementing similar approach by either refactoring and DRY'ing the code or by copying bits and pieces should be way easier than doing that all from the scratch.

[jedcunningham]

Fair point on being able to decouple core from a specific FAB version. I hadn't considered that aspect, and that does make it worthwhile.

I worry that having airflow db migrate do core + providers, but the rest just core will be confusing. I'm sure we can come up with something that makes sense though.

[hussein-awala]

Also. purely from the speed of delivering security fixes (which we had plenty of in FAB). We could also handle bugfixes faster - we would not have to wait with releasing a security release for the next airflow version - it could just come in the next provider's wave.

+1

For the db command, I think it will be complex to ensure compatibility between all the combinations (Airflow since min version, FAB provider version), so I wonder if for this new proposed provider, we can apply a different strategy for min_airflow_version, by updating it on each release to latest Airflow version, in this case we can benefit from the frequent releases, without worrying about backward compatibility.