Add a way to skip the secret_backend

[raphaelauv]

Description

I use the gcp secret_manager as a secret_backend

2 problems :

• the implementation always first look for the secret_backend before trying the airflow variables , no way to skip the check to the secret_backend

something like

Variable.get("totot",skip_secret_backend=True)

so change variable.py

    @classmethod
    def get(
        cls,
        key: str,
        default_var: Any = __NO_DEFAULT_SENTINEL,
        deserialize_json: bool = False,
        skip_secret_backend: bool = False,
    ) -> Any:

and also change the macro

{{ var.value.get('my.var', 'fallback') }}

• every variable that is not in the secret_backend but in the airflow variable will produce an ERROR log line , for some dag is really confusing to see at every run :

[2021-10-27 10:16:19,103] {secret_manager_client.py:93} ERROR - Google Cloud API Call Error (PermissionDenied): No access for Secret ID airflow-prod-variable-XXXXXX-XXXXX.
                Did you add 'secretmanager.versions.access' permission?

Replace the log level ERROR to WARNING would be better since we don't know if the secret do not exist or if it's really a problem of access permission.

Use case/motivation

Every Variable.get make a call to the secret_backend , would be great to make it configurable ( to first control the cost and the load on the secret_backend )

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

Yep. I also thought this might be a good idea. I often found that things would be simpler this way, and there is a huge potential of optimizing the traffic to secret backends. I tihnk the ratio of "secret" variables vs. "non-secret ones" are like 1-100 and seems that we have to pay the price of roundtrip to secret backends every time we access it. Of course, we then have to go to DB instead, but if the secret backend does not contain the variable we do it anyway, so we can save a lot by adding a context "this variable is not supposed to be looked up in context"

And yeah - cost for accessing secrets also matters.

@kaxil - WDYT ? I think you were the master-mind behind the secrets implementation, do you see any obvious problems with this approach ? I cannot see any "bad scenario" here.

[kaxil]

Yeah I was thinking of this sort of feature, I think I described in one of the issues too.

Instead of skip, should we instead include what backend it should look at? No flaw for either of the features though. And for sure they would be useful.

cc @fhoda

[raphaelauv]

I implemented the skip option in the PR , I could also implement the select option

But what should be the augment string give by the user ?

The name of the class of the secret_back to only look to?

Like

Variable.get("toto",look_in="MetastoreBackend")

[potiuk]

I personally think skip is a bit more "straightforward" and simple. Using more than one secret backend is not supported currently and we would have to implement the capability of definining multiple backands (with multiple configs) for the "look_in" type of setting to make sense.

I think the most "common" use cases are where secrets are kept in "secret" backend and all other variables are kept in metastore, so having just "skip_external_backend" makes sense (however I'd rename the parameter to 'skip_secret_backend` to be more precise.

[raphaelauv]

ok , current implementation is only with the skip_*** arg

(but never skipping EnvironmentVariablesBackend, LocalFilesystemBackend and MetastoreBackend )

[ashb]

Do you want to skip the secrets backend for all variables, or just a specific one?

[ashb]

This is already possible at the secrets backend configuration

airflow/airflow/providers/google/cloud/secrets/secret_manager.py

Lines 60 to 62 in ae04488

	:param variables_prefix: Specifies the prefix of the secret to read to get Variables.
	If set to None (null), requests for variables will not be sent to GCP Secrets Manager
	:type variables_prefix: str

[raphaelauv]

no you didn't understood @ashb

I need to keep the secret_backend but I don't want airflow to try to read ALL the variables from it all the times

I want an option to skip the secret_backend lookup for some variables ( the vast majority like said @potiuk )

[ashb]

You want to use variables from the secret backend for some variables but not others?

[potiuk]

I would not close @ashb, but rather discuss how we can implement, I think there are varying opinions here. At the very least, if others also have such strong opinion as you - we should document how to handle this - very common it seems - use case.

I think we still can discuss how to do it - and maybe we should do it differently - but I think the use/case and rationale is very-sound and - more than that - pretty common. It has been raised quite a few times by different people.

Since we have connections where most people keep their secret passwords, I think the ratio of secret variables vs. the non-secret ones is small and this use case is very common. I'd say more often than not people will have connections kept in secret, but they will have no secret variables. We are basically telling people to pay more for their secret backends and we do not even tell them how to avoid this. And I think making people write their onwn secret backend to handle such case is kinda over-the-top - especially that it seems common pattern for all the different secret backends. And our users do not often realise, that they could do their own implementation very easily.

I actually see the point of the "different access" pattern that @ashb and I agree it is not perfect. I think what is even bigger problem is that it's the "DAG" writers to decide how each variable should be retrieved, which - I quite agree - is very prone to different kinds of problems

But maybe we can find a a good solution that serves the use case but does not introduce the "different access" pattern. I

Maybe a good solution will be to give them ready-to use simple implementation of such custom backend that you could "MixIn" with any other secret backend where for example chosing whether to use secret or not would be done based on Regexp match of the variable name?

Maybe simply have a possibility to define an optional callable configurable in secrets group, that will get the "name" and "type" (variable/connection/config) of the retrieved object and return true/false if it should use secret ? That could also give an opportunity to handle a different case we have currently problem with. Currentlly when someone tries to set a variable whcih is already available as "secret". this variable is set in the Metadatada DB (and not accessible). You do not get any warning or error when you do so, which might lead to really strange problems.

If the installation has this "callable" defined however, that could be much better, because we could check whether the variable should be read from secret and if so - we would just fail such an attempt.

I think that would be a really nice and consistent approach.

WDYT everyone?