Apache Airflow Provider(s)
salesforce
Versions of Apache Airflow Providers
apache-airflow-providers-salesforce==1!3.2.0
Apache Airflow version
2.1.4
Operating System
Debian GNU/Linux 10 (buster)
Deployment
Astronomer
Deployment details
No response
What happened
Right now the SalesforceHook is requiring all of the extras for a Salesforce connection type in Airflow.
I am assuming this issue was never brought up before because users of this hook have been using the Airflow UI to make connections (which presents all extra fields), however with things such as Secrets Backends, it should be possible to set up a Salesforce connection URI without having to explicitly provide all of the extras.
The issue is the hook's author designed the hook's get_conn to pass in every extra value using an improper/invalid method of defaulting to None by using or None but still referencing the extras key directly. I tested this in as low as Python 2.7 and as high as Python 3.7 and in both versions if any one of these extras are not provided you will get a KeyError
|
security_token=extras["extra__salesforce__security_token"] or None, |
|
domain=extras["extra__salesforce__domain"] or None, |
|
session_id=self.session_id, |
|
instance=extras["extra__salesforce__instance"] or None, |
|
instance_url=extras["extra__salesforce__instance_url"] or None, |
|
organizationId=extras["extra__salesforce__organization_id"] or None, |
|
version=extras["extra__salesforce__version"] or api.DEFAULT_API_VERSION, |
|
proxies=extras["extra__salesforce__proxies"] or None, |
|
session=self.session, |
|
client_id=extras["extra__salesforce__client_id"] or None, |
|
consumer_key=extras["extra__salesforce__consumer_key"] or None, |
|
privatekey_file=extras["extra__salesforce__private_key_file_path"] or None, |
|
privatekey=extras["extra__salesforce__private_key"] or None, |
What you expected to happen
Any extras value not provided should just default to None (or to api.DEFAULT_API_VERSION for the "version" extra). It should be possible to set up a Salesforce connection URI using a secrets backend without having to explicitly provide all of the extras.
How to reproduce
Set up a secrets backend (such as via environment variable) and pass in the minimum connection values needed for "Password" connection type:
(this is an example, no real passwords shown)
export AIRFLOW_CONN_SALESFORCE_DEFAULT='http://your_username:your_password@https%3A%2F%2Fyour_host.lightning.force.com?extra__salesforce__security_token=your_token'
It will error with KeyError: 'extra__salesforce__domain' and keep resulting in key errors for each extras key until you finally provide all extras, like so:
(this is an example, no real passwords shown)
export AIRFLOW_CONN_SALESFORCE_DEFAULT='http://your_username:your_password@https%3A%2F%2Fyour_host.lightning.force.com?extra__salesforce__security_token=your_token&extra__salesforce__domain=&extra__salesforce__instance=&extra__salesforce__instance_url=&extra__salesforce__organization_id=&extra__salesforce__version=&extra__salesforce__proxies=&extra__salesforce__client_id=&extra__salesforce__consumer_key=&extra__salesforce__private_key_file_path=&extra__salesforce__private_key='
Anything else
In addition to this, the SalesforceHook should also accept extras without the need for the extra__salesforce__ prefix, like many other connections do.
Are you willing to submit PR?
Code of Conduct
Apache Airflow Provider(s)
salesforce
Versions of Apache Airflow Providers
apache-airflow-providers-salesforce==1!3.2.0
Apache Airflow version
2.1.4
Operating System
Debian GNU/Linux 10 (buster)
Deployment
Astronomer
Deployment details
No response
What happened
Right now the
SalesforceHookis requiring all of the extras for a Salesforce connection type in Airflow.I am assuming this issue was never brought up before because users of this hook have been using the Airflow UI to make connections (which presents all extra fields), however with things such as Secrets Backends, it should be possible to set up a Salesforce connection URI without having to explicitly provide all of the extras.
The issue is the hook's author designed the hook's
get_connto pass in every extra value using an improper/invalid method of defaulting to None by usingor Nonebut still referencing the extras key directly. I tested this in as low as Python 2.7 and as high as Python 3.7 and in both versions if any one of these extras are not provided you will get aKeyErrorairflow/airflow/providers/salesforce/hooks/salesforce.py
Lines 137 to 149 in e9a72a4
What you expected to happen
Any extras value not provided should just default to None (or to
api.DEFAULT_API_VERSIONfor the "version" extra). It should be possible to set up a Salesforce connection URI using a secrets backend without having to explicitly provide all of the extras.How to reproduce
Set up a secrets backend (such as via environment variable) and pass in the minimum connection values needed for "Password" connection type:
(this is an example, no real passwords shown)
export AIRFLOW_CONN_SALESFORCE_DEFAULT='http://your_username:your_password@https%3A%2F%2Fyour_host.lightning.force.com?extra__salesforce__security_token=your_token'It will error with
KeyError: 'extra__salesforce__domain'and keep resulting in key errors for each extras key until you finally provide all extras, like so:(this is an example, no real passwords shown)
export AIRFLOW_CONN_SALESFORCE_DEFAULT='http://your_username:your_password@https%3A%2F%2Fyour_host.lightning.force.com?extra__salesforce__security_token=your_token&extra__salesforce__domain=&extra__salesforce__instance=&extra__salesforce__instance_url=&extra__salesforce__organization_id=&extra__salesforce__version=&extra__salesforce__proxies=&extra__salesforce__client_id=&extra__salesforce__consumer_key=&extra__salesforce__private_key_file_path=&extra__salesforce__private_key='Anything else
In addition to this, the
SalesforceHookshould also accept extras without the need for theextra__salesforce__prefix, like many other connections do.Are you willing to submit PR?
Code of Conduct