Description
Can we add the option to configure the Retry policy of the kubernetes CoreV1Api? Or set it to default have some more resilient configuration.
Today it appears to retry operations 3 times but with 0 backoff in between each try. Causing temporary network glitches to result in fatal errors.
Following the flow below:
airflow.kubernetes.kube_client.get_kube_client()
Calls load_kube_config() without any configuration set, this assigns a default configuration with retries=None to CoreV1Api.set_default()
1b. Creates CoreV1Api() with api_client=None
1c. ApiClient() default constructor creates a default configuration object via Configuration.get_default_copy(), this is the default injected above- On request, through some complicated flow inside
ApiClient and urllib3, this configuration.retries eventually finds its way into urllib HTTPConnectionPool, where if unset, it uses urllib3.util.Retry.DEFAULT, this has a policy of 3x retries with 0 backoff time in between.
Configuring the ApiClient would mean changing the get_kube_client() to something roughly resembling:
client_config = Configuration()
client_config.retries = Retry(total=3, backoff=LOAD_FROM_CONFIG)
config.load_kube_config(...., client_configuration=client_config)
apiclient = ApiClient(client_config)
return CoreV1Api(apiclient)
I don't know myself how fine granularity is best to expose to be configurable from airflow. The retry object has a lot of different options, so do the rest of the kubernetes client Configuration object. Maybe it should be injected from a plugin rather than config-file? Maybe urllib or kubernets library have other ways to set default config?
Use case/motivation
Our Kubernetes API server had some unknown hickup for 10 seconds, this caused the Airflow kubernetes executor to crash, restarting airflow and then it started killing pods that were running fine, showing following log: "Reset the following 1 orphaned TaskInstances"
If the retries would have had some backoff it would have likely survived this hickup.
See attachment for the full stack trace, it's too long to include inline. Here is the most interesting parts:
2022-06-29 21:25:49 Class={kubernetes_executor.py:111} Level=ERROR Unknown error in KubernetesJobWatcher. Failing
...
2022-06-29 21:25:49 Class={connectionpool.py:810} Level=WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fbe35de0c70>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/default/pods/REDACTED
2022-06-29 21:25:49 urllib3.exceptions.ProtocolError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
2022-06-29 21:25:49 Class={connectionpool.py:810} Level=WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fbe315ec040>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/default/pods/REDACTED
2022-06-29 21:25:49 Class={connectionpool.py:810} Level=WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fbe315ec670>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/default/pods/REDACTED
...
2022-06-29 21:25:50 Class={kubernetes_executor.py:813} Level=INFO Shutting down Kubernetes executor
...
2022-06-29 21:26:08 Class={scheduler_job.py:696} Level=INFO Starting the scheduler
...
2022-06-29 21:27:29 Class={scheduler_job.py:1285} Level=INFO Message=Reset the following 1 orphaned TaskInstances:
airflowkubernetsretrycrash.log
From airflow version 2.3.2
Related issues
No response
Are you willing to submit a PR?
Code of Conduct
Description
Can we add the option to configure the Retry policy of the kubernetes CoreV1Api? Or set it to default have some more resilient configuration.
Today it appears to retry operations 3 times but with 0 backoff in between each try. Causing temporary network glitches to result in fatal errors.
Following the flow below:
airflow.kubernetes.kube_client.get_kube_client()Calls
load_kube_config()without any configuration set, this assigns a default configuration withretries=NonetoCoreV1Api.set_default()1b. Creates
CoreV1Api()withapi_client=None1c.
ApiClient()default constructor creates a default configuration object viaConfiguration.get_default_copy(), this is the default injected aboveApiClientand urllib3, thisconfiguration.retrieseventually finds its way into urllibHTTPConnectionPool, where if unset, it usesurllib3.util.Retry.DEFAULT, this has a policy of 3x retries with 0 backoff time in between.Configuring the ApiClient would mean changing the
get_kube_client()to something roughly resembling:I don't know myself how fine granularity is best to expose to be configurable from airflow. The retry object has a lot of different options, so do the rest of the kubernetes client Configuration object. Maybe it should be injected from a plugin rather than config-file? Maybe urllib or kubernets library have other ways to set default config?
Use case/motivation
Our Kubernetes API server had some unknown hickup for 10 seconds, this caused the Airflow kubernetes executor to crash, restarting airflow and then it started killing pods that were running fine, showing following log: "Reset the following 1 orphaned TaskInstances"
If the retries would have had some backoff it would have likely survived this hickup.
See attachment for the full stack trace, it's too long to include inline. Here is the most interesting parts:
airflowkubernetsretrycrash.log
From airflow version 2.3.2
Related issues
No response
Are you willing to submit a PR?
Code of Conduct