You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Mar 6, 2026. It is now read-only.
The gRPC documentation for AuthMetadataPlugin.__call__ states: "Implementations of this method must not block." Unfortunately, google.auth.transport.grpc.AuthMetadataPlugin violates this requirement. I believe this causes all gRPC calls using one instance of AuthMetadataPlugin to be blocked as long as token refreshing is happening. This may be causing occasional errors where the timeout we pass to Google Cloud API calls is not being respected, since the calls are all blocked waiting on the OAuth token to be refreshed.
Possible fix
It seems likely this plugin needs to do OAuth token refreshing using a thread pool, like the grpc._auth.GoogleCallCredentials implementation does. This will have the side benefit of not having multiple threads "race" to refresh the credentials quite as much.
A quick-and-dirty "fix" that would reduce the impact of this error would be to add a timeout on the token refreshing requests. This is probably a good idea anyway. However, this would at least ensure the "impact" of slow/blocked token refreshes is minimized. On our case, we have seen processes that may have been "stuck" for ~15 minutes due to this issue.
Blocking call path
When using google.api_core.grpc_helpers.create_channel, which seems to be what is used by Google's generated GAPIC clients:
Calls self._credentials.before_request(...) which is google.auth.credentials.Credentials.before_request
If the credentials have expired, it calls self.refresh(...). In the case of service accounts, this is a google.oauth2.service_account.Credentials object.
Calls _client.jwt_grant(...) which calls _token_endpoint_request(...)
Calls request(...) which is an instance of google.auth.transport.requests.Request
This calls in to the requests API, which eventually makes a blocking HTTP request.
The gRPC documentation for
AuthMetadataPlugin.__call__states: "Implementations of this method must not block." Unfortunately,google.auth.transport.grpc.AuthMetadataPluginviolates this requirement. I believe this causes all gRPC calls using one instance of AuthMetadataPlugin to be blocked as long as token refreshing is happening. This may be causing occasional errors where the timeout we pass to Google Cloud API calls is not being respected, since the calls are all blocked waiting on the OAuth token to be refreshed.Possible fix
It seems likely this plugin needs to do OAuth token refreshing using a thread pool, like the
grpc._auth.GoogleCallCredentialsimplementation does. This will have the side benefit of not having multiple threads "race" to refresh the credentials quite as much.A quick-and-dirty "fix" that would reduce the impact of this error would be to add a timeout on the token refreshing requests. This is probably a good idea anyway. However, this would at least ensure the "impact" of slow/blocked token refreshes is minimized. On our case, we have seen processes that may have been "stuck" for ~15 minutes due to this issue.
Blocking call path
When using
google.api_core.grpc_helpers.create_channel, which seems to be what is used by Google's generated GAPIC clients:google.auth.transport.grpc.AuthMetadataPlugin.__call__(...)google.auth.transport.grpc.AuthMetadataPlugin._get_authorization_headers(...)self._credentials.before_request(...)which isgoogle.auth.credentials.Credentials.before_requestself.refresh(...). In the case of service accounts, this is agoogle.oauth2.service_account.Credentialsobject._client.jwt_grant(...)which calls_token_endpoint_request(...)request(...)which is an instance ofgoogle.auth.transport.requests.RequestrequestsAPI, which eventually makes a blocking HTTP request.