DefaultCredentialsError intermittently in google-cloud-storage running in Cloud Dataflow

[dmho418]

I'm having an issue similar to #2995

I'm calling the google-cloud-storage API, after a few thousand successful calls, it suddenly crashes with the following error:

  File "/usr/local/lib/python2.7/dist-packages/google/cloud/datastore/client.py", line 204, in __init__
    project=project, credentials=credentials, _http=_http)
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/client.py", line 212, in __init__
    Client.__init__(self, credentials=credentials, _http=_http)
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/client.py", line 125, in __init__
    credentials, _ = google.auth.default()
  File "/usr/local/lib/python2.7/dist-packages/google/auth/_default.py", line 286, in default
    raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)
DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or
explicitly create credential and re-run the application. For more
information, please see
https://developers.google.com/accounts/docs/application-default-credentials.

Dataflow auto-retries the operation a few times, but it also fails all the retries.

[dhermes]

That's very interesting! Is it safe to assume you are setting credentials via the GOOGLE_APPLICATION_CREDENTIALS env. var?

We should probably move this over to the https://github.com/GoogleCloudPlatform/google-auth-library-python/ issue tracker?

[dmho418]

Could be something in the auth library, could also be related to GCE.
Cloud Dataflow creates and manages these VMs automatically, and they are all set with default credentials by default.
I limited the pipeline to 1 VM and I noticed one eerie coincidence with #2995 . It always fails on the 4th time the program creates a cloud storage client with storage.Client(project=project)

Interestingly, the problem doesn't happen when I run locally on my computer.

Versions:
google-cloud-storage-1.6.0
google-auth-1.2.0

[dhermes]

@dmho418 I see. I'd bet that it's failing the request to the GCE metadata server (and all other methods for "auto-detecting" the credentials).

[dmho418]

Seems to be the case. I moved some things around and now I'm getting the following error on the 4th call:

File "/usr/local/lib/python2.7/dist-packages/google/cloud/storage/blob.py", line 946, in upload_from_file
    client, file_obj, content_type, size, num_retries)
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/storage/blob.py", line 867, in _do_upload
    client, stream, content_type, size, num_retries)
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/storage/blob.py", line 700, in _do_multipart_upload
    transport, data, object_metadata, content_type)
  File "/usr/local/lib/python2.7/dist-packages/google/resumable_media/requests/upload.py", line 97, in transmit
    retry_strategy=self._retry_strategy)
  File "/usr/local/lib/python2.7/dist-packages/google/resumable_media/requests/_helpers.py", line 101, in http_request
    func, RequestsMixin._get_status_code, retry_strategy)
  File "/usr/local/lib/python2.7/dist-packages/google/resumable_media/_helpers.py", line 146, in wait_and_retry
    response = func()
  File "/usr/local/lib/python2.7/dist-packages/google/auth/transport/requests.py", line 176, in request
    self._auth_request, method, url, request_headers)
  File "/usr/local/lib/python2.7/dist-packages/google/auth/credentials.py", line 121, in before_request
    self.refresh(request)
  File "/usr/local/lib/python2.7/dist-packages/google/auth/compute_engine/credentials.py", line 93, in refresh
    raise exceptions.RefreshError(exc)
RuntimeError: RefreshError: HTTPConnectionPool(host='metadata.google.internal', port=80): Max retries exceeded with url: /computeMetadata/v1/instance/service-accounts/default/?recursive=true

Where should this bug go then?

[dhermes]

Moved to googleapis/google-auth-library-python#211

[dhermes]

@dmho418 That error certainly looks more actionable! For now, let's put it in googleapis/google-auth-library-python#211