Skip to content

Commit 7ad11b2

Browse files
author
Ryan Kohler
authored
test: Create BYOID Integration tests (#719)
1 parent 1dcc804 commit 7ad11b2

4 files changed

Lines changed: 280 additions & 11 deletions

File tree

packages/google-auth/CONTRIBUTING.rst

Lines changed: 23 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -42,9 +42,9 @@ You can run the system tests with ``nox``::
4242
To run a single session, specify it with ``nox -s``::
4343

4444
$ nox -f system_tests/noxfile.py -s service_account
45-
46-
First, set the environemnt variable ``GOOGLE_APPLICATION_CREDENTIALS`` to a valid service account.
47-
See `Creating and Managing Service Account Keys`_ for how to obtain a service account.
45+
46+
First, set the environment variable ``GOOGLE_APPLICATION_CREDENTIALS`` to a valid service account.
47+
See `Creating and Managing Service Account Keys`_ for how to obtain a service account.
4848

4949
Project and Credentials Setup
5050
-------------------------------
@@ -86,26 +86,40 @@ This will allow the user to impersonate service accounts on the project.
8686
``service_account.json``
8787
~~~~~~~~~~~~~~~~~~~~~~~~
8888

89-
Follow `Creating and Managing Service Account Keys`_ to create a service account.
89+
Follow `Creating and Managing Service Account Keys`_ to create a service account.
9090

9191
Copy the credentials file to ``service_account.json``.
9292

9393
Grant the account associated with ``service_account.json`` the following roles.
9494

9595
- App Engine Admin (for App Engine tests)
96-
- Service Account Token Creator (for impersonated credentials tests)
96+
- Service Account Token Creator (for impersonated credentials and workload identity federation tests)
9797
- Pub/Sub Viewer (for gRPC tests)
9898
- Storage Object Viewer (for impersonated credentials tests)
99+
- DNS Viewer (for workload identity federation tests)
99100

100101
``impersonated_service_account.json``
101102
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
102103

103-
Follow `Creating and Managing Service Account Keys`_ to create a service account.
104+
Follow `Creating and Managing Service Account Keys`_ to create a service account.
104105

105106
Copy the credentials file to ``impersonated_service_account.json``.
106107

107108
.. _Creating and Managing Service Account Keys: https://cloud.google.com/iam/docs/creating-managing-service-account-keys
108109

110+
``setup_external_accounts``
111+
~~~~~~~~~~~~~~~~
112+
113+
In order to run the workload identity federation tests, you will need to set up
114+
a Workload Identity Pool, as well as attach relevant policy bindings for this
115+
new resource to our service account. To do this, make sure you have IAM Workload
116+
Identity Pool Admin and Security Admin permissions, and then run:
117+
118+
$ ./scripts/setup_external_accounts.sh
119+
120+
and then use the output to replace the variables near
121+
the top of system_tests/system_tests_sync/test_external_accounts.py
122+
109123
App Engine System Tests
110124
~~~~~~~~~~~~~~~~~~~~~~~~
111125

@@ -118,16 +132,16 @@ From ``system_tests/app_engine_test_app`` run the following commands ::
118132
$ pip install --target lib -r requirements.txt
119133
$ gcloud app deploy -q app.yaml
120134

121-
After the app is deployed, change ``service`` in ``app.yaml`` back to ``google-auth-system-tests``.
135+
After the app is deployed, change ``service`` in ``app.yaml`` back to ``google-auth-system-tests``.
122136
You can now run the App Engine tests: ::
123137

124138
$ nox -f system_tests/noxfile.py -s app_engine
125-
139+
126140
Compute Engine Tests
127141
^^^^^^^^^^^^^^^^^^^^
128142

129143
These tests cannot be run locally and will be skipped if they are run outside of Google Compute Engine.
130-
144+
131145
grpc Tests
132146
^^^^^^^^^^^^
133147

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
#!/bin/bash
2+
# Copyright 2021 Google LLC
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
16+
# This file is a mostly common setup file to ensure all workload identity
17+
# federation integration tests are set up in a consistent fashion across the
18+
# languages in our various client libraries. It assumes that the current user
19+
# has the relevant permissions to run each of the commands listed.
20+
21+
# This script needs to be run once. It will do the following:
22+
# 1. Create a random workload identity pool.
23+
# 2. Create a random OIDC provider in that pool which uses the
24+
# accounts.google.com as the issuer and the default STS audience as the
25+
# allowed audience. This audience will be validated on STS token exchange.
26+
# 3. Enable OIDC tokens generated by the current service account to impersonate
27+
# the service account. (Identified by the OIDC token sub field which is the
28+
# service account client ID).
29+
# 4. Create a random AWS provider in that pool which uses the provided AWS
30+
# account ID.
31+
# 5. Enable AWS provider to impersonate the service account. (Principal is
32+
# identified by the AWS role name).
33+
# 6. Print out the STS audience fields associated with the created providers
34+
# after the setup completes successfully so that they can be used in the
35+
# tests. These will be copied and used as the global _AUDIENCE_OIDC and
36+
# _AUDIENCE_AWS constants in system_tests/system_tests_sync/test_external_accounts.py.
37+
#
38+
# It is safe to run the setup script again. A new pool is created and new
39+
# audiences are printed. If run multiple times, it is advisable to delete
40+
# unused pools. Note that deleted pools are soft deleted and may remain for
41+
# a while before they are completely deleted. The old pool ID cannot be used
42+
# in the meantime.
43+
#
44+
# For AWS tests, an AWS developer account is needed.
45+
# The following AWS prerequisite setup is needed.
46+
# 1. An OIDC Google identity provider needs to be created with the following:
47+
# issuer: accounts.google.com
48+
# audience: Use the client_id of the service account.
49+
# 2. A role for OIDC web identity federation is needed with the created Google
50+
# provider as a trusted entity:
51+
# "accounts.google.com:aud": "$CLIENT_ID"
52+
# The steps are documented at:
53+
# https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-idp_oidc.html
54+
55+
suffix=""
56+
57+
function generate_random_string () {
58+
local valid_chars=abcdefghijklmnopqrstuvwxyz0123456789
59+
for i in {1..8} ; do
60+
suffix+="${valid_chars:RANDOM%${#valid_chars}:1}"
61+
done
62+
}
63+
64+
generate_random_string
65+
66+
pool_id="pool-"$suffix
67+
oidc_provider_id="oidc-"$suffix
68+
aws_provider_id="aws-"$suffix
69+
70+
# TODO: Fill in.
71+
project_id="stellar-day-254222"
72+
project_number="79992041559"
73+
aws_account_id="077071391996"
74+
aws_role_name="ci-python-test"
75+
service_account_email="kokoro@stellar-day-254222.iam.gserviceaccount.com"
76+
sub="104692443208068386138"
77+
78+
oidc_aud="//iam.googleapis.com/projects/$project_number/locations/global/workloadIdentityPools/$pool_id/providers/$oidc_provider_id"
79+
aws_aud="//iam.googleapis.com/projects/$project_number/locations/global/workloadIdentityPools/$pool_id/providers/$aws_provider_id"
80+
81+
gcloud config set project $project_id
82+
83+
# Create the Workload Identity Pool.
84+
gcloud beta iam workload-identity-pools create $pool_id \
85+
--location="global" \
86+
--description="Test pool" \
87+
--display-name="Test pool for Python"
88+
89+
# Create the OIDC Provider.
90+
gcloud beta iam workload-identity-pools providers create-oidc $oidc_provider_id \
91+
--workload-identity-pool=$pool_id \
92+
--issuer-uri="https://accounts.google.com" \
93+
--location="global" \
94+
--attribute-mapping="google.subject=assertion.sub"
95+
96+
# Create the AWS Provider.
97+
gcloud beta iam workload-identity-pools providers create-aws $aws_provider_id \
98+
--workload-identity-pool=$pool_id \
99+
--account-id=$aws_account_id \
100+
--location="global"
101+
102+
# Give permission to impersonate the service account.
103+
gcloud iam service-accounts add-iam-policy-binding $service_account_email \
104+
--role roles/iam.workloadIdentityUser \
105+
--member "principal://iam.googleapis.com/projects/$project_number/locations/global/workloadIdentityPools/$pool_id/subject/$sub"
106+
107+
gcloud iam service-accounts add-iam-policy-binding $service_account_email \
108+
--role roles/iam.workloadIdentityUser \
109+
--member "principalSet://iam.googleapis.com/projects/$project_number/locations/global/workloadIdentityPools/$pool_id/attribute.aws_role/arn:aws:sts::$aws_account_id:assumed-role/$aws_role_name"
110+
111+
echo "OIDC audience: "$oidc_aud
112+
echo "AWS audience: "$aws_aud

packages/google-auth/system_tests/noxfile.py

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -354,9 +354,14 @@ def mtls_http(session):
354354
)
355355

356356

357-
# ASYNC SYSTEM TESTS
357+
@nox.session(python=PYTHON_VERSIONS_SYNC)
358+
def external_accounts(session):
359+
session.install(*TEST_DEPENDENCIES_SYNC, "google-auth", "google-api-python-client", "enum34")
360+
default(session, "system_tests_sync/test_external_accounts.py")
358361

359362

363+
# ASYNC SYSTEM TESTS
364+
360365
@nox.session(python=PYTHON_VERSIONS_ASYNC)
361366
def service_account_async(session):
362367
session.install(*(TEST_DEPENDENCIES_SYNC + TEST_DEPENDENCIES_ASYNC))
@@ -374,7 +379,7 @@ def default_explicit_service_account_async(session):
374379
session.install(LIBRARY_DIR)
375380
default(
376381
session,
377-
"system_tests_async/test_default.py",
382+
"system_tests_async/test_default.py",
378383
"system_tests_async/test_id_token.py",
379384
)
380385

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
# Copyright 2021 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
# Prerequisites:
16+
# Make sure to run the setup in scripts/setup_external_accounts.sh
17+
# and copy the logged constant strings (_AUDIENCE_OIDC, _AUDIENCE_AWS)
18+
# into this file before running this test suite.
19+
# Once that is done, this test can be run indefinitely.
20+
#
21+
# The only requirement for this test suite to run is to set the environment
22+
# variable GOOGLE_APPLICATION_CREDENTIALS to point to the expected service
23+
# account keys whose email is referred to in the setup script.
24+
#
25+
# This script follows the following logic.
26+
# OIDC provider (file-sourced and url-sourced credentials):
27+
# Use the service account keys to generate a Google ID token using the
28+
# iamcredentials generateIdToken API, using the default STS audience.
29+
# This will use the service account client ID as the sub field of the token.
30+
# This OIDC token will be used as the external subject token to be exchanged
31+
# for a Google access token via GCP STS endpoint and then to impersonate the
32+
# original service account key.
33+
34+
35+
import json
36+
import os
37+
from tempfile import NamedTemporaryFile
38+
39+
import sys
40+
import google.auth
41+
from googleapiclient import discovery
42+
from google.oauth2 import service_account
43+
import pytest
44+
from mock import patch
45+
46+
# Populate values from the output of scripts/setup_external_accounts.sh.
47+
_AUDIENCE_OIDC = "//iam.googleapis.com/projects/79992041559/locations/global/workloadIdentityPools/pool-73wslmxn/providers/oidc-73wslmxn"
48+
49+
50+
def dns_access_direct(request, project_id):
51+
# First, get the default credentials.
52+
credentials, _ = google.auth.default(
53+
scopes=["https://www.googleapis.com/auth/cloud-platform.read-only"],
54+
request=request,
55+
)
56+
57+
# Apply the default credentials to the headers to make the request.
58+
headers = {}
59+
credentials.apply(headers)
60+
response = request(
61+
url="https://dns.googleapis.com/dns/v1/projects/{}".format(project_id),
62+
headers=headers,
63+
)
64+
65+
if response.status == 200:
66+
return response.data
67+
68+
69+
def dns_access_client_library(_, project_id):
70+
service = discovery.build("dns", "v1")
71+
request = service.projects().get(project=project_id)
72+
return request.execute()
73+
74+
75+
@pytest.fixture(params=[dns_access_direct, dns_access_client_library])
76+
def dns_access(request, http_request, service_account_info):
77+
# Fill in the fixtures on the functions,
78+
# so that we don't have to fill in the parameters manually.
79+
def wrapper():
80+
return request.param(http_request, service_account_info["project_id"])
81+
82+
yield wrapper
83+
84+
85+
@pytest.fixture
86+
def oidc_credentials(service_account_file, http_request):
87+
result = service_account.IDTokenCredentials.from_service_account_file(
88+
service_account_file, target_audience=_AUDIENCE_OIDC
89+
)
90+
result.refresh(http_request)
91+
yield result
92+
93+
94+
@pytest.fixture
95+
def service_account_info(service_account_file):
96+
with open(service_account_file) as f:
97+
yield json.load(f)
98+
99+
100+
# Our external accounts tests involve setting up some preconditions, setting a
101+
# credential file, and then making sure that our client libraries can work with
102+
# the set credentials.
103+
def get_project_dns(dns_access, credential_data):
104+
with NamedTemporaryFile() as credfile:
105+
credfile.write(json.dumps(credential_data).encode("utf-8"))
106+
credfile.flush()
107+
old_credentials = os.environ.get("GOOGLE_APPLICATION_CREDENTIALS")
108+
109+
with patch.dict(os.environ, {"GOOGLE_APPLICATION_CREDENTIALS": credfile.name}):
110+
# If our setup and credential file are correct,
111+
# discovery.build should be able to establish these as the default credentials.
112+
return dns_access()
113+
114+
115+
# This test makes sure that setting an accesible credential file
116+
# works to allow access to Google resources.
117+
def test_file_based_external_account(
118+
oidc_credentials, service_account_info, dns_access
119+
):
120+
with NamedTemporaryFile() as tmpfile:
121+
tmpfile.write(oidc_credentials.token.encode("utf-8"))
122+
tmpfile.flush()
123+
124+
assert get_project_dns(
125+
dns_access,
126+
{
127+
"type": "external_account",
128+
"audience": _AUDIENCE_OIDC,
129+
"subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
130+
"token_url": "https://sts.googleapis.com/v1/token",
131+
"service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/{}:generateAccessToken".format(
132+
oidc_credentials.service_account_email
133+
),
134+
"credential_source": {
135+
"file": tmpfile.name,
136+
},
137+
},
138+
)

0 commit comments

Comments
 (0)