Merge run provider tests and use built provider by trodge · Pull Request #8229 · GoogleCloudPlatform/magic-modules

trodge · 2023-06-28T20:38:22Z

Merge the "Run provider tests" and "Use built provider" pages into "Run tests" page.

If this PR is for Terraform, I acknowledge that I have:

Searched through the issue tracker for an open issue that this either resolves or contributes to, commented on it to claim it, and written "fixes {url}" or "part of {url}" in this PR description. If there were no relevant open issues, I opened one and commented that I would like to work on it (not necessary for very small changes).
Ensured that all new fields I added that can be set by a user appear in at least one example (for generated resources) or third_party test (for handwritten resources or update tests).
Generated Terraform providers, and ran make test and make lint in the generated providers to ensure it passes unit and linter tests.
Ran relevant acceptance tests using my own Google Cloud project and credentials (If the acceptance tests do not yet pass or you are unable to run them, please let your reviewer know).
Read Write release notes before writing my release note below.

Release Note Template for Downstream PRs (will be copied)

modular-magician · 2023-06-28T20:53:42Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR hasn't generated any diffs, but I'll let you know if a future commit does.

modular-magician · 2023-06-28T23:30:54Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR hasn't generated any diffs, but I'll let you know if a future commit does.

modular-magician · 2023-06-29T00:55:53Z

Tests analytics

Total tests: 2832
Passed tests 2536
Skipped tests: 295
Affected tests: 1

Action taken

Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccComputeFirewallPolicyRule_multipleRules

Get to know how VCR tests work

modular-magician · 2023-06-29T00:58:38Z

Tests failed during RECORDING mode:
TestAccComputeFirewallPolicyRule_multipleRules[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

melinath

Thanks for putting this together - I left some change requests, mostly style cleanup.

modular-magician · 2023-06-29T18:24:48Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR hasn't generated any diffs, but I'll let you know if a future commit does.

modular-magician · 2023-06-29T19:49:21Z

Tests analytics

Total tests: 2834
Passed tests 2531
Skipped tests: 295
Affected tests: 8

Action taken

Found 8 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccFilestoreInstance_update|TestAccFilestoreInstance_filestoreInstanceFullExample|TestAccFilestoreInstance_filestoreInstanceBasicExample|TestAccFilestoreBackup_update|TestAccFilestoreBackup_filestoreBackupBasicExample|TestAccFilestoreSnapshot_filestoreSnapshotFullExample|TestAccFilestoreSnapshot_filestoreSnapshotBasicExample|TestAccComputeFirewallPolicyRule_multipleRules

Get to know how VCR tests work

modular-magician · 2023-06-29T20:10:26Z

Tests passed during RECORDING mode:
TestAccFilestoreBackup_update[Debug log]
TestAccFilestoreSnapshot_filestoreSnapshotBasicExample[Debug log]

Tests failed during RECORDING mode:
TestAccComputeFirewallPolicyRule_multipleRules[Error message] [Debug log]

Several tests got terminated during RECORDING mode.
Please fix these to complete your PR
View the build log or the debug log for each test

modular-magician · 2023-06-29T22:14:06Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR hasn't generated any diffs, but I'll let you know if a future commit does.

modular-magician · 2023-06-29T23:35:59Z

Tests analytics

Total tests: 2834
Passed tests 2536
Skipped tests: 295
Affected tests: 3

Action taken

Found 3 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccFilestoreSnapshot_filestoreSnapshotFullExample|TestAccFilestoreInstance_filestoreInstanceBasicExample|TestAccComputeFirewallPolicyRule_multipleRules

Get to know how VCR tests work

modular-magician · 2023-06-29T23:52:42Z

Tests passed during RECORDING mode:
TestAccFilestoreSnapshot_filestoreSnapshotFullExample[Debug log]
TestAccFilestoreInstance_filestoreInstanceBasicExample[Debug log]

Rerun these tests in REPLAYING mode to catch issues

All tests passed after rerunning REPLAYING mode.

Tests failed during RECORDING mode:
TestAccComputeFirewallPolicyRule_multipleRules[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

melinath

could you add some guidance on how folks can create a test configuration to do manual tests with?

melinath · 2023-06-30T18:04:34Z

 Sometimes, for example for manual testing, you may want to build the provider from source and use it with `terraform`.

-## Developer Overrides
+### Build provider


Possibly put this after the developer overrides section.

melinath · 2023-06-30T18:07:08Z

-## Alternative: using a global CLI configuration file
+### Alternative: using a global CLI configuration file

 If you do not want to use the `TF_CLI_CONFIG_FILE` environment variable, as described above, you can instead create a global version of the CLI configuration file. This configuration will be used automatically by Terraform. To do this, follow [the guidance in the official documentation](https://developer.hashicorp.com/terraform/cli/config/config-file#locations).


We should recommend a specific path here and link to the Hashicorp docs for more details to reduce the amount of inline context

melinath · 2023-06-30T18:10:26Z

btw this is a huge improvement already, thank you!

modular-magician · 2023-07-05T17:44:21Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR hasn't generated any diffs, but I'll let you know if a future commit does.

modular-magician · 2023-07-05T19:02:36Z

Tests analytics

Total tests: 2836
Passed tests 2540
Skipped tests: 295
Affected tests: 1

Action taken

Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccComputeFirewallPolicyRule_multipleRules

Get to know how VCR tests work

modular-magician · 2023-07-05T19:05:08Z

Tests failed during RECORDING mode:
TestAccComputeFirewallPolicyRule_multipleRules[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

modular-magician · 2023-07-05T20:44:58Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR hasn't generated any diffs, but I'll let you know if a future commit does.

modular-magician · 2023-07-05T22:04:57Z

Tests analytics

Total tests: 2836
Passed tests 2540
Skipped tests: 295
Affected tests: 1

Action taken

Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccComputeFirewallPolicyRule_multipleRules

Get to know how VCR tests work

modular-magician · 2023-07-05T22:07:41Z

Tests failed during RECORDING mode:
TestAccComputeFirewallPolicyRule_multipleRules[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

modular-magician · 2023-07-05T22:24:57Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR hasn't generated any diffs, but I'll let you know if a future commit does.

modular-magician · 2023-07-05T23:42:45Z

Tests analytics

Total tests: 2836
Passed tests 2540
Skipped tests: 295
Affected tests: 1

Action taken

Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccComputeFirewallPolicyRule_multipleRules

Get to know how VCR tests work

modular-magician · 2023-07-05T23:45:19Z

Tests failed during RECORDING mode:
TestAccComputeFirewallPolicyRule_multipleRules[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

melinath

This is looking really good! I left a few comments but then realized there are still a few things pending from the last review - apologies for the double-review!

Co-authored-by: Stephen Lewis (Burrows) <stephen.r.burrows@gmail.com>

modular-magician · 2023-07-10T22:16:26Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR hasn't generated any diffs, but I'll let you know if a future commit does.

modular-magician · 2023-07-10T23:59:37Z

Tests analytics

Total tests: 2848
Passed tests 2550
Skipped tests: 295
Affected tests: 3

Action taken

Found 3 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccFolderIamPolicy_basic|TestAccContainerAwsNodePool_BetaBasicHandWritten|TestAccComputeFirewallPolicyRule_multipleRules

Get to know how VCR tests work

modular-magician · 2023-07-11T00:03:08Z

$\textcolor{green}{\textsf{Tests passed during RECORDING mode:}}$
TestAccFolderIamPolicy_basic[Debug log]

Rerun these tests in REPLAYING mode to catch issues

$\textcolor{green}{\textsf{No issues found for passed tests after REPLAYING rerun.}}$

$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccContainerAwsNodePool_BetaBasicHandWritten[Error message] [Debug log]
TestAccComputeFirewallPolicyRule_multipleRules[Error message] [Debug log]

$\textcolor{red}{\textsf{Please fix these to complete your PR.}}$
View the build log or the debug log for each test

melinath

Thanks, looking better! Some additional change requests below.

Co-authored-by: Stephen Lewis (Burrows) <stephen.r.burrows@gmail.com>

modular-magician · 2023-07-11T17:42:00Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 1 file changed, 1 insertion(+))
TF Conversion: Diff ( 2 files changed, 3 insertions(+), 3 deletions(-))

modular-magician · 2023-07-11T18:58:46Z

Tests analytics

Total tests: 2851
Passed tests 2552
Skipped tests: 296
Affected tests: 3

Action taken

Found 3 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccContainerCluster_withMonitoringConfig|TestAccContainerAwsNodePool_BetaBasicHandWritten|TestAccComputeFirewallPolicyRule_multipleRules

Get to know how VCR tests work

modular-magician · 2023-07-11T19:37:34Z

$\textcolor{green}{\textsf{Tests passed during RECORDING mode:}}$
TestAccContainerCluster_withMonitoringConfig[Debug log]

Rerun these tests in REPLAYING mode to catch issues

$\textcolor{green}{\textsf{No issues found for passed tests after REPLAYING rerun.}}$

$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccContainerAwsNodePool_BetaBasicHandWritten[Error message] [Debug log]
TestAccComputeFirewallPolicyRule_multipleRules[Error message] [Debug log]

$\textcolor{red}{\textsf{Please fix these to complete your PR.}}$
View the build log or the debug log for each test

modular-magician · 2023-07-14T20:44:27Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR hasn't generated any diffs, but I'll let you know if a future commit does.

modular-magician · 2023-07-14T22:28:57Z

Tests analytics

Total tests: 2856
Passed tests 2554
Skipped tests: 299
Affected tests: 3

Action taken

Found 3 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccContainerAwsNodePool_BetaBasicHandWritten|TestAccContainerCluster_withMonitoringConfig|TestAccComputeFirewallPolicyRule_multipleRules

Get to know how VCR tests work

modular-magician · 2023-07-14T23:13:09Z

$\textcolor{green}{\textsf{Tests passed during RECORDING mode:}}$
TestAccContainerCluster_withMonitoringConfig[Debug log]

Rerun these tests in REPLAYING mode to catch issues

$\textcolor{green}{\textsf{No issues found for passed tests after REPLAYING rerun.}}$

$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccContainerAwsNodePool_BetaBasicHandWritten[Error message] [Debug log]
TestAccComputeFirewallPolicyRule_multipleRules[Error message] [Debug log]

$\textcolor{red}{\textsf{Please fix these to complete your PR.}}$
View the build log or the debug log for each test

melinath

A few more change requests and then I think this will be good to go!

Co-authored-by: Stephen Lewis (Burrows) <stephen.r.burrows@gmail.com>

modular-magician · 2023-07-19T21:57:07Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

TF Conversion: Diff ( 2 files changed, 3 insertions(+), 3 deletions(-))

melinath

LGTM - this is a significant improvement over the previous pages. Thanks for your work on this!

…8229) * Merge run provider tests and use built provider * Add aliases, fix link * Apply suggestions from code review Co-authored-by: Stephen Lewis (Burrows) <stephen.r.burrows@gmail.com> * Move hint and link to wiki * Reorganize into tabs * Apply suggestions from code review Co-authored-by: Stephen Lewis (Burrows) <stephen.r.burrows@gmail.com> * Apply suggestions from code review Co-authored-by: Stephen Lewis (Burrows) <stephen.r.burrows@gmail.com> * Apply suggestions from code review Co-authored-by: Stephen Lewis (Burrows) <stephen.r.burrows@gmail.com> * Add tabs for provider version * Fix link * Apply suggestions from code review Co-authored-by: Stephen Lewis (Burrows) <stephen.r.burrows@gmail.com> * Move delve into tabs * Fix ref links * Apply suggestions from code review Co-authored-by: Stephen Lewis (Burrows) <stephen.r.burrows@gmail.com> * Apply suggestions from code review * Apply suggestions from code review Co-authored-by: Stephen Lewis (Burrows) <stephen.r.burrows@gmail.com> --------- Co-authored-by: Stephen Lewis (Burrows) <stephen.r.burrows@gmail.com>

trodge requested a review from melinath June 28, 2023 22:31

trodge force-pushed the run-tests-guide branch from 03d656d to 82356fa Compare June 28, 2023 23:16

melinath requested changes Jun 29, 2023

View reviewed changes

trodge force-pushed the run-tests-guide branch from 95b35be to dbc931d Compare June 29, 2023 21:58

trodge requested a review from melinath June 29, 2023 22:00

melinath requested changes Jun 30, 2023

View reviewed changes

melinath self-requested a review July 10, 2023 16:58

melinath requested changes Jul 10, 2023

View reviewed changes

trodge added 2 commits July 10, 2023 21:54

Merge run provider tests and use built provider

194c2bd

Add aliases, fix link

00d3cf1

trodge and others added 2 commits July 10, 2023 21:55

Apply suggestions from code review

f2622fd

Co-authored-by: Stephen Lewis (Burrows) <stephen.r.burrows@gmail.com>

Move delve into tabs

1326a94

trodge force-pushed the run-tests-guide branch from 670d93e to 1326a94 Compare July 10, 2023 21:55

trodge requested a review from melinath July 10, 2023 21:55

Fix ref links

3d75da4

melinath requested changes Jul 11, 2023

View reviewed changes

Apply suggestions from code review

d1ed715

Co-authored-by: Stephen Lewis (Burrows) <stephen.r.burrows@gmail.com>

Apply suggestions from code review

17ce48f

trodge requested a review from melinath July 14, 2023 20:30

melinath requested changes Jul 17, 2023

View reviewed changes

Comment thread docs/content/develop/run-tests.md Outdated

Comment thread docs/content/develop/run-tests.md Outdated

Comment thread docs/content/develop/run-tests.md Outdated

Comment thread docs/content/develop/run-tests.md Outdated

Comment thread docs/content/develop/run-tests.md Outdated

Apply suggestions from code review

57b1dfc

Co-authored-by: Stephen Lewis (Burrows) <stephen.r.burrows@gmail.com>

trodge force-pushed the run-tests-guide branch from fb9b6cd to 57b1dfc Compare July 19, 2023 21:44

trodge requested a review from melinath July 19, 2023 21:45

melinath approved these changes Jul 25, 2023

View reviewed changes

melinath merged commit 8863502 into GoogleCloudPlatform:main Jul 25, 2023

trodge deleted the run-tests-guide branch July 25, 2023 20:28

Uh oh!

Conversation

trodge commented Jun 28, 2023

Uh oh!

modular-magician commented Jun 28, 2023

Diff report

Uh oh!

modular-magician commented Jun 28, 2023

Diff report

Uh oh!

modular-magician commented Jun 29, 2023

Tests analytics

Action taken

Uh oh!

modular-magician commented Jun 29, 2023

Uh oh!

melinath left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

modular-magician commented Jun 29, 2023

Diff report

Uh oh!

modular-magician commented Jun 29, 2023

Tests analytics

Action taken

Uh oh!

modular-magician commented Jun 29, 2023

Uh oh!

modular-magician commented Jun 29, 2023

Diff report

Uh oh!

modular-magician commented Jun 29, 2023

Tests analytics

Action taken

Uh oh!

modular-magician commented Jun 29, 2023

Rerun these tests in REPLAYING mode to catch issues

Uh oh!

melinath left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

melinath Jun 30, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

melinath Jun 30, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

melinath commented Jun 30, 2023

Uh oh!

modular-magician commented Jul 5, 2023

Diff report

Uh oh!

modular-magician commented Jul 5, 2023

Tests analytics

Action taken

Uh oh!

modular-magician commented Jul 5, 2023

Uh oh!

modular-magician commented Jul 5, 2023

Diff report

Uh oh!

modular-magician commented Jul 5, 2023