Skip to content

Add support for ResourceRefs attaching to arbitrary types and referencing multiple targets#8127

Merged
NickElliot merged 16 commits into
GoogleCloudPlatform:mainfrom
NickElliot:resourcerefs
Jul 11, 2023
Merged

Add support for ResourceRefs attaching to arbitrary types and referencing multiple targets#8127
NickElliot merged 16 commits into
GoogleCloudPlatform:mainfrom
NickElliot:resourcerefs

Conversation

@NickElliot

@NickElliot NickElliot commented Jun 9, 2023

Copy link
Copy Markdown
Contributor

Fundamentally this PR is removing validation -- the Terraform schema is unable to access the data of resources other than itself for runtime validation, and generator does not access files outside of the folder from which it is currently generating. In current Terraform-only support for Magic Modules, the resource ref ultimately does nothing other than apply some logic on self-links to create a bit of validation for compute resources. Every resource-ref outside of compute engine is a url_parameter, or uses separate custom expands to process it as a self link generated off of a name/process the input value as a string directly.

This update moves the current compute-first implementation to a separate custom_expand that needs to be specifically declared, and updates resource_ref to use the "process as a string" logic. If the resource_ref is to a resource within the same product (and no resource_product is declared in case of a shared resource name), building MMV1 will still attempt to validate the import field exists in the target resource.

Additionally, the type.rb file has been updated to have an output property that directly contains the suppled "resource", "resourceproduct" and "imports" strings from the yaml attributes. Nothing uses them at this time, but it makes those pieces more available to potential custom_* functions in the future.

If all goes well this PR should generate almost no diffs.

If this PR is for Terraform, I acknowledge that I have:

  • Searched through the issue tracker for an open issue that this either resolves or contributes to, commented on it to claim it, and written "fixes {url}" or "part of {url}" in this PR description. If there were no relevant open issues, I opened one and commented that I would like to work on it (not necessary for very small changes).
  • Ensured that all new fields I added that can be set by a user appear in at least one example (for generated resources) or third_party test (for handwritten resources or update tests).
  • Generated Terraform providers, and ran make test and make lint in the generated providers to ensure it passes unit and linter tests.
  • Ran relevant acceptance tests using my own Google Cloud project and credentials (If the acceptance tests do not yet pass or you are unable to run them, please let your reviewer know).
  • Read the Release Notes Guide before writing my release note below.

Release Note Template for Downstream PRs (will be copied)


@NickElliot

NickElliot commented Jun 9, 2023

Copy link
Copy Markdown
Contributor Author

The generate diffs appears to be failing because there is no diffs.

@NickElliot
NickElliot marked this pull request as ready for review June 9, 2023 22:07
@NickElliot
NickElliot requested a review from rileykarson June 9, 2023 22:08
@rileykarson

Copy link
Copy Markdown
Member

/gcbrun

@modular-magician

Copy link
Copy Markdown
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR hasn't generated any diffs, but I'll let you know if a future commit does.

@modular-magician

Copy link
Copy Markdown
Collaborator

Tests analytics

Total tests: 2784
Passed tests 2486
Skipped tests: 295
Affected tests: 3

Action taken

Found 3 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
TestAccComputeNetworkEndpoints_networkEndpointsBasic|TestAccComputeFirewallPolicyRule_multipleRules|TestAccApigeeKeystoresAliasesKeyCertFile_apigeeKeystoresAliasesKeyCertFileTestExample

Get to know how VCR tests work

@modular-magician

Copy link
Copy Markdown
Collaborator

Tests passed during RECORDING mode:
TestAccComputeNetworkEndpoints_networkEndpointsBasic[Debug log]
TestAccApigeeKeystoresAliasesKeyCertFile_apigeeKeystoresAliasesKeyCertFileTestExample[Debug log]

Tests failed during RECORDING mode:
TestAccComputeFirewallPolicyRule_multipleRules[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

Comment thread mmv1/api/type.rb Outdated
Comment thread mmv1/api/type.rb Outdated
Comment thread mmv1/api/type.rb Outdated
@modular-magician

Copy link
Copy Markdown
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR hasn't generated any diffs, but I'll let you know if a future commit does.

@modular-magician

Copy link
Copy Markdown
Collaborator

Tests analytics

Total tests: 2787
Passed tests 2489
Skipped tests: 296
Affected tests: 2

Action taken

Found 2 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
TestAccComputeNetworkEndpoints_networkEndpointsBasic|TestAccComputeFirewallPolicyRule_multipleRules

Get to know how VCR tests work

@modular-magician

Copy link
Copy Markdown
Collaborator

Tests passed during RECORDING mode:
TestAccComputeNetworkEndpoints_networkEndpointsBasic[Debug log]

Tests failed during RECORDING mode:
TestAccComputeFirewallPolicyRule_multipleRules[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

@NickElliot
NickElliot requested a review from rileykarson June 20, 2023 16:44
Comment thread mmv1/api/type.rb Outdated
@modular-magician

Copy link
Copy Markdown
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform Beta: Diff ( 1 file changed, 1 insertion(+), 5 deletions(-))

@modular-magician

Copy link
Copy Markdown
Collaborator

Tests analytics

Total tests: 2843
Passed tests 2501
Skipped tests: 295
Affected tests: 47

Action taken

Found 47 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
TestAccContainerAwsCluster_BasicHandWritten|TestAccComputeRegionInstanceTemplate_with18TbScratchDisk|TestAccComputeRegionInstanceTemplate_withScratchDisk|TestAccComputeNetworkEdgeSecurityService_update|TestAccComputeRegionInstanceTemplate_guestAccelerator|TestAccComputeRegionInstanceTemplate_primaryAliasIpRange|TestAccComputeInstanceTemplate_guestAccelerator|TestAccComputeInstanceTemplate_secondaryAliasIpRange|TestAccComputeRegionAutoscaler_regionAutoscalerBasicExample|TestAccComputeInstanceTemplate_primaryAliasIpRange|TestAccComputeInstanceFromTemplate_self_link_unique|TestAccComputeInstanceFromTemplate_basic|TestAccComputeInstanceTemplate_metadata_startup_script|TestAccComputeInstanceTemplate_subnet_auto|TestAccComputeInstanceTemplate_withScratchDisk|TestAccComputeInstanceTemplate_regionDisks|TestAccComputeRegionInstanceTemplate_metadata_startup_script|TestAccComputeInstanceTemplate_disks|TestAccComputeInstanceTemplate_NetworkAttachment|TestAccComputeRegionInstanceTemplate_subnet_custom|TestAccComputeRegionInstanceTemplate_subnet_auto|TestAccComputeInstanceTemplate_guestAcceleratorSkip|TestAccComputeInstanceFromTemplate_012_removableFields|TestAccComputeRegionInstanceTemplate_minCpuPlatform|TestAccComputeInstanceFromRegionTemplate_basic|TestAccComputeInstanceTemplate_minCpuPlatform|TestAccComputeInstanceFromTemplate_overrideScratchDisk|TestAccClouddeployDeliveryPipeline_DeliveryPipeline|TestAccClouddeployDeliveryPipeline_CanaryrunDeliveryPipeline|TestAccClouddeployDeliveryPipeline_CanaryServiceNetworkingDeliveryPipeline|TestAccComputeInstanceTemplate_subnet_custom|TestAccComputeRegionInstanceTemplate_guestAcceleratorSkip|TestAccCloudbuildv2Connection_GlePrivUpdateConnection|TestAccComputeInstanceFromTemplate_overrideAttachedDisk|TestAccCloudbuildv2Connection_GleOldConnection|TestAccCloudbuildv2Connection_GleConnection|TestAccComputeInstanceFromTemplate_overrideBootDisk|TestAccContainerAwsNodePool_BetaBasicHandWritten|TestAccContainerAwsCluster_BetaBasicHandWritten|TestAccComputeFirewallPolicyRule_multipleRules|TestAccClouddeployDeliveryPipeline_CanaryDeliveryPipeline|TestAccCloudbuildv2Connection_GlePrivConnection|TestAccCloudbuildv2Connection_GitlabConnection|TestAccComputeRegionInstanceTemplate_secondaryAliasIpRange|TestAccContainerAwsNodePool_BasicHandWritten|TestAccComputeInstanceTemplate_with18TbScratchDisk|TestAccComputeRegionInstanceTemplate_regionDisks

Get to know how VCR tests work

@rileykarson rileykarson left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rake and yaml lint look unhappy!

@modular-magician

Copy link
Copy Markdown
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform Beta: Diff ( 1 file changed, 1 insertion(+), 5 deletions(-))

@modular-magician

Copy link
Copy Markdown
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR hasn't generated any diffs, but I'll let you know if a future commit does.

@modular-magician

Copy link
Copy Markdown
Collaborator

Tests analytics

Total tests: 2843
Passed tests 2501
Skipped tests: 295
Affected tests: 47

Action taken

Found 47 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
TestAccContainerAwsNodePool_BetaBasicHandWritten|TestAccContainerAwsNodePool_BasicHandWritten|TestAccContainerAwsCluster_BetaBasicHandWritten|TestAccContainerAwsCluster_BasicHandWritten|TestAccComputeRegionInstanceTemplate_subnet_auto|TestAccComputeRegionInstanceTemplate_regionDisks|TestAccComputeRegionAutoscaler_regionAutoscalerBasicExample|TestAccComputeRegionInstanceTemplate_with18TbScratchDisk|TestAccComputeInstanceTemplate_withScratchDisk|TestAccComputeInstanceTemplate_with18TbScratchDisk|TestAccComputeRegionInstanceTemplate_withScratchDisk|TestAccComputeInstanceTemplate_NetworkAttachment|TestAccComputeInstanceTemplate_minCpuPlatform|TestAccComputeInstanceFromTemplate_012_removableFields|TestAccComputeRegionInstanceTemplate_minCpuPlatform|TestAccComputeInstanceFromTemplate_overrideScratchDisk|TestAccComputeRegionInstanceTemplate_guestAcceleratorSkip|TestAccComputeInstanceTemplate_guestAccelerator|TestAccComputeInstanceFromTemplate_overrideAttachedDisk|TestAccComputeInstanceTemplate_primaryAliasIpRange|TestAccComputeInstanceTemplate_secondaryAliasIpRange|TestAccComputeInstanceTemplate_metadata_startup_script|TestAccComputeRegionInstanceTemplate_guestAccelerator|TestAccComputeRegionInstanceTemplate_secondaryAliasIpRange|TestAccComputeInstanceTemplate_subnet_custom|TestAccComputeInstanceTemplate_subnet_auto|TestAccComputeRegionInstanceTemplate_primaryAliasIpRange|TestAccComputeInstanceTemplate_regionDisks|TestAccComputeRegionInstanceTemplate_metadata_startup_script|TestAccComputeInstanceTemplate_guestAcceleratorSkip|TestAccComputeInstanceTemplate_disks|TestAccComputeRegionInstanceTemplate_subnet_custom|TestAccComputeInstanceFromTemplate_overrideBootDisk|TestAccComputeFirewallPolicyRule_multipleRules|TestAccCloudbuildv2Connection_GlePrivUpdateConnection|TestAccComputeInstanceFromRegionTemplate_basic|TestAccCloudbuildv2Connection_GlePrivConnection|TestAccCloudbuildv2Connection_GleOldConnection|TestAccCloudbuildv2Connection_GleConnection|TestAccComputeInstanceFromTemplate_self_link_unique|TestAccCloudbuildv2Connection_GitlabConnection|TestAccComputeInstanceFromTemplate_basic|TestAccBigtableAppProfile_bigtableAppProfileAnyclusterExample|TestAccClouddeployDeliveryPipeline_DeliveryPipeline|TestAccClouddeployDeliveryPipeline_CanaryrunDeliveryPipeline|TestAccClouddeployDeliveryPipeline_CanaryServiceNetworkingDeliveryPipeline|TestAccClouddeployDeliveryPipeline_CanaryDeliveryPipeline

Get to know how VCR tests work

@NickElliot

Copy link
Copy Markdown
Contributor Author

/gcbrun

@modular-magician

Copy link
Copy Markdown
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

TF Conversion: Diff ( 2 files changed, 3 insertions(+), 3 deletions(-))

@modular-magician

Copy link
Copy Markdown
Collaborator

Tests analytics

Total tests: 2848
Passed tests 2551
Skipped tests: 295
Affected tests: 2

Action taken

Found 2 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
TestAccContainerAwsNodePool_BetaBasicHandWritten|TestAccComputeFirewallPolicyRule_multipleRules

Get to know how VCR tests work

@modular-magician

Copy link
Copy Markdown
Collaborator

$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccContainerAwsNodePool_BetaBasicHandWritten[Error message] [Debug log]
TestAccComputeFirewallPolicyRule_multipleRules[Error message] [Debug log]

$\textcolor{red}{\textsf{Please fix these to complete your PR.}}$
View the build log or the debug log for each test

@NickElliot
NickElliot merged commit fe1852c into GoogleCloudPlatform:main Jul 11, 2023
Khaledmohamedrefaat pushed a commit to Khaledmohamedrefaat/magic-modules that referenced this pull request Jul 13, 2023
…cing multiple targets (GoogleCloudPlatform#8127)

* updated type.rb to remove validation on resource refs. moved existing resourceref parsing/validation to a custom_expand

* added resource_ref_with_validation custom expand to all compute resource refs without existing custom expands or url_param_only

* adding expand to the single field in the entirety of MMV1 outside of compute that uses the baseline resourceref logic

* removing resourceref as string
ericayyliu pushed a commit to ericayyliu/magic-modules that referenced this pull request Jul 26, 2023
…cing multiple targets (GoogleCloudPlatform#8127)

* updated type.rb to remove validation on resource refs. moved existing resourceref parsing/validation to a custom_expand

* added resource_ref_with_validation custom expand to all compute resource refs without existing custom expands or url_param_only

* adding expand to the single field in the entirety of MMV1 outside of compute that uses the baseline resourceref logic

* removing resourceref as string
wj-chen pushed a commit to wj-chen/magic-modules that referenced this pull request Aug 1, 2023
…cing multiple targets (GoogleCloudPlatform#8127)

* updated type.rb to remove validation on resource refs. moved existing resourceref parsing/validation to a custom_expand

* added resource_ref_with_validation custom expand to all compute resource refs without existing custom expands or url_param_only

* adding expand to the single field in the entirety of MMV1 outside of compute that uses the baseline resourceref logic

* removing resourceref as string
hao-nan-li pushed a commit to hao-nan-li/magic-modules that referenced this pull request Aug 9, 2023
…cing multiple targets (GoogleCloudPlatform#8127)

* updated type.rb to remove validation on resource refs. moved existing resourceref parsing/validation to a custom_expand

* added resource_ref_with_validation custom expand to all compute resource refs without existing custom expands or url_param_only

* adding expand to the single field in the entirety of MMV1 outside of compute that uses the baseline resourceref logic

* removing resourceref as string
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants