Terraform Support for Dataplex Tasks.

[sachitjaggi156]

If this PR is for Terraform, I acknowledge that I have:

• Searched through the issue tracker for an open issue that this either resolves or contributes to, commented on it to claim it, and written "fixes {url}" or "part of {url}" in this PR description. If there were no relevant open issues, I opened one and commented that I would like to work on it (not necessary for very small changes).
• Ensured that all new fields I added that can be set by a user appear in at least one example (for generated resources) or third_party test (for handwritten resources or update tests).
• Generated Terraform providers, and ran make test and make lint in the generated providers to ensure it passes unit and linter tests.
• Ran relevant acceptance tests using my own Google Cloud project and credentials (If the acceptance tests do not yet pass or you are unable to run them, please let your reviewer know).
• Read the Release Notes Guide before writing my release note below.

Release Note Template for Downstream PRs (will be copied)

`google_dataplex_task`

[modular-magician]

Hello! I am a robot who works on Magic Modules PRs.

I've detected that you're a community contributor. @NickElliot, a repository maintainer, has been assigned to assist you and help review your changes.

❓ First time contributing? Click here for more details

---

Your assigned reviewer will help review your code by:

• Ensuring it's backwards compatible, covers common error cases, etc.
• Summarizing the change into a user-facing changelog note.
• Passes tests, either our "VCR" suite, a set of presubmit tests, or with manual test runs.

You can help make sure that review is quick by running local tests and ensuring they're passing in between each push you make to your PR's branch. Also, try to leave a comment with each push you make, as pushes generally don't generate emails.

If your reviewer doesn't get back to you within a week after your most recent change, please feel free to leave a comment on the issue asking them to take a look! In the absence of a dedicated review dashboard most maintainers manage their pending reviews through email, and those will sometimes get lost in their inbox.

---

[melinath]

It looks like this failure might resolve itself with another /gcbrun - giving it a shot.

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 9 files changed, 3709 insertions(+), 3 deletions(-))
Terraform Beta: Diff ( 9 files changed, 3709 insertions(+), 3 deletions(-))
TF Conversion: Diff ( 6 files changed, 1153 insertions(+), 3 deletions(-))

[modular-magician]

Tests analytics

Total tests: 2786
Passed tests 2483
Skipped tests: 294
Affected tests: 9

Action taken

Found 9 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccDataSourceGoogleFirebaseAndroidAppConfig|TestAccDataplexTask_dataplexTaskPrimaryExample|TestAccComputeNetworkEndpoints_networkEndpointsBasic|TestAccDataSourceAlloydbLocations_basic|TestAccComputeFirewallPolicyRule_multipleRules|TestAccApigeeKeystoresAliasesKeyCertFile_apigeeKeystoresAliasesKeyCertFileTestExample|TestAccDataplexTaskIamPolicyGenerated|TestAccDataplexTaskIamBindingGenerated|TestAccDataplexTaskIamMemberGenerated

Get to know how VCR tests work

[melinath]

Note: the missing-test and breaking-change detector unit tests are failing for unrelated reasons.

[modular-magician]

Tests passed during RECORDING mode:
TestAccDataSourceGoogleFirebaseAndroidAppConfig[Debug log]
TestAccDataplexTask_dataplexTaskPrimaryExample[Debug log]
TestAccComputeNetworkEndpoints_networkEndpointsBasic[Debug log]
TestAccDataSourceAlloydbLocations_basic[Debug log]
TestAccApigeeKeystoresAliasesKeyCertFile_apigeeKeystoresAliasesKeyCertFileTestExample[Debug log]

Tests failed during RECORDING mode:
TestAccComputeFirewallPolicyRule_multipleRules[Error message] [Debug log]
TestAccDataplexTaskIamPolicyGenerated[Error message] [Debug log]
TestAccDataplexTaskIamBindingGenerated[Error message] [Debug log]
TestAccDataplexTaskIamMemberGenerated[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

[NickElliot]

Added a few comments on the implementation of the resource itself.

Other than that, because it looks like this resource is mostly updateable in-place, could you add some handwritten update tests that flex that functionality?

[sachitjaggi156]

Added a few comments on the implementation of the resource itself.

Other than that, because it looks like this resource is mostly undateable in-place, could you add some handwritten update tests that flex that functionality?

Have a customer deadline so removing the update functionality for now .

Will provide support for update in a later PR.

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 9 files changed, 3709 insertions(+), 3 deletions(-))
Terraform Beta: Diff ( 9 files changed, 3709 insertions(+), 3 deletions(-))
TF Conversion: Diff ( 6 files changed, 1153 insertions(+), 3 deletions(-))

Missing test report

Your PR includes resource fields which are not covered by any test.

Resource: google_dataplex_task (6 total tests)
Untested fields: trigger_spec.max_retries, trigger_spec.schedule, trigger_spec.start_time, trigger_spec.disabled, notebook.notebook, notebook.archive_uris, notebook.file_uris, notebook.infrastructure_spec.container_image.properties, notebook.infrastructure_spec.container_image.python_packages, notebook.infrastructure_spec.container_image.image, notebook.infrastructure_spec.container_image.java_jars, notebook.infrastructure_spec.vpc_network.network, notebook.infrastructure_spec.vpc_network.network_tags, notebook.infrastructure_spec.vpc_network.sub_network, notebook.infrastructure_spec.batch.executors_count, notebook.infrastructure_spec.batch.max_executors_count, display_name, spark.sql_script, spark.sql_script_file, spark.archive_uris, spark.file_uris, spark.infrastructure_spec.batch.executors_count, spark.infrastructure_spec.batch.max_executors_count, spark.infrastructure_spec.container_image.image, spark.infrastructure_spec.container_image.java_jars, spark.infrastructure_spec.container_image.properties, spark.infrastructure_spec.container_image.python_packages, spark.infrastructure_spec.vpc_network.network, spark.infrastructure_spec.vpc_network.network_tags, spark.infrastructure_spec.vpc_network.sub_network, spark.main_class, spark.main_jar_file_uri, execution_spec.args, execution_spec.kms_key, execution_spec.max_job_execution_lifetime, execution_spec.project, description, labels

Please add acceptance tests which include these fields.
Resource: google_dataplex_task_iam_binding (2 total tests)
Untested fields: task_id, condition.expression, condition.title, condition.description

Please add acceptance tests which include these fields.
Resource: google_dataplex_task_iam_member (1 total tests)
Untested fields: condition.title, condition.description, condition.expression, task_id

Please add acceptance tests which include these fields.
Resource: google_dataplex_task_iam_policy (2 total tests)
Untested fields: task_id

Please add acceptance tests which include these fields.

[modular-magician]

Tests analytics

Total tests: 2800
Passed tests 2494
Skipped tests: 301
Affected tests: 5

Action taken

Found 5 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccComputeNetworkEndpoints_networkEndpointsBasic|TestAccComputeFirewallPolicyRule_multipleRules|TestAccDataplexTaskIamMemberGenerated|TestAccDataplexTaskIamBindingGenerated|TestAccDataplexTaskIamPolicyGenerated

Get to know how VCR tests work

[modular-magician]

Tests passed during RECORDING mode:
TestAccComputeNetworkEndpoints_networkEndpointsBasic[Debug log]

Tests failed during RECORDING mode:
TestAccComputeFirewallPolicyRule_multipleRules[Error message] [Debug log]
TestAccDataplexTaskIamMemberGenerated[Error message] [Debug log]
TestAccDataplexTaskIamBindingGenerated[Error message] [Debug log]
TestAccDataplexTaskIamPolicyGenerated[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

[NickElliot]

As this resource is entirely standard, there is not a major reason to avoid launching with update support, and if it can be updated in place, it needs an update test.

It shouldn't need to be a particularly complicated update test as the point is mostly to flex that functionality period (that the resource is able to undergo applying a change, then having that changed state imported) rather than that every single field can be updated. Here is an example. If there is any questions I'll try to respond ASAP.

However regardless of the update support/not, we do still need full detailed test samples added that cover all the optional fields (at least two additional samples needed -- one for spark, one for notebook). Thanks!

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 10 files changed, 3888 insertions(+), 3 deletions(-))
Terraform Beta: Diff ( 10 files changed, 3888 insertions(+), 3 deletions(-))
TF Conversion: Diff ( 6 files changed, 1153 insertions(+), 3 deletions(-))

Missing test report

Your PR includes resource fields which are not covered by any test.

Resource: google_dataplex_task (8 total tests)
Untested fields: description, display_name, trigger_spec.disabled, trigger_spec.max_retries, trigger_spec.schedule, trigger_spec.start_time, labels, notebook.notebook, notebook.archive_uris, notebook.file_uris, notebook.infrastructure_spec.container_image.image, notebook.infrastructure_spec.container_image.java_jars, notebook.infrastructure_spec.container_image.properties, notebook.infrastructure_spec.container_image.python_packages, notebook.infrastructure_spec.vpc_network.network_tags, notebook.infrastructure_spec.vpc_network.sub_network, notebook.infrastructure_spec.vpc_network.network, notebook.infrastructure_spec.batch.executors_count, notebook.infrastructure_spec.batch.max_executors_count, spark.main_jar_file_uri, spark.sql_script, spark.sql_script_file, spark.archive_uris, spark.file_uris, spark.infrastructure_spec.batch.executors_count, spark.infrastructure_spec.batch.max_executors_count, spark.infrastructure_spec.container_image.image, spark.infrastructure_spec.container_image.java_jars, spark.infrastructure_spec.container_image.properties, spark.infrastructure_spec.container_image.python_packages, spark.infrastructure_spec.vpc_network.network, spark.infrastructure_spec.vpc_network.network_tags, spark.infrastructure_spec.vpc_network.sub_network, spark.main_class, execution_spec.project, execution_spec.args, execution_spec.kms_key, execution_spec.max_job_execution_lifetime

Please add acceptance tests which include these fields.
Resource: google_dataplex_task_iam_binding (2 total tests)
Untested fields: condition.expression, condition.title, condition.description, task_id

Please add acceptance tests which include these fields.
Resource: google_dataplex_task_iam_member (1 total tests)
Untested fields: task_id, condition.expression, condition.title, condition.description

Please add acceptance tests which include these fields.
Resource: google_dataplex_task_iam_policy (2 total tests)
Untested fields: task_id

Please add acceptance tests which include these fields.

[modular-magician]

Tests analytics

Total tests: 13
Passed tests 13
Skipped tests: 0
Affected tests: 0

Errors occurred during REPLAYING mode. Please fix them to complete your PR
View the build log

[NickElliot]

Addressed questions and issues. All that is missing now is the expanded full test coverage :)

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 10 files changed, 3882 insertions(+), 3 deletions(-))
Terraform Beta: Diff ( 10 files changed, 3882 insertions(+), 3 deletions(-))
TF Conversion: Diff ( 6 files changed, 1153 insertions(+), 3 deletions(-))

Missing test report

Your PR includes resource fields which are not covered by any test.

Resource: google_dataplex_task (8 total tests)
Untested fields: description, execution_spec.args, execution_spec.kms_key, execution_spec.max_job_execution_lifetime, execution_spec.project, spark.file_uris, spark.infrastructure_spec.vpc_network.sub_network, spark.infrastructure_spec.vpc_network.network, spark.infrastructure_spec.vpc_network.network_tags, spark.infrastructure_spec.batch.executors_count, spark.infrastructure_spec.batch.max_executors_count, spark.infrastructure_spec.container_image.image, spark.infrastructure_spec.container_image.java_jars, spark.infrastructure_spec.container_image.properties, spark.infrastructure_spec.container_image.python_packages, spark.main_class, spark.main_jar_file_uri, spark.sql_script, spark.sql_script_file, spark.archive_uris, labels, trigger_spec.schedule, trigger_spec.start_time, trigger_spec.disabled, trigger_spec.max_retries, notebook.archive_uris, notebook.file_uris, notebook.infrastructure_spec.vpc_network.network_tags, notebook.infrastructure_spec.vpc_network.sub_network, notebook.infrastructure_spec.vpc_network.network, notebook.infrastructure_spec.batch.executors_count, notebook.infrastructure_spec.batch.max_executors_count, notebook.infrastructure_spec.container_image.image, notebook.infrastructure_spec.container_image.java_jars, notebook.infrastructure_spec.container_image.properties, notebook.infrastructure_spec.container_image.python_packages, notebook.notebook, display_name

Please add acceptance tests which include these fields.
Resource: google_dataplex_task_iam_binding (2 total tests)
Untested fields: condition.title, condition.description, condition.expression

Please add acceptance tests which include these fields.
Resource: google_dataplex_task_iam_member (1 total tests)
Untested fields: condition.description, condition.expression, condition.title

Please add acceptance tests which include these fields.

[modular-magician]

Tests analytics

Total tests: 2821
Passed tests 2513
Skipped tests: 301
Affected tests: 7

Action taken

Found 7 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccDataplexTaskDataplexTask_update|TestAccDataplexTask_dataplexTaskBasicExample|TestAccComputeNetworkEndpoints_networkEndpointsBasic|TestAccComputeFirewallPolicyRule_multipleRules|TestAccDataplexTaskIamMemberGenerated|TestAccDataplexTaskIamBindingGenerated|TestAccDataplexTaskIamPolicyGenerated

Get to know how VCR tests work

[modular-magician]

Tests passed during RECORDING mode:
TestAccDataplexTaskDataplexTask_update[Debug log]
TestAccDataplexTask_dataplexTaskBasicExample[Debug log]
TestAccComputeNetworkEndpoints_networkEndpointsBasic[Debug log]

Tests failed during RECORDING mode:
TestAccComputeFirewallPolicyRule_multipleRules[Error message] [Debug log]
TestAccDataplexTaskIamMemberGenerated[Error message] [Debug log]
TestAccDataplexTaskIamBindingGenerated[Error message] [Debug log]
TestAccDataplexTaskIamPolicyGenerated[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

[NickElliot]

You did not update the correct area and caused the build to fail. Please look at the existing comments I provided. Thanks!

[sachitjaggi156]

You did not update the correct area and caused the build to fail. Please look at the existing comments I provided. Thanks!

Sorry by mistake had updated the wrong verb.

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 10 files changed, 4297 insertions(+), 3 deletions(-))
Terraform Beta: Diff ( 10 files changed, 4297 insertions(+), 3 deletions(-))
TF Conversion: Diff ( 6 files changed, 1153 insertions(+), 3 deletions(-))

Missing test report

Your PR includes resource fields which are not covered by any test.

Resource: google_dataplex_task (10 total tests)
Untested fields: notebook.infrastructure_spec.vpc_network.sub_network, spark.infrastructure_spec.vpc_network.network, spark.main_class, spark.main_jar_file_uri, spark.sql_script_file

Please add acceptance tests which include these fields.
Resource: google_dataplex_task_iam_binding (2 total tests)
Untested fields: condition.title, condition.description, condition.expression

Please add acceptance tests which include these fields.
Resource: google_dataplex_task_iam_member (1 total tests)
Untested fields: condition.expression, condition.title, condition.description

Please add acceptance tests which include these fields.

[modular-magician]

Tests analytics

Total tests: 2860
Passed tests 2552
Skipped tests: 297
Affected tests: 11

Action taken

Found 11 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccDataplexTask_dataplexTaskNotebookExample|TestAccDataplexTask_dataplexTaskSparkExample|TestAccDataplexTaskDataplexTask_update|TestAccContainerAwsNodePool_BetaBasicHandWritten|TestAccVertexAIIndex_updated|TestAccVertexAIIndex_vertexAiIndexStreamingExample|TestAccVertexAIIndex_vertexAiIndexExample|TestAccComputeFirewallPolicyRule_multipleRules|TestAccDataplexTaskIamPolicyGenerated|TestAccDataplexTaskIamMemberGenerated|TestAccDataplexTaskIamBindingGenerated

Get to know how VCR tests work

[modular-magician]

$\textcolor{green}{\textsf{Tests passed during RECORDING mode:}}$
TestAccDataplexTask_dataplexTaskNotebookExample[Debug log]
TestAccDataplexTask_dataplexTaskSparkExample[Debug log]
TestAccVertexAIIndex_updated[Debug log]
TestAccVertexAIIndex_vertexAiIndexStreamingExample[Debug log]
TestAccVertexAIIndex_vertexAiIndexExample[Debug log]
TestAccDataplexTaskIamPolicyGenerated[Debug log]
TestAccDataplexTaskIamMemberGenerated[Debug log]
TestAccDataplexTaskIamBindingGenerated[Debug log]

Rerun these tests in REPLAYING mode to catch issues

$\textcolor{green}{\textsf{No issues found for passed tests after REPLAYING rerun.}}$

---

$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccDataplexTaskDataplexTask_update[Error message] [Debug log]
TestAccContainerAwsNodePool_BetaBasicHandWritten[Error message] [Debug log]
TestAccComputeFirewallPolicyRule_multipleRules[Error message] [Debug log]

$\textcolor{red}{\textsf{Please fix these to complete your PR.}}$
View the build log or the debug log for each test

[NickElliot]

The wrong verb was configured when undoing your preceding commit.

[NickElliot]

Suggested change

	update_verb: :POST
	update_verb: :PATCH

When you changed this back you put the incorrect update verb.

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 10 files changed, 4297 insertions(+), 3 deletions(-))
Terraform Beta: Diff ( 10 files changed, 4297 insertions(+), 3 deletions(-))
TF Conversion: Diff ( 6 files changed, 1153 insertions(+), 3 deletions(-))

Missing test report

Your PR includes resource fields which are not covered by any test.

Resource: google_dataplex_task (10 total tests)
Untested fields: notebook.infrastructure_spec.vpc_network.sub_network, spark.main_jar_file_uri, spark.sql_script_file, spark.infrastructure_spec.vpc_network.network, spark.main_class

Please add acceptance tests which include these fields.
Resource: google_dataplex_task_iam_binding (2 total tests)
Untested fields: condition.description, condition.expression, condition.title

Please add acceptance tests which include these fields.
Resource: google_dataplex_task_iam_member (1 total tests)
Untested fields: condition.expression, condition.title, condition.description

Please add acceptance tests which include these fields.

[modular-magician]

Tests analytics

Total tests: 2860
Passed tests 2561
Skipped tests: 297
Affected tests: 2

Action taken

Found 2 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccComputeFirewallPolicyRule_multipleRules|TestAccContainerAwsNodePool_BetaBasicHandWritten

Get to know how VCR tests work

[modular-magician]

$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccComputeFirewallPolicyRule_multipleRules[Error message] [Debug log]
TestAccContainerAwsNodePool_BetaBasicHandWritten[Error message] [Debug log]

$\textcolor{red}{\textsf{Please fix these to complete your PR.}}$
View the build log or the debug log for each test

[NickElliot]

LGTM :)

To @trodge for final review