Add tool to generate services.kt file that contains mmv1 and tpgtools/DCL services

[SarahFrench]

Problem

Currently this PR #8254 generates the .teamcity folder in the downstream TPG and TPGB repos, and the generated code creates projects and build configurations in TeamCity without issue. One of the files needed for this is a map of the service packages, found in .teamcity/components/generated/services.kt

However, when this map of services is generated it contains only services generated by MMv1 and lacks services generated by tpgtools/DCL. This be because MMv1 generates files before tpgtools (see ordering here), so when the files in the .teamcity/ folder are generated the DCL services aren't created yet. The generator code iterates through folders in ./google/services/ and finds an incomplete list.

Proposed solution

The solutions for this could be:

• Make the file mostly manually curated, with unless version == 'ga' logic controlling inclusion of beta-only services
  • This is not a good idea as it will fall out of date
• Move generation of the services.kt file to tpgtools/DCL as it runs last in the generation process, so it will contain all services
• Update tpgtools/DCL to generate a second map of services, e.g. services-dcl.kt file, and update the Kotlin code to merge the two maps together when it creates build configs for each service.
• Create a separate tool to create the single services.kt file after both MMv1 and tpgtools/DCL are finished

I've chosen to do the last option of a separate tool, because this can be deleted away from the repo once all the DCL services are migrated to MMv1, all without any changes to the existing Kotlin code.

---

If this PR is for Terraform, I acknowledge that I have:

• Searched through the issue tracker for an open issue that this either resolves or contributes to, commented on it to claim it, and written "fixes {url}" or "part of {url}" in this PR description. If there were no relevant open issues, I opened one and commented that I would like to work on it (not necessary for very small changes).
• Ensured that all new fields I added that can be set by a user appear in at least one example (for generated resources) or third_party test (for handwritten resources or update tests).
• Generated Terraform providers, and ran make test and make lint in the generated providers to ensure it passes unit and linter tests.
• Ran relevant acceptance tests using my own Google Cloud project and credentials (If the acceptance tests do not yet pass or you are unable to run them, please let your reviewer know).
• Read Write release notes before writing my release note below.

Release Note Template for Downstream PRs (will be copied)

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR hasn't generated any diffs, but I'll let you know if a future commit does.

[SarahFrench]

👆 not sure if there are no diffs due to an error in my work, or protection in the CI against arbitrary code being executed

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR hasn't generated any diffs, but I'll let you know if a future commit does.

[zli82016]

👆 not sure if there are no diffs due to an error in my work, or protection in the CI against arbitrary code being executed

I tested it locally and don't see errors. The DCL service packages are generated in the downstream providers.

[zli82016]

LGTM. Thanks so much for the quick fix.

[modular-magician]

Tests analytics

Total tests: 2853
Passed tests 2553
Skipped tests: 297
Affected tests: 3

Action taken

Found 3 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccComputeFirewallPolicyRule_multipleRules|TestAccSpannerDatabaseIamPolicy|TestAccContainerAwsNodePool_BetaBasicHandWritten

Get to know how VCR tests work

[modular-magician]

$\textcolor{green}{\textsf{Tests passed during RECORDING mode:}}$
TestAccSpannerDatabaseIamPolicy[Debug log]

Rerun these tests in REPLAYING mode to catch issues

$\textcolor{green}{\textsf{No issues found for passed tests after REPLAYING rerun.}}$

---

$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccComputeFirewallPolicyRule_multipleRules[Error message] [Debug log]
TestAccContainerAwsNodePool_BetaBasicHandWritten[Error message] [Debug log]

$\textcolor{red}{\textsf{Please fix these to complete your PR.}}$
View the build log or the debug log for each test

[SarahFrench]

@zli82016 I'm a bit nervous about merging this - is it definitely not going to impact the alpha/google-private provider?

Edit: 😅 I realised that the make terraform command isn't actually used in the scripts to generate the downstream, and instead MMv1 is generated using bundle and tpgtools is triggered via its Make target. My commit below is trying to add use of make teamcity-servicemap-generate in a way that only affects TPG and TPGB

[zli82016]

@zli82016 I'm a bit nervous about merging this - is it definitely not going to impact the alpha/google-private provider?

Edit: 😅 I realised that the make terraform command isn't actually used in the scripts to generate the downstream, and instead MMv1 is generated using bundle and tpgtools is triggered via its Make target. My commit below is trying to add use of make teamcity-servicemap-generate in a way that only affects TPG and TPGB

Yeah, I think it does not affect impact alpha provider.

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 1 file changed, 9 insertions(+), 1 deletion(-))
Terraform Beta: Diff ( 1 file changed, 8 insertions(+), 1 deletion(-))
TF Conversion: Diff ( 2 files changed, 3 insertions(+), 3 deletions(-))

[modular-magician]

Tests analytics

Total tests: 2854
Passed tests 2555
Skipped tests: 297
Affected tests: 2

Action taken

Found 2 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccContainerAwsNodePool_BetaBasicHandWritten|TestAccComputeFirewallPolicyRule_multipleRules

Get to know how VCR tests work

[modular-magician]

$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccContainerAwsNodePool_BetaBasicHandWritten[Error message] [Debug log]
TestAccComputeFirewallPolicyRule_multipleRules[Error message] [Debug log]

$\textcolor{red}{\textsf{Please fix these to complete your PR.}}$
View the build log or the debug log for each test

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR hasn't generated any diffs, but I'll let you know if a future commit does.

[modular-magician]

Tests analytics

Total tests: 2856
Passed tests 2555
Skipped tests: 299
Affected tests: 2

Action taken

Found 2 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccContainerAwsNodePool_BetaBasicHandWritten|TestAccComputeFirewallPolicyRule_multipleRules

Get to know how VCR tests work

[modular-magician]

$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccContainerAwsNodePool_BetaBasicHandWritten[Error message] [Debug log]
TestAccComputeFirewallPolicyRule_multipleRules[Error message] [Debug log]

$\textcolor{red}{\textsf{Please fix these to complete your PR.}}$
View the build log or the debug log for each test