[Feature Request] Gemini Enterprise CMEK index pre-warming, companion seed data store, declarative CmekConfig, and readiness probe #390

Feature Description

Improve the Gemini Enterprise (Gemini for Government) CMEK deployment lifecycle across FAST Stage 1 (fast/stages-aw/1-resman) and the Gemini Enterprise blueprint (blueprints/gemini-enterprise) in four areas:

1. Optional Stage 1 (1-resman) tenant CMEK index pre-warming (enable_gemini_enterprise) so the 24 to 48 hour Discovery Engine single-tenant CMEK search index allocation starts during tenant creation rather than blocking Stage 0 application deployment.
2. Default companion NO_CONTENT seed data store (enable_cmek_seed_data_store = true) in blueprints/gemini-enterprise/gemini-stage-0/discovery-engine.tf so every google_discovery_engine_search_engine instance links at least two data stores (BLENDED_SEARCH), preventing Discovery Engine single-store linkage lock-in (Engines linked to a single data store cannot change their linked data store).
3. Declarative google_discovery_engine_cmek_config resource management in blueprints/gemini-enterprise/gemini-stage-0/cmek.tf with explicit depends_on ordering from google_discovery_engine_data_store resources.
4. A gem4gov datastore check-readiness CLI subcommand and deploy.sh helper option that probes servingConfigs/default_search:search (pageSize=1) to distinguish INDEXING (HTTP 400 FAILED_PRECONDITION while the initial CMEK index is building) from READY (HTTP 200).

Use Case

1. Registering a new regional or multi-regional CmekConfig in Discovery Engine provisions a dedicated single-tenant search index in the background that takes 24 to 48 hours to finish building (FAILED_PRECONDITION: The initial index is being built). Starting this provisioning step only when deploying the Stage 0 workload blueprint delays end-to-end validation by up to two days.
2. When google_discovery_engine_search_engine is created with only a single linked data_store_ids entry (data_store_ids.size() == 1), the Discovery Engine API permanently blocks subsequent PatchEngine updates from adding, swapping, or removing data_store_ids on that engine. Adding a second data store later forces Terraform to destroy and recreate the search engine, invalidating the Stage 1 load balancer backend service and URL map references.
3. Operators running standalone terraform apply inside gemini-stage-0 without deploy.sh need Terraform to register google_discovery_engine_cmek_config before creating any google_discovery_engine_data_store resources.
4. Operators currently have no visibility in gem4gov-cli or deploy.sh into whether the background CMEK search index allocation is still INDEXING or READY.

Proposed Solution

1. In fast/stages-aw/1-resman/variables.tf, fast/stages-aw/1-resman/branch-tenants.tf, fast/stages-aw/1-resman/kms.tf, and fast/stages-aw/1-resman/outputs-tenants.tf, add an opt-in enable_gemini_enterprise = optional(bool, false) tenant attribute that enables discoveryengine.googleapis.com, creates the <env>-us KMS keyring and gemini-enterprise key in us, grants roles/cloudkms.cryptoKeyEncrypterDecrypter to service-<project_number>@gcp-sa-discoveryengine.iam.gserviceaccount.com, registers google_discovery_engine_cmek_config, and provisions an empty NO_CONTENT seed data store (${tenant}-ge-cmek-seed-ds).
2. In blueprints/gemini-enterprise/gemini-stage-0/discovery-engine.tf and blueprints/gemini-enterprise/gemini-stage-0/variables.tf, provision google_discovery_engine_data_store.cmek_seed (enable_cmek_seed_data_store = true by default) alongside default_connector_datastore and wire both into google_discovery_engine_search_engine.gemini_enterprise_search_engine.data_store_ids so data_store_ids always contains at least two stores.
3. In blueprints/gemini-enterprise/gemini-stage-0/cmek.tf, manage google_discovery_engine_cmek_config.default (set_default = true, deletion_policy = "ABANDON") and add depends_on = [google_discovery_engine_cmek_config.default] to all google_discovery_engine_data_store resources in discovery-engine.tf.
4. In blueprints/gemini-enterprise/gem4gov-cli/data_stores.py, blueprints/gemini-enterprise/gem4gov-cli/gem4gov.py, and blueprints/gemini-enterprise/deploy.sh, add gem4gov datastore check-readiness to query servingConfigs/default_search:search (pageSize=1) and report INDEXING versus READY.

Compliance & Deployment Context

• Target Deployment Type(s):
  • US Region Restricted (e.g., Access Policy constraint)
  • FedRAMP Medium
  • FedRAMP High
  • DoD IL4
  • DoD IL5
  • All / General
• Relevant NIST 800-53r5 Controls: SC-12 (Cryptographic Key Establishment and Management), SC-28 (Protection of Information at Rest), CM-2 (Baseline Configuration).

Reusability Check

Stellar Engine prioritizes reusability.

• I have checked if this can be implemented using or extending an existing module in Cloud Foundation Fabric (which Stellar Engine is forked from).
• I have checked if this functionality already exists in another part of Stellar Engine and can be refactored for reuse.

Alternatives Considered

• Waiting until Stage 0 application rollout to register CmekConfig leaves operators blocked for 24 to 48 hours while the single-tenant CMEK search index initializes.
• Creating the search engine with a single data store forces destructive replacement of google_discovery_engine_search_engine whenever additional GCS or BigQuery data stores are linked later.

Additional Context

• Affects fast/stages-aw/1-resman and blueprints/gemini-enterprise (gemini-stage-0, gem4gov-cli, and deploy.sh).