Detected unresolved variables after 1 resolution rounds

[peter-ocelka]

Describe the issue

With bundle validation and deploying using databricks CLI I'm getting:

Warning: Detected unresolved variables after 1 resolution rounds

After deployment everything is resolved correctly, but still I would rather not see this warning.

Configuration

Please note that since I removed company settings these examples might not be exactly reproducible.

databrick.yml

bundle:
  name: xxx

# These are any additional configuration files to include.
include:
  - bundle/jobs/*.yml
  - bundle/variables/*.yml

# Global Variables - the same for each target
variables:
  timeout:
    description: Timeout for the job tasks.
    type: string
    default: 3600

# YAML anchors for job permissions definition
non_production_job_permissions: &non_prod_job_permissions
  permissions:
    - level: CAN_VIEW
      service_principal_name: ${var.tidal_service_account}

targets:
  personal:
    mode: development
    default: true
    presets:
      trigger_pause_status: PAUSED
      jobs_max_concurrent_runs: 1
    workspace:
      host: https://dummy.cloud.databricks.com
    resources:
      jobs:
        xxx_job:
          git_source:
            git_branch: ${var.git_branch}
          <<: *non_prod_job_permissions
    variables:
#      tidal_service_account:
#        description: Gets tidal service account name/id.
#        lookup:
#          service_principal: "TIDALDBServAccount - usdev"
      environment_id:
        description: Environment for custom tags.
        type: string
        default: "DEV"
      account_type:
        description: Environment for custom tags.
        default: "Development"
      uc_catalog:
        description: Unity Catalog prefix.
        default: "usdev"

global_vars.yml

variables:
  developer_id:
    description: "Unique identifier for developer."
    type: string
    default: "service_account"
  git_branch:
    description: Branch where code resides
    type: string
    default: ""
  git_tag:
    description: Tag to which we are pointing the code.
    type: string
    default: ""
#  tidal_service_account:
#    description: Gets tidal service account name/id.
#    default: ""
  tidal_service_account:
    description: Gets tidal service account name/id.
    lookup:
      service_principal: "TIDALDBServAccount - ${var.uc_catalog}"
  custom_tags:
    description: "Tags which should be set on job cluster definition."
    type: complex
    default:
      environment-id: ${var.environment_id}
      account-type: ${var.account_type}
  environment_id:
    description: Environment for custom tags.
    type: string
    default: ""
  account_type:
    description: Environment for custom tags.
    type: string
    default: ""
  uc_catalog:
    description: Unity Catalog prefix.
    type: string
    default: ""

I think that the warning is tied to the way how I'm trying to work with tidal_service_account, because when I use my old approach it works. Was just trying to make it little bit more read-able and matainable.

Steps to reproduce the behavior

Run databricks bundle validate ...

Expected Behavior

I shouldn't see the warning in validation because everything seems to be resolved.

Actual Behavior

Warning: Detected unresolved variables after 1 resolution rounds

Name: xxx
Target: personal
Workspace:

OS and CLI version

OS: Ubuntu WSL
CLI: v0.254.0

Is this a regression?

n/a

Debug Logs

00:51:24 Info: start pid=28242 version=0.254.0 args="databricks, bundle, validate, --target, personal, --profile, dev-svc, --var=git_branch=feat/bundle_changes, --var=developer_id=pocelka, --log-level=debug"
00:51:24 Debug: Found bundle root at /home/pocelka/repos/pah_de/pipelines/xxx (file /home/pocelka/repos/pah_de/pipelines/xxx/databricks.yml) pid=28242
00:51:24 Info: Phase: load pid=28242
00:51:24 Debug: Apply pid=28242 mutator=EntryPoint
00:51:24 Debug: Apply pid=28242 mutator=scripts.preinit
00:51:24 Debug: No script defined for preinit, skipping pid=28242 mutator=scripts.preinit
00:51:24 Debug: Apply pid=28242 mutator=ProcessRootIncludes
00:51:24 Debug: Apply pid=28242 mutator=ProcessRootIncludes mutator=ProcessInclude(bundle/jobs/xxx_job.yml)
00:51:24 Debug: Apply pid=28242 mutator=ProcessRootIncludes mutator=ProcessInclude(bundle/variables/global_vars.yml)
00:51:24 Debug: Apply pid=28242 mutator=VerifyCliVersion
00:51:24 Debug: Apply pid=28242 mutator=EnvironmentsToTargets
00:51:24 Debug: Apply pid=28242 mutator=ComputeIdToClusterId
00:51:24 Debug: Apply pid=28242 mutator=InitializeVariables
00:51:24 Debug: Apply pid=28242 mutator=DefineDefaultTarget(default)
00:51:24 Debug: Apply pid=28242 mutator=validate:unique_resource_keys
00:51:24 Debug: Apply pid=28242 mutator=SelectTarget(personal)
00:51:24 Debug: Apply pid=28242 mutator=<func>
00:51:24 Debug: Loading dev-svc profile from /home/pocelka/.databrickscfg pid=28242 sdk=true
00:51:24 Debug: Apply pid=28242 mutator=<func>
00:51:24 Info: Phase: initialize pid=28242
00:51:24 Debug: Apply pid=28242 mutator=validate:AllResourcesHaveValues
00:51:24 Debug: Apply pid=28242 mutator=validate:interpolation_in_auth_config
00:51:24 Debug: Apply pid=28242 mutator=RewriteSyncPaths
00:51:24 Debug: Apply pid=28242 mutator=SyncDefaultPath
00:51:24 Debug: Apply pid=28242 mutator=SyncInferRoot
00:51:24 Debug: Apply pid=28242 mutator=PopulateCurrentUser
00:51:25 Debug: GET /api/2.0/preview/scim/v2/Me
< HTTP/2.0 200 OK
< } pid=28242 mutator=PopulateCurrentUser sdk=true
00:51:25 Debug: Apply pid=28242 mutator=LoadGitDetails
00:51:25 Debug: Apply pid=28242 mutator=ApplySourceLinkedDeploymentPreset
00:51:25 Debug: Apply pid=28242 mutator=DefineDefaultWorkspaceRoot
00:51:25 Debug: Apply pid=28242 mutator=ExpandWorkspaceRoot
00:51:25 Debug: Apply pid=28242 mutator=DefaultWorkspacePaths
00:51:25 Debug: Apply pid=28242 mutator=PrependWorkspacePrefix
00:51:25 Debug: Apply pid=28242 mutator=RewriteWorkspacePrefix
00:51:25 Debug: Apply pid=28242 mutator=SetVariables
00:51:25 Debug: Apply pid=28242 mutator=ResolveVariableReferences
00:51:25 Debug: Apply pid=28242 mutator=ResolveResourceReferences
00:51:25 Debug: GET /api/2.0/policies/clusters/list
< HTTP/2.0 200 OK
< } pid=28242 mutator=ResolveResourceReferences sdk=true
00:51:26 Debug: GET /api/2.0/preview/scim/v2/ServicePrincipals?count=100&startIndex=1
< HTTP/2.0 200 OK
< } pid=28242 mutator=ResolveResourceReferences sdk=true
00:51:26 Debug: GET /api/2.0/preview/scim/v2/ServicePrincipals?count=100&startIndex=1
< HTTP/2.0 200 OK
< } pid=28242 mutator=ResolveResourceReferences sdk=true
00:51:27 Debug: GET /api/2.0/preview/scim/v2/ServicePrincipals?count=100&startIndex=101
< HTTP/2.0 200 OK
< } pid=28242 mutator=ResolveResourceReferences sdk=true
00:51:27 Debug: GET /api/2.0/preview/scim/v2/ServicePrincipals?count=100&startIndex=101
< HTTP/2.0 200 OK
< } pid=28242 mutator=ResolveResourceReferences sdk=true
00:51:28 Debug: GET /api/2.0/preview/scim/v2/ServicePrincipals?count=100&startIndex=201
< HTTP/2.0 429 Too Many Requests
< {
<   "detail": "REQUEST_LIMIT_EXCEEDED: Workspace 0 exceeded the concurrent limit of 2 requests.",
<   "schemas": [
<     "urn:ietf:params:scim:api:messages:2.0:Error"
<   ],
<   "scimType": "tooMany",
<   "status": "429"
< } pid=28242 mutator=ResolveResourceReferences sdk=true
00:51:28 Debug: GET /api/2.0/preview/scim/v2/ServicePrincipals?count=100&startIndex=201
< HTTP/2.0 200 OK
< } pid=28242 mutator=ResolveResourceReferences sdk=true
00:51:29 Debug: GET /api/2.0/preview/scim/v2/ServicePrincipals?count=100&startIndex=301
< HTTP/2.0 200 OK
< } pid=28242 mutator=ResolveResourceReferences sdk=true
00:51:30 Debug: GET /api/2.0/preview/scim/v2/ServicePrincipals?count=100&startIndex=326
< HTTP/2.0 200 OK
< } pid=28242 mutator=ResolveResourceReferences sdk=true
00:51:30 Debug: GET /api/2.0/preview/scim/v2/ServicePrincipals?count=100&startIndex=201
< HTTP/2.0 200 OK
< } pid=28242 mutator=ResolveResourceReferences sdk=true
00:51:31 Debug: GET /api/2.0/preview/scim/v2/ServicePrincipals?count=100&startIndex=301
< HTTP/2.0 200 OK
< } pid=28242 mutator=ResolveResourceReferences sdk=true
00:51:32 Debug: GET /api/2.0/preview/scim/v2/ServicePrincipals?count=100&startIndex=326
< HTTP/2.0 200 OK
< {
< } pid=28242 mutator=ResolveResourceReferences sdk=true
00:51:32 Debug: Apply pid=28242 mutator=ResolveVariableReferences
00:51:32 Debug: Apply pid=28242 mutator=validate:volume-path
00:51:32 Debug: Apply pid=28242 mutator=ApplyTargetMode
00:51:32 Info: Development mode: disabling deployment lock since bundle.deployment.lock.enabled is not set to true pid=28242 mutator=ApplyTargetMode
00:51:32 Debug: Apply pid=28242 mutator=ConfigureWSFS
00:51:32 Debug: Apply pid=28242 mutator=ProcessStaticResources
00:51:32 Debug: Apply pid=28242 mutator=ProcessStaticResources mutator=ResolveVariableReferences(resources)
00:51:32 Debug: Apply pid=28242 mutator=ProcessStaticResources mutator=NormalizePaths
00:51:32 Debug: Apply pid=28242 mutator=ProcessStaticResources mutator=ResolveVariableReferences(resources)
00:51:32 Debug: Apply pid=28242 mutator=ProcessStaticResources mutator=ExpandPipelineGlobPaths
00:51:32 Debug: Apply pid=28242 mutator=ProcessStaticResources mutator=MergeJobClusters
00:51:32 Debug: Apply pid=28242 mutator=ProcessStaticResources mutator=MergeJobParameters
00:51:32 Debug: Apply pid=28242 mutator=ProcessStaticResources mutator=MergeJobTasks
00:51:32 Debug: Apply pid=28242 mutator=ProcessStaticResources mutator=MergePipelineClusters
00:51:32 Debug: Apply pid=28242 mutator=ProcessStaticResources mutator=MergeApps
00:51:32 Debug: Apply pid=28242 mutator=ProcessStaticResources mutator=CaptureSchemaDependency
00:51:32 Debug: Apply pid=28242 mutator=ProcessStaticResources mutator=ConfigureDashboardSerializedDashboard
00:51:32 Debug: Apply pid=28242 mutator=ProcessStaticResources mutator=SetRunAs
00:51:32 Debug: Apply pid=28242 mutator=ProcessStaticResources mutator=OverrideCompute
00:51:32 Debug: Apply pid=28242 mutator=ProcessStaticResources mutator=ApplyPresets
00:51:32 Debug: Apply pid=28242 mutator=ProcessStaticResources mutator=validate:SingleNodeCluster
00:51:32 Debug: Apply pid=28242 mutator=ProcessStaticResources mutator="SetDefaultMutator([{resources 0} {dashboards 0} {}], parent_path, /Workspace/...)"
00:51:32 Debug: Apply pid=28242 mutator=ProcessStaticResources mutator="SetDefaultMutator([{resources 0} {dashboards 0} {}], embed_credentials, false)"
00:51:32 Debug: Apply pid=28242 mutator=ProcessStaticResources mutator="SetDefaultMutator([{resources 0} {volumes 0} {}], volume_type, MANAGED)"
00:51:32 Debug: Apply pid=28242 mutator=ProcessStaticResources mutator="SetDefaultMutator([{resources 0} {jobs 0} {}], name, Untitled)"
00:51:32 Debug: Apply pid=28242 mutator=ProcessStaticResources mutator="SetDefaultMutator([{resources 0} {jobs 0} {}], max_concurrent_runs, 1)"
00:51:32 Debug: Apply pid=28242 mutator=ProcessStaticResources mutator="SetDefaultMutator([{resources 0} {jobs 0} {} {schedule 0}], pause_status, UNPAUSED)"
00:51:32 Debug: Apply pid=28242 mutator=ProcessStaticResources mutator="SetDefaultMutator([{resources 0} {jobs 0} {} {trigger 0}], pause_status, UNPAUSED)"
00:51:32 Debug: Apply pid=28242 mutator=ProcessStaticResources mutator="SetDefaultMutator([{resources 0} {jobs 0} {} {continuous 0}], pause_status, UNPAUSED)"
00:51:32 Debug: Apply pid=28242 mutator=ProcessStaticResources mutator="SetDefaultMutator([{resources 0} {jobs 0} {} {task 0} {} {dbt_task 0}], schema, default)"
00:51:32 Debug: Apply pid=28242 mutator=ProcessStaticResources mutator="SetDefaultMutator([{resources 0} {jobs 0} {} {task 0} {} {for_each_task 0} {task 0} {dbt_task 0}], schema, default)"
00:51:32 Debug: Apply pid=28242 mutator=ProcessStaticResources mutator="SetDefaultMutator([{resources 0} {jobs 0} {} {job_clusters 0} {} {new_cluster 0}], num_workers, 0)"
00:51:32 Debug: Apply pid=28242 mutator=ProcessStaticResources mutator="SetDefaultMutator([{resources 0} {jobs 0} {} {job_clusters 0} {} {new_cluster 0} {workload_type 0} {clients 0}], notebooks, true)"
00:51:32 Debug: Apply pid=28242 mutator=ProcessStaticResources mutator="SetDefaultMutator([{resources 0} {jobs 0} {} {job_clusters 0} {} {new_cluster 0} {workload_type 0} {clients 0}], jobs, true)"
00:51:32 Debug: Apply pid=28242 mutator=ProcessStaticResources mutator="SetDefaultMutator([{resources 0} {pipelines 0} {}], edition, ADVANCED)"
00:51:32 Debug: Apply pid=28242 mutator=ProcessStaticResources mutator="SetDefaultMutator([{resources 0} {pipelines 0} {}], channel, CURRENT)"
00:51:32 Debug: Apply pid=28242 mutator=ProcessStaticResources mutator=DefaultQueueing
00:51:32 Debug: Apply pid=28242 mutator=ProcessStaticResources mutator=ApplyBundlePermissions
00:51:32 Debug: Apply pid=28242 mutator=ProcessStaticResources mutator=FilterCurrentUserFromPermissions
00:51:32 Debug: Apply pid=28242 mutator=PythonMutator(load)
00:51:32 Debug: Apply pid=28242 mutator=PythonMutator(init)
00:51:32 Debug: Apply pid=28242 mutator=PythonMutator(load_resources)
00:51:32 Debug: Apply pid=28242 mutator=PythonMutator(apply_mutators)
00:51:32 Debug: Apply pid=28242 mutator=CheckPermissions
00:51:32 Debug: Apply pid=28242 mutator=TranslatePaths
00:51:32 Debug: Apply pid=28242 mutator=PythonWrapperWarning
00:51:32 Debug: Apply pid=28242 mutator=artifacts.Prepare
00:51:32 Info: No local tasks in databricks.yml config, skipping auto detect pid=28242 mutator=artifacts.Prepare
00:51:32 Debug: Apply pid=28242 mutator=apps.Validate
00:51:32 Debug: Apply pid=28242 mutator=ValidateTargetMode
00:51:32 Debug: Apply pid=28242 mutator=ValidateSharedRootPermissions
00:51:32 Debug: Apply pid=28242 mutator=metadata.AnnotateJobs
00:51:32 Debug: Apply pid=28242 mutator=metadata.AnnotatePipelines
00:51:32 Debug: Apply pid=28242 mutator=terraform.Initialize
00:51:32 Debug: DATABRICKS_TF_EXEC_PATH is not defined pid=28242 mutator=terraform.Initialize
00:51:32 Debug: Using Terraform at /home/pocelka/repos/pah_de/pipelines/xxx/.databricks/bundle/personal/bin/terraform pid=28242 mutator=terraform.Initialize
00:51:32 Debug: DATABRICKS_TF_CLI_CONFIG_FILE is not defined pid=28242 mutator=terraform.Initialize
00:51:32 Debug: Environment variables for Terraform: DATABRICKS_HOST, DATABRICKS_TOKEN, DATABRICKS_AUTH_TYPE, HOME, PATH, HTTP_PROXY, DATABRICKS_USER_AGENT_EXTRA pid=28242 mutator=terraform.Initialize
00:51:32 Debug: Apply pid=28242 mutator=scripts.postinit
00:51:32 Debug: No script defined for postinit, skipping pid=28242 mutator=scripts.postinit
00:51:32 Debug: ApplyParallel pid=28242 mutator=fast_validate(readonly)
00:51:32 Debug: ApplyParallel pid=28242 mutator=validate:files_to_sync
00:51:32 Debug: ApplyParallel pid=28242 mutator=validate:folder_permissions
00:51:32 Debug: ApplyParallel pid=28242 mutator=validate:validate_sync_patterns
00:51:32 Debug: ApplyParallel pid=28242 mutator=fast_validate(readonly) mutator=validate:job_cluster_key_defined
00:51:32 Debug: ApplyParallel pid=28242 mutator=fast_validate(readonly) mutator=validate:job_task_cluster_spec
00:51:32 Debug: ApplyParallel pid=28242 mutator=fast_validate(readonly) mutator=validate:artifact_paths
00:51:33 Debug: GET /api/2.0/workspace/get-status?path=/Workspace/...
< HTTP/2.0 404 Not Found
< {
<   "error_code": "RESOURCE_DOES_NOT_EXIST",
<   "message": "Path (/Workspace/... (17 more bytes)"
< } pid=28242 mutator=validate:files_to_sync sdk=true
00:51:33 Debug: non-retriable error: Path (/Workspace/...) doesn't exist. pid=28242 mutator=validate:files_to_sync sdk=true
00:51:33 Debug: POST /api/2.0/workspace/mkdirs
> {
>   "path": "/Workspace/..."
> }
< HTTP/2.0 200 OK
< {} pid=28242 mutator=validate:files_to_sync sdk=true
00:51:33 Debug: GET /api/2.0/workspace/get-status?path=/Workspace/...
< HTTP/2.0 200 OK
< {
<   "object_id": 542845632094699,
<   "object_type": "DIRECTORY",
<   "path": "/Workspace/...",
<   "resource_id": "542845632094699"
< } pid=28242 mutator=validate:files_to_sync sdk=true
00:51:33 Debug: Path /Workspace/... pid=28242 mutator=validate:files_to_sync
Warning: Detected unresolved variables after 1 resolution rounds

Name: xxx
Target: personal
Workspace:
  ...

Found 1 warning
00:51:33 Info: completed execution pid=28242 exit_code=0
00:51:33 Debug: no telemetry logs to upload pid=28242

Looking into the debug log I got HTTP/2.0 429 Too Many Requests during one of the api calls for service principals. Could this be related? Can be the limit somehow increased by our platform team?

EDIT:
I tried it 2nd time with debug log enabled. This time I didn't have any HTTP 429, but still got warning at the end.

[denik]

Thanks for the report, we'll take a look.

[denik]

Fixed in #3054

[peter-ocelka]

@denik A very quick question. Any chance that warning message could also include more information if some variables are left unresolved? I think it would help users with debug or thinking about other possible solutions?

[denik]

@peter-ocelka thanks, that's a good suggestion. The current design of this does not make this information readily available, so it would require some additional work. On the other hand we're looking to revamp variable resolution at some point, so we'll probably tackle it then.