Non schema conform fail_on_active_runs validates and deploys, fail_on_active_runs forces to set lock property

[janstoecklin-mb]

Describe the issue

If a databricks.yaml file includes the following section the vscode language server extensions shows the following errors.

Invalid yaml

bundle:
  name:  "bundlebug"
  deployment:
    fail_on_active_runs: "true"

Errors

Missing property "lock".
Incorrect type. Expected "boolean".

Despite this part is not conform to the schema, the bundle validates and can be deployed and the fail_on_active_runs works as expected, when a job is deployed that currently runs

According to the schema, this is how a valid configuration shall look (note: lock must be present):

bundle:
  name:  "bundlebug"
  deployment:
    lock:
      enabled: true
    fail_on_active_runs: true

Configuration

# yaml-language-server: $schema=bundle_schema_config.json

bundle:
  name:  "bundlebug"
  deployment:
    fail_on_active_runs: "true"

workspace:
  root_path: "/Users/${workspace.current_user.userName}/bundle-deployments/${bundle.name}/${bundle.target}"

resources:
  jobs:
    New_Job_2024_07_03_15_10_40:
      name: New Job 2024-07-03 15:10:40
      tasks:
        - task_key: minimal
          notebook_task:
            notebook_path: test.py
          job_cluster_key: Job_cluster
      job_clusters:
        - job_cluster_key: Job_cluster
          new_cluster:
            spark_version: 14.3.x-scala2.12
            azure_attributes:
              first_on_demand: 1
              availability: ON_DEMAND_AZURE
              spot_bid_max_price: -1
            node_type_id: Standard_D4ds_v5
            spark_env_vars:
              PYSPARK_PYTHON: /databricks/python3/bin/python3
            enable_elastic_disk: true
            data_security_mode: LEGACY_SINGLE_USER_STANDARD
            runtime_engine: PHOTON
            num_workers: 8
      queue:
        enabled: true

Steps to reproduce the behavior

• databricks bundle schema > bundle_schema_config.json
• configure vs-code extension to use language server as described here:
  https://docs.databricks.com/en/dev-tools/bundles/work-tasks.html

root@a96f51496585:/workspace# databricks bundle validate -t cmi_int        
Name: bundlebug
Target: cmi_int
Workspace:
  Host: https://adb-XXX.azuredatabricks.net
  User: XXX
  Path: /Users/XXX.net/bundle-deployments/bundlebug/cmi_int

Validation OK!

root@a96f51496585:/workspace# databricks bundle deploy -t cmi_int 
Uploading bundle files to /Users/XXX/bundle-deployments/bundlebug/cmi_int/files...
Deploying resources...
Updating deployment state...
Deployment complete!

• Both do not return an error and the fail_on_active_runs option during a deployment actually works as expected

Expected Behavior

• if something is not according to the bundle schema, validate and deploy shall fail, ideally printing the reasons similar to the vscode language server extension
• using fail_on_active_runs forces to set a value for lock, which is unexpected

Actual Behavior

• although invalid, it can be validated and deployed successfully
• using fail_on_active_runs forces me to set lock
• the actual behavior of fail_on_active_runs works as expected, meaning if a job is running, it is not being deployed

OS and CLI version

• inside dev container, base image ubuntu:23.04
  • root@a96f51496585:/workspace# uname -a
    Linux a96f51496585 6.6.26-linuxkit Bump github.com/databrickslabs/terraform-provider-databricks from 0.5.7 to 0.5.8 #1 SMP Sat Apr 27 04:13:19 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux
  • Databricks CLI v0.222.0
  • redhat.vscode-yaml, 1.15
• host where vscode runs
  • Sonoma 14.5 (23F79)
  • vscode: Version: 1.90.2 (Universal)

Is this a regression?

I only tested v0.222.0

Debug Logs

Will do so if requested, I don't think it adds value here

[shreyas-goenka]

Thanks for reporting @janstoecklin-mb! Indeed lock should be optional in the JSON schema. I have opened a PR to make it so.