Skip to content

27/6/24 Error for deploying asset bundles with a singlenode. It states that numworkers is not set to 0. #1549

@clintdk

Description

@clintdk

Describe the issue

Since 27/6/24 I receive the following error when trying to deploy my asset bundles:
Error: cannot create job: NumWorkers could be 0 only for SingleNode clusters.

My asset bundle configuration hasn't changed over time but suddenly breaks. I saw that there were already multiple fixes being done recently surrounding single-node clusters and num of workers #1472 and a fix. But it doesn't seem to solve my specific problem.

Configuration

Please provide a minimal reproducible configuration for the issue

...
single-node-job-cluster-prd: &single-node-job-cluster-prd
  spark_version: 14.3.x-scala2.12
  policy_id: anonymized
  num_workers: 0
  spark_conf:
    spark.databricks.cluster.profile: singleNode
    spark.master: local[*]

...
targets:
  prd-configfiles:
    workspace:
      profile: anonymized
      host: anonymized
      root_path: anonymized
      artifact_path: anonymized
      file_path: anonymized
      state_path: anonymized
    resources:
      jobs:
        prd-configfiles:
          name: mlt4-configfiles
          max_concurrent_runs: 1
          tags:
            use-case: mlt4-configfiles
            stage: prd-configfiles
            job-type: write-configfiles
          permissions: *permissions-prd
          job_clusters:
            - job_cluster_key: workflow-cluster
              new_cluster:
                <<: *single-node-job-cluster-prd
          tasks:
            - task_key: write-update-configfiles
              job_cluster_key: workflow-cluster
              python_wheel_task:
                package_name: anonymized
                entry_point: anonymized
              libraries:
                - whl: ./dist/*.whl

Steps to reproduce the behavior

Please list the steps required to reproduce the issue, for example:

  1. Create databricks.yaml
  2. run databricks bundle deploy -t prd-configfiles --profile something

Expected Behavior

Deployment of job to Databricks

Actual Behavior

No deployment:

Uploading  ....
Uploading bundle files to...
Deploying resources...
Updating deployment state...
Error: terraform apply: exit status 1

Error: cannot create job: NumWorkers could be 0 only for SingleNode clusters. See https://docs.databricks.com/clusters/single-node.html for more details

  with databricks_job.prd-configfiles,
  on bundle.tf.json line 61, in resource.databricks_job.prd-configfiles:
  61:       }

OS and CLI version

WSL Ubuntu 22.04.3 and Windows 10

Is this a regression?

It even worked in this version. I think it's a problem with databricks itself, not the cli. Not sure though.

Debug Logs

12:44:22 ERROR Error: terraform apply: exit status 1

Error: cannot create job: NumWorkers could be 0 only for SingleNode clusters. See https://docs.databricks.com/clusters/single-node.html for more details

  with databricks_job.prd-configfiles,
  on bundle.tf.json line 61, in resource.databricks_job.prd-configfiles:
  61:       }

 pid=5110 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred
12:44:22 ERROR Error: terraform apply: exit status 1

Error: cannot create job: NumWorkers could be 0 only for SingleNode clusters. See https://docs.databricks.com/clusters/single-node.html for more details

  with databricks_job.prd-configfiles,
  on bundle.tf.json line 61, in resource.databricks_job.prd-configfiles:
  61:       }

 pid=5110 mutator=seq mutator=deploy mutator=seq mutator=seq
12:44:22 ERROR Error: terraform apply: exit status 1

Error: cannot create job: NumWorkers could be 0 only for SingleNode clusters. See https://docs.databricks.com/clusters/single-node.html for more details

  with databricks_job.prd-configfiles,
  on bundle.tf.json line 61, in resource.databricks_job.prd-configfiles:
  61:       }

 pid=5110 mutator=seq mutator=deploy mutator=seq
12:44:22 ERROR Error: terraform apply: exit status 1

Error: cannot create job: NumWorkers could be 0 only for SingleNode clusters. See https://docs.databricks.com/clusters/single-node.html for more details

  with databricks_job.prd-configfiles,
  on bundle.tf.json line 61, in resource.databricks_job.prd-configfiles:
  61:       }

 pid=5110 mutator=seq mutator=deploy
12:44:22 ERROR Error: terraform apply: exit status 1

Error: cannot create job: NumWorkers could be 0 only for SingleNode clusters. See https://docs.databricks.com/clusters/single-node.html for more details

  with databricks_job.prd-configfiles,
  on bundle.tf.json line 61, in resource.databricks_job.prd-configfiles:
  61:       }

 pid=5110 mutator=seq
Error: terraform apply: exit status 1

Error: cannot create job: NumWorkers could be 0 only for SingleNode clusters. See https://docs.databricks.com/clusters/single-node.html for more details

  with databricks_job.prd-configfiles,
  on bundle.tf.json line 61, in resource.databricks_job.prd-configfiles:
  61:       }


12:44:22 ERROR failed execution pid=5110 exit_code=1 error="terraform apply: exit status 1\n\nError: cannot create job: NumWorkers could be 0 only for SingleNode clusters. See https://docs.databricks.com/clusters/single-node.html for more details\n\n  with databricks_job.prd-configfiles,\n  on bundle.tf.json line 61, in resource.databricks_job.prd-configfiles:\n  61:       }\n\n"

Metadata

Metadata

Assignees

No one assigned

    Labels

    BugSomething isn't workingDABsDABs related issues

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions