Define all-purpose job clusters

[ryanhubbell]

Is there a way to define all-purpose job clusters? Looking at default_python_job, it seems that the clusters are defined per job. I don't see any way to reference the cluster from another job (at least using ${resources.jobs.job-name.job_clusters} doesn't work. Just using the cluster key doesn't work either.

[munish7771]

@ryanhubbell were you able to find a workaround for this? If so, could you please share.

[ryanhubbell]

I was not able to do it using Databricks Asset Bundles. I instead defined it via Terraform. In fact, I'm moving all of our stuff over to just using Terraform. It supports a lot more than DAB does, and it doesn't use yaml (not my favorite).

[andrewnester]

It's possible to create all purpose clusters starting form 0.229.0 version. Here's an example of how to achieve that

https://github.com/databricks/bundle-examples/tree/main/knowledge_base/development_cluster