workspace files are not removed when renamed or moved

[smokedlinq]

Describe the issue

The workspace files content does not remove old files that are either excluded in the sync configuration or when files are renamed/moved to a different directory. This seems similar to #822

Configuration

# yaml-language-server: $schema=bundle_config_schema.json
bundle:
  name: REDACTED

variables:
  service_principal_name:
    description: The name of the service principal to use for the job.
  catalog_name:
    description: The name of the catalog to use for the job.
  root_path:
    description: The root path to use for the workspace.
    default: /Shared
  pull_request_number:
    description: The number of the pull request to use for the job.
    default: "0"

sync:
  exclude:
    - bundle_config_schema.json

workspace:
  auth_type: azure-cli
  root_path: ${var.root_path}

resources:
  jobs:
    REDACTED:
      name: REDACTED
      job_clusters:
        - job_cluster_key: default
          new_cluster:
            spark_version: 12.2.x-scala2.12
            node_type_id: Standard_F4s_v2
            num_workers: 1
      tasks:
        - task_key: REDACTED
          job_cluster_key: default
          notebook_task:
            notebook_path: ${var.root_path}/files/REDACTED
            base_parameters:
              catalog_name: ${var.catalog_name}
      run_as:
        service_principal_name: ${var.service_principal_name}

targets:
  pull-request:
    variables:
      catalog_name: sandbox
      root_path: /Shared/REDACTED/pull_requests/${var.pull_request_number}
    resources:
      jobs:
        REDACTED:
          name: REDACTED-pr${var.pull_request_number}-REDACTED

Steps to reproduce the behavior

Please list the steps required to reproduce the issue, for example:

1. Run databricks bundle deploy ...
2. Look at the workspace files location to observe file content
3. Rename a file that is in the files content
4. Run databricks bundle deploy ...
5. Observe the workspace files location to verify the old file no longer exists

Expected Behavior

File content mirrors the source when deployed

Actual Behavior

Deleted, moved, or renamed files are not removed from the workspace files folder

OS and CLI version

Databricks CLI v0.208.1

Is this a regression?

Nope, we are just getting started using bundles

Debug Logs

time=2023-11-01T18:01:17.912Z level=INFO source="root.go 55}" msg=start version=0.208.2 args="databricks, bundle, deploy, --log-level, DEBUG"
time=2023-11-01T18:01:17.913Z level=DEBUG source="bundle.go 78}" msg="Loading bundle configuration from: /home/runner/work/REDACTED/REDACTED/notebooks/bundle.yml"
time=2023-11-01T18:01:17.914Z level=DEBUG source="mutator.go 22}" msg=Apply mutator=seq
time=2023-11-01T18:01:17.914Z level=DEBUG source="mutator.go 22}" msg=Apply mutator=seq mutator=scripts.preinit
time=2023-11-01T18:01:17.915Z level=DEBUG source="scripts.go 37}" msg="No script defined for preinit, skipping" mutator=seq mutator=scripts.preinit
time=2023-11-01T18:01:17.915Z level=DEBUG source="mutator.go 22}" msg=Apply mutator=seq mutator=ProcessRootIncludes
time=2023-11-01T18:01:17.915Z level=DEBUG source="mutator.go 22}" msg=Apply mutator=seq mutator=ProcessRootIncludes mutator=seq
time=2023-11-01T18:01:17.916Z level=DEBUG source="mutator.go 22}" msg=Apply mutator=seq mutator=DefineDefaultTarget(default)
time=2023-11-01T18:01:17.916Z level=DEBUG source="mutator.go 22}" msg=Apply mutator=seq mutator=LoadGitDetails
time=2023-11-01T18:01:17.916Z level=DEBUG source="mutator.go 22}" msg=Apply mutator=SelectTarget(pull-request)
time=2023-11-01T18:01:17.917Z level=DEBUG source="mutator.go 22}" msg=Apply mutator=seq
time=2023-11-01T18:01:17.917Z level=DEBUG source="mutator.go 22}" msg=Apply mutator=seq mutator=initialize
time=2023-11-01T18:01:17.917Z level=INFO source="phase.go 30}" msg="Phase: initialize" mutator=seq mutator=initialize
time=2023-11-01T18:01:17.918Z level=DEBUG source="mutator.go 22}" msg=Apply mutator=seq mutator=initialize mutator=seq
time=2023-11-01T18:01:17.918Z level=DEBUG source="mutator.go 22}" msg=Apply mutator=seq mutator=initialize mutator=seq mutator=PopulateCurrentUser
time=2023-11-01T18:01:17.919Z level=INFO source="auth_default.go 49}" msg="Ignoring pat auth, because azure-cli is preferred" mutator=seq mutator=initialize mutator=seq mutator=PopulateCurrentUser sdk=true
time=2023-11-01T18:01:17.919Z level=INFO source="auth_default.go 49}" msg="Ignoring basic auth, because azure-cli is preferred" mutator=seq mutator=initialize mutator=seq mutator=PopulateCurrentUser sdk=true
time=2023-11-01T18:01:17.919Z level=INFO source="auth_default.go 49}" msg="Ignoring oauth-m2m auth, because azure-cli is preferred" mutator=seq mutator=initialize mutator=seq mutator=PopulateCurrentUser sdk=true
time=2023-11-01T18:01:17.920Z level=INFO source="auth_default.go 49}" msg="Ignoring databricks-cli auth, because azure-cli is preferred" mutator=seq mutator=initialize mutator=seq mutator=PopulateCurrentUser sdk=true
time=2023-11-01T18:01:17.920Z level=INFO source="auth_default.go 49}" msg="Ignoring metadata-service auth, because azure-cli is preferred" mutator=seq mutator=initialize mutator=seq mutator=PopulateCurrentUser sdk=true
time=2023-11-01T18:01:17.920Z level=INFO source="auth_default.go 49}" msg="Ignoring azure-msi auth, because azure-cli is preferred" mutator=seq mutator=initialize mutator=seq mutator=PopulateCurrentUser sdk=true
time=2023-11-01T18:01:17.921Z level=INFO source="auth_default.go 49}" msg="Ignoring azure-client-secret auth, because azure-cli is preferred" mutator=seq mutator=initialize mutator=seq mutator=PopulateCurrentUser sdk=true
time=2023-11-01T18:01:18.533Z level=INFO source="auth_azure_cli.go 129}" msg="Refreshed OAuth token for REDACTED from Azure CLI, which expires on 2023-11-01 19:01:17.000000" sdk=true
time=2023-11-01T18:01:19.005Z level=INFO source="auth_azure_cli.go 129}" msg="Refreshed OAuth token for https://management.core.windows.net/ from Azure CLI, which expires on 2023-11-01 19:01:15.000000" sdk=true
time=2023-11-01T18:01:19.005Z level=INFO source="auth_azure_cli.go 87}" msg="Using Azure CLI authentication with AAD tokens" mutator=seq mutator=initialize mutator=seq mutator=PopulateCurrentUser sdk=true
time=2023-11-01T18:01:19.266Z level=DEBUG source="client.go 383}" msg="GET /api/2.0/preview/scim/v2/Me\n< HTTP/2.0 200 OK\n< {\n<   \"active\": true,\n<   \"displayName\": \"github.com/REDACTED/REDACTED\",\n<   \"emails\": [\n<     {\n<       \"primary\": true,\n<       \"type\": \"work\",\n<       \"value\": \"REDACTED\"\n<     }\n<   ],\n<   \"externalId\": \"3d11cbc4-eb54-433c-8d79-6b00f4f1b4e6\",\n<   \"groups\": null,\n<   \"id\": \"3795757075996507\",\n<   \"name\": {\n<     \"givenName\": \"github.com/REDACTED/REDACTED\"\n<   },\n<   \"schemas\": [\n<     \"urn:ietf:params:scim:schemas:core:2.0:User\",\n<     \"urn:ietf:params:scim:schemas:extension:workspace:2.0:User\"\n<   ],\n<   \"userName\": \"REDACTED\"\n< }" mutator=seq mutator=initialize mutator=seq mutator=PopulateCurrentUser sdk=true
time=2023-11-01T18:01:19.267Z level=DEBUG source="mutator.go 22}" msg=Apply mutator=seq mutator=initialize mutator=seq mutator=SetRunAs
time=2023-11-01T18:01:19.267Z level=DEBUG source="mutator.go 22}" msg=Apply mutator=seq mutator=initialize mutator=seq mutator=DefineDefaultWorkspaceRoot
time=2023-11-01T18:01:19.267Z level=DEBUG source="mutator.go 22}" msg=Apply mutator=seq mutator=initialize mutator=seq mutator=ExpandWorkspaceRoot
time=2023-11-01T18:01:19.267Z level=DEBUG source="mutator.go 22}" msg=Apply mutator=seq mutator=initialize mutator=seq mutator=DefaultWorkspacePaths
time=2023-11-01T18:01:19.267Z level=DEBUG source="mutator.go 22}" msg=Apply mutator=seq mutator=initialize mutator=seq mutator=SetVariables
time=2023-11-01T18:01:19.267Z level=DEBUG source="mutator.go 22}" msg=Apply mutator=seq mutator=initialize mutator=seq mutator=Interpolate
time=2023-11-01T18:01:19.268Z level=DEBUG source="mutator.go 22}" msg=Apply mutator=seq mutator=initialize mutator=seq mutator=OverrideCompute
time=2023-11-01T18:01:19.268Z level=DEBUG source="mutator.go 22}" msg=Apply mutator=seq mutator=initialize mutator=seq mutator=ProcessTargetMode
time=2023-11-01T18:01:19.268Z level=DEBUG source="mutator.go 22}" msg=Apply mutator=seq mutator=initialize mutator=seq mutator=ExpandPipelineGlobPaths
time=2023-11-01T18:01:19.268Z level=DEBUG source="mutator.go 22}" msg=Apply mutator=seq mutator=initialize mutator=seq mutator=TranslatePaths
time=2023-11-01T18:01:19.268Z level=DEBUG source="mutator.go 22}" msg=Apply mutator=seq mutator=initialize mutator=seq mutator=PythonWrapperWarning
time=2023-11-01T18:01:19.268Z level=DEBUG source="mutator.go 22}" msg=Apply mutator=seq mutator=initialize mutator=seq mutator=terraform.Initialize
Uploaded bundle files at /Shared/REDACTED/pull_requests/7/files!

time=2023-11-01T18:01:21.408Z level=DEBUG source="mutator.go 22}" msg=Apply mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=Interpolate
time=2023-11-01T18:01:21.408Z level=DEBUG source="mutator.go 22}" msg=Apply mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=terraform.Write
time=2023-11-01T18:01:21.416Z level=DEBUG source="mutator.go 22}" msg=Apply mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=terraform:state-pull
time=2023-11-01T18:01:21.416Z level=INFO source="state_pull.go 34}" msg="Opening remote state file" mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=terraform:state-pull
time=2023-11-01T18:01:21.491Z level=DEBUG source="client.go 383}" msg="GET /api/2.0/workspace/get-status?path=/Shared/REDACTED/pull_requests/7/state/terraform.tfstate\n< HTTP/2.0 200 OK\n< {\n<   \"created_at\": 1698848704786,\n<   \"modified_at\": 1698861487182,\n<   \"object_id\": 1073706241443050,\n<   \"object_type\": \"FILE\",\n<   \"path\": \"/Shared/REDACTED/pull_requests/7/state/terraform.tfstate\"\n< }" mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=terraform:state-pull sdk=true
time=2023-11-01T18:01:21.572Z level=DEBUG source="client.go 383}" msg="GET /api/2.0/workspace/export?direct_download=true&path=/Shared/REDACTED/pull_requests/7/state/terraform.tfstate\n< HTTP/2.0 200 OK\n< [non-JSON document of 15 bytes]. <io.ReadCloser>" mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=terraform:state-pull sdk=true
time=2023-11-01T18:01:21.573Z level=INFO source="state_pull.go 62}" msg="Writing remote state file to local cache directory" mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=terraform:state-pull
time=2023-11-01T18:01:21.573Z level=DEBUG source="mutator.go 22}" msg=Apply mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=terraform.Apply
Starting resource deployment
Resource deployment completed!
time=2023-11-01T18:01:26.680Z level=DEBUG source="mutator.go 22}" msg=Apply mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=terraform:state-push
time=2023-11-01T18:01:26.680Z level=INFO source="state_push.go 38}" msg="Writing local state file to remote state directory" mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=terraform:state-push
time=2023-11-01T18:01:26.918Z level=DEBUG source="client.go 383}" msg="POST /api/2.0/workspace-files/import-file/Shared/REDACTED/pull_requests/7/state/terraform.tfstate?overwrite=true\n> {\n>   \"check_results\": null,\n>   \"lineage\": \"46474c39-1b0c-3bb5-c11f-3b47bc80a066\",\n>   \"outputs\": {},\n>   \"resources\": [\n>     {\n>       \"instances\": [\n>         {\n>           \"attributes\": {\n>             \"always_running\": false,\n>             \"compute\": null,\n>             \"continuous\": null,\n>             \"control_run_state\": false,\n>             \"dbt_task\": null,\n>             \"email_notifications\": [\n>               {\n>                 \"alert_on_last_attempt\": false,\n>                 \"no_alert_for_skipped_runs\": false,\n>                 \"on_duration_warning_threshold_exceeded\": null,\n>                 \"on_failure\": null,\n>                 \"on_start\": null,\n>                 \"on_success\": null\n>               }\n>             ],\n>             \"existing_cluster_id\": null,\n>             \"format\": \"MULTI_TASK\",\n>             \"git_source\": null,\n>             \"health\": null,\n>             \"id\": \"941231850420558\",\n>             \"job_cluster\": [\n>               {\n>                 \"job_cluster_key\": \"default\",\n>                 \"new_cluster\": [\n>                   {\n>                     \"apply_policy_default_values\": false,\n>                     \"autoscale\": null,\n>                     \"autotermination_minutes\": 0,\n>                     \"aws_attributes\": null,\n>                     \"azure_attributes\": [\n>                       {\n>                         \"availability\": \"ON_DEMAND_AZURE\",\n>                         \"first_on_demand\": 0,\n>                         \"spot_bid_max_price\": 0\n>                       }\n>                     ],\n>                     \"cluster_id\": \"\",\n>                     \"cluster_log_conf\": null,\n>                     \"cluster_mount_info\": null,\n>                     \"cluster_name\": \"\",\n>                     \"custom_tags\": null,\n>                     \"data_security_mode\": \"SINGLE_USER\",\n>                     \"docker_image\": null,\n>                     \"driver_instance_pool_id\": \"\",\n>                     \"driver_node_type_id\": \"\",\n>                     \"enable_elastic_disk\": true,\n>                     \"enable_local_disk_encryption\": false,\n>                     \"gcp_attributes\": null,\n>                     \"idempotency_token\": \"\",\n>                     \"init_scripts\": null,\n>                     \"instance_pool_id\": \"\",\n>                     \"node_type_id\": \"Standard_F4s_v2\",\n>                     \"num_workers\": 1,\n>                     \"policy_id\": \"\",\n>                     \"runtime_engine\": \"\",\n>                     \"single_user_name\": \"\",\n>                     \"spark_conf\": null,\n>                     \"spark_env_vars\": null,\n>                     \"spark_version\": \"12.2.x-scala2.12\",\n>                     \"ssh_public_keys\": null,\n>                     \"workload_type\": null\n>                   }\n>                 ]\n>               }\n>             ],\n>             \"library\": null,\n>             \"max_concurrent_runs\": 1,\n>             \"max_retries\": 0,\n>             \"min_retry_interval_millis\": 0,\n>             \"name\": \"REDACTED-pr7-REDACTED\",\n>             \"new_cluster\": null,\n>             \"notebook_task\": null,\n>             \"notification_settings\": null,\n>             \"parameter\": null,\n>             \"pipeline_task\": null,\n>             \"python_wheel_task\": null,\n>             \"queue\": null,\n>             \"retry_on_timeout\": false,\n>             \"run_as\": [\n>               {\n>                 \"service_principal_name\": \"REDACTED\",\n>                 \"user_name\": \"\"\n>               }\n>             ],\n>             \"run_job_task\": null,\n>             \"schedule\": null,\n>             \"spark_jar_task\": null,\n>             \"spark_python_task\": null,\n>             \"spark_submit_task\": null,\n>             \"tags\": null,\n>             \"task\": [\n>               {\n>                 \"compute_key\": \"\",\n>                 \"condition_task\": null,\n>                 \"dbt_task\": null,\n>                 \"depends_on\": null,\n>                 \"description\": \"\",\n>                 \"email_notifications\": [\n>                   {\n>                     \"alert_on_last_attempt\": false,\n>                     \"no_alert_for_skipped_runs\": false,\n>                     \"on_duration_warning_threshold_exceeded\": null,\n>                     \"on_failure\": null,\n>                     \"on_start\": null,\n>                     \"on_success\": null\n>                   }\n>                 ],\n>                 \"existing_cluster_id\": \"\",\n>                 \"health\": null,\n>                 \"job_cluster_key\": \"default\",\n>                 \"library\": null,\n>                 \"max_retries\": 0,\n>                 \"min_retry_interval_millis\": 0,\n>                 \"new_cluster\": null,\n>                 \"notebook_task\": [\n>                   {\n>                     \"base_parameters\": {\n>                       \"catalog_name\": \"sandbox\"\n>                     },\n>                     \"notebook_path\": \"/Shared/REDACTED/pull_requests/7/files/REDACTED\",\n>                     \"source\": \"WORKSPACE\"\n>                   }\n>                 ],\n>                 \"notification_settings\": null,\n>                 \"pipeline_task\": null,\n>                 \"python_wheel_task\": null,\n>                 \"retry_on_timeout\": false,\n>                 \"run_if\": \"ALL_SUCCESS\",\n>                 \"run_job_task\": null,\n>                 \"spark_jar_task\": null,\n>                 \"spark_python_task\": null,\n>                 \"spark_submit_task\": null,\n>                 \"sql_task\": null,\n>                 \"task_key\": \"REDACTED\",\n>                 \"timeout_seconds\": 0\n>               }\n>             ],\n>             \"timeout_seconds\": 0,\n>             \"timeouts\": null,\n>             \"trigger\": null,\n>             \"url\": \"https://adb-541145334391411.11.azuredatabricks.net/#job/941231850420558\",\n>             \"webhook_notifications\": [\n>               {\n>                 \"on_duration_warning_threshold_exceeded\": null,\n>                 \"on_failure\": null,\n>                 \"on_start\": null,\n>                 \"on_success\": null\n>               }\n>             ]\n>           },\n>           \"private\": \"eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoxODAwMDAwMDAwMDAwLCJ1cGRhdGUi... (52 more bytes)\",\n>           \"schema_version\": 2,\n>           \"sensitive_attributes\": null\n>         }\n>       ],\n>       \"mode\": \"managed\",\n>       \"name\": \"REDACTED\",\n>       \"provider\": \"provider[\\\"registry.terraform.io/databricks/databricks\\\"]\",\n>       \"type\": \"databricks_job\"\n>     }\n>   ],\n>   \"serial\": 19,\n>   \"terraform_version\": \"1.5.5\",\n>   \"version\": 4\n> }\n< HTTP/2.0 200 OK\n< [non-JSON document of 15 bytes]. <io.ReadCloser>" mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=terraform:state-push sdk=true
time=2023-11-01T18:01:26.918Z level=DEBUG source="mutator.go 22}" msg=Apply mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=lock:release
time=2023-11-01T18:01:26.918Z level=INFO source="release.go 45}" msg="Releasing deployment lock" mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=lock:release
time=2023-11-01T18:01:26.996Z level=DEBUG source="client.go 383}" msg="GET /api/2.0/workspace/get-status?path=/Shared/REDACTED/pull_requests/7/state/deploy.lock\n< HTTP/2.0 200 OK\n< {\n<   \"created_at\": 1698861680424,\n<   \"modified_at\": 1698861680424,\n<   \"object_id\": 1417620599784186,\n<   \"object_type\": \"FILE\",\n<   \"path\": \"/Shared/REDACTED/pull_requests/7/state/deploy.lock\"\n< }" mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=lock:release sdk=true
time=2023-11-01T18:01:27.058Z level=DEBUG source="client.go 383}" msg="GET /api/2.0/workspace/get-status?path=/Shared/REDACTED/pull_requests/7/state/deploy.lock\n< HTTP/2.0 200 OK\n< {\n<   \"created_at\": 1698861680424,\n<   \"modified_at\": 1698861680424,\n<   \"object_id\": 1417620599784186,\n<   \"object_type\": \"FILE\",\n<   \"path\": \"/Shared/REDACTED/pull_requests/7/state/deploy.lock\"\n< }" mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=lock:release sdk=true
time=2023-11-01T18:01:27.153Z level=DEBUG source="client.go 383}" msg="GET /api/2.0/workspace/export?direct_download=true&path=/Shared/REDACTED/pull_requests/7/state/deploy.lock\n< HTTP/2.0 200 OK\n< [non-JSON document of 15 bytes]. <io.ReadCloser>" mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=lock:release sdk=true
time=2023-11-01T18:01:27.223Z level=DEBUG source="client.go 383}" msg="POST /api/2.0/workspace/delete\n> {\n>   \"path\": \"/Shared/REDACTED/pull_requests/7/state/deploy.lock\"\n> }\n< HTTP/2.0 200 OK\n< {}" mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=lock:release sdk=true
time=2023-11-01T18:01:27.223Z level=DEBUG source="mutator.go 22}" msg=Apply mutator=seq mutator=deploy mutator=seq mutator=seq mutator=scripts.postdeploy
time=2023-11-01T18:01:27.223Z level=DEBUG source="scripts.go 37}" msg="No script defined for postdeploy, skipping" mutator=seq mutator=deploy mutator=seq mutator=seq mutator=scripts.postdeploy
time=2023-11-01T18:01:27.223Z level=INFO source="root.go 111}" msg="completed execution" exit_code=0

[andrewnester]

@smokedlinq Thanks for providing a lot of useful details! A few things to clarify though

The workspace files content does not remove old files that are either excluded in the sync configuration or when files are renamed/moved to a different directory

Does the issue only happen if you move the file to directory outside of your bundle or to the folder which is in sync.exclude? What if you do just a simple rename of the file like from file.ext to file1.ext?

[smokedlinq]

I've noticed it not removing them when we don't use sync (include or exclude), renaming the file, moving the file, or deleting the file. We tried to add the file to sync exclude, but it didn't delete the file.

[shreyas-goenka]

@smokedlinq Thanks for reporting this issue!

From where did you rename / move the file? If you did so using the Databricks UI then we explictly do not track those changes.

If you rename a file locally in your bundle, then older version of those files would should be deleted.

Howerver, if multiple users use the same bundle root_path, then this is indeed an issue and the previous files will not be deleted. This is becaulse we keep track of uploaded files individually for each user.

Can you expand in which context are these files being orphaned and is this a cause for concern for you?

[smokedlinq]

Moving them in the source with the dab config. Deploy the notebooks with dab, rename a notebook file or move it to another folder, deploy the dab again, then the original file isn't being removed from the workspace in its original name/location.

[shreyas-goenka]

@smokedlinq Just to clarify, you are renaming the notebook on your local file system and then doing a deploy

And in that case the previously uploaded copy of the notebook is not being deleted in the databricks workspace filesystem?

[smokedlinq]

Correct

[andrewnester]

@smokedlinq do you execute databricks bundle deploy command from some CI / VM / etc. or locally on your machine as well?

[mosch911]

@smokedlinq Thanks for reporting this issue!

From where did you rename / move the file? If you did so using the Databricks UI then we explictly do not track those changes.

If you rename a file locally in your bundle, then older version of those files would should be deleted.

Howerver, if multiple users use the same bundle root_path, then this is indeed an issue and the previous files will not be deleted. This is becaulse we keep track of uploaded files individually for each user.

Can you expand in which context are these files being orphaned and is this a cause for concern for you?

where is this information stored about which user uploaded which subset of files to which shared folder? Can we somehow erase this information?

[andrewnester]

@mosch911 it's not stored in Databricks filesystem and only exists locally in .databricks folder within your bundle