[LuciferYang]

Adds lance_ray.cleanup_old_versions and lance_ray.explain_cleanup_old_versions. The driver plans the cleanup with LanceDataset.plan_cleanup_old_versions from lance-format/lance#10235, runs each sweep task on an idle Ray worker, retries a failed task up to max_task_retries times, and finishes the plan. Lance decides which files are removed; lance-ray only schedules the ranges. The driver logs a line as each range finishes, with the directory, the range and the running number of files and bytes removed.

Closes #98.

• delete_rate_limit covers the whole cleanup and is split evenly between the workers, with at least one request per second each. delete_concurrency applies per worker.
• max_tasks defaults to four tasks per worker.
• Each worker opens the dataset once and reuses it for its tasks.
• The dataset can be given by URI or by namespace_impl and table_id.
• If a task still fails after its retries, a RuntimeError is raised. By then the plan has already removed the old versions, and later cleanups only remove the files the failed tasks left as unverified files.
• With a pylance that lacks plan_cleanup_old_versions, including the 12.0.0 pinned here, the cleanup runs on the driver and logs a warning.

_map_async_with_pool and _BatchWorker move from index.py to _dispatch.py, so index building and cleanup share them. The function now takes any list of items and an on_result(index, result) callback, called once per item with its final result. Index building behaves as before, except that its retry warnings now come from the lance_ray._dispatch logger instead of lance_ray.index.

Testing

The pinned pylance lacks the new API, so the tests that need it skip in CI. To cover the driver there, a fake plan with the real plan_cleanup_old_versions signature checks that every task is dispatched and its result passed to finish in task order, that a failed task is retried, that a task that still fails raises without finishing the plan, that the rate limit is split, that max_tasks reaches the plan, and that progress is logged. With a pylance wheel built from lance-format/lance#10235, the other tests compare against single-process explain_cleanup_old_versions and cleanup_old_versions, run through a directory namespace, and check what a failed task leaves. On the same wheel, a 30-version local table with overwrites, deletions, a BTREE index, a tag, and old and new orphans ended with the same files, stats, data and tagged version under single-process cleanup and under lance-ray with four workers (one task per directory on the local filesystem).

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.