[pbleser-oc]

Description

The OpenCloud side of the revision cleanup: old revisions (versions) of files are deleted according to a configurable cleanup strategy, for the decomposed, decomposeds3 and posix storage drivers. The cleanup itself is in reva, opencloud-eu/reva#895; this PR configures it and adds a command to run it on demand.

Caution

Draft until opencloud-eu/reva#895 is merged. The first commit temporarily points github.com/opencloud-eu/reva/v2 at the head of #895 in its fork, with a replace in go.mod, and re-vendors, so that this PR builds and CI runs. Once #895 is merged, that commit is replaced by a regular reva bump.

DO NOT MERGE BEFORE opencloud-eu/reva#895

Configuration

A revisions_cleanup section of the storage-users configuration, passed to all three drivers, in the storage provider and the data provider:

environment variable	default	meaning
OC_REVISIONS_CLEANUP_STRATEGY	disabled	which old revisions to delete whenever a file gets a new revision, e.g. auto, 30d, 10, 5, 30d or 7d, auto, 90d
OC_REVISIONS_CLEANUP_ASYNC	true	run the cleanup in background workers, so it doesn't slow down the upload, restore or change on disk that created the revision
OC_REVISIONS_CLEANUP_ASYNC_WORKERS	0 (half the CPUs, at least 2)	number of background workers
OC_REVISIONS_CLEANUP_ASYNC_QUEUE_SIZE	0 (64 per worker)	number of files that can wait for a worker
OC_REVISIONS_CLEANUP_ASYNC_QUEUE_FULL	block	block waits for room, drop skips the cleanup of that file until its next revision

The default keeps all revisions, as before. An invalid strategy makes the storage-users service fail at startup.

opencloud storage-users revisions cleanup

The service only cleans up a file when it gets a new revision. The command covers the files that aren't changed anymore, and applies a different strategy once, e.g. after the strategy was changed or to free storage:

• which files: --space (repeatable, with --path for a file or folder within the space), --all, or --fs-path (a file or folder by its path on disk, posix only);
• how: --strategy and the settings of the background workers (--async, --workers, --queue-size, --queue-full) default to the configured ones;
• output: a line for every file whose revisions were deleted, that failed or that was skipped, with its path within its space, and a summary; --verbose adds every revision with the decision and its reason; --json prints JSON objects, one per line;
• --dry-run deletes nothing;
• --timeout stops gracefully, as does ctrl-c; --shuffle walks the spaces and folders in a random order, so that repeated runs, e.g. a nightly cron job, eventually reach all files;
• exit codes: 0 when done, 1 if the cleanup or any file failed, 2 if the timeout stopped it before all files were cleaned up.

Short flags: -a, -p, -s, -n, -v, -j, -t. The command uses the storage driver directly, in its own process, like uploads sessions: posix neither scans nor watches the file system, and no driver joins the postprocessing consumer group.

opencloud storage-users revisions cleanup --all --strategy "auto, 90d" --dry-run --verbose
opencloud storage-users revisions cleanup --space <space id> --path /Documents --strategy 10
opencloud storage-users revisions cleanup --all --timeout 1h --shuffle

The storage-users README documents the configuration, the strategies, the command and its output, with examples. The admin documentation in opencloud-eu/docs follows separately.

Commits

1. chore: temporarily use reva from opencloud-eu/reva#895 (to be replaced by a reva bump)
2. feat: 1419: configure the revision cleanup of the storage-users drivers
3. feat: 1419: add a command to clean up old revisions on demand
4. test: 1419: acceptance tests for the revision cleanup

Every commit builds from vendor/ and passes the storage-users tests.

Related Issue

• Fixes 🧹 Clean-Up File Versions #1419
• needs feat: clean up old revisions according to a cleanup strategy reva#895
• follow-ups: Per-space revision cleanup strategies #3733 (strategies per space or file type), Unify the deletion of single revisions in decomposedfs #3734 (unify the deletion of single revisions in reva), Revision cleanup: should the age of a revision be the age of its content, or the time it was replaced? #3735 (the age of a revision: content age vs. replacement age)

Motivation and Context

Every change to a file keeps the previous content as a revision, forever. Without a cleanup, revisions accumulate and use up storage, and the only way to get rid of them was to delete the files or to disable versioning, which keeps the revisions that exist. #1419 asks for a configurable cleanup that runs automatically, without slowing down writes, and that can also be run on demand for spaces, folders or files.

How Has This Been Tested?

• unit tests: the configuration reaching all three drivers (revaconfig), and the command: flag validation, the strategy and worker defaults, text and JSON output, the exit codes (command)
• acceptance tests: a new feature, tests/acceptance/features/cliCommands/cleanupFileVersions.feature, with 10 scenarios against a running server, checking the remaining versions through the WebDAV versions API: all spaces, a dry run, a folder within the personal space, a project space, the JSON output, no strategy while the configured one is disabled (exit code 1), a timeout (exit code 2), a strategy without spaces after its commas, a folder by its path on disk (posix only), and the configured strategy cleaning up on every new version. Run locally on posix (10/10) and decomposed (9/9, the posix-only scenario is skipped).
• end to end: a scripted run against a single-binary server with posix: files with revisions created over WebDAV, then the command (dry run, a folder, JSON output, an invalid scope, a timeout, all spaces) next to the running server, checking the revisions on disk, the versions API and the file contents.
• the reva side is tested in feat: clean up old revisions according to a cleanup strategy reva#895.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Technical debt
• Tests only (no source changes)

Checklist:

• Code changes
• Unit tests added
• Acceptance tests added
• Documentation added

[codacy-production]

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 130 complexity

Metric	Results
Complexity	130

View in Codacy

🟢 Coverage 78.87% diff coverage

Metric	Results
Coverage variation	Report missing for 5d745311
Diff coverage	✅ 78.87% diff coverage

View coverage diff in Codacy

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (5d74531)	Report Missing	Report Missing	Report Missing
Head commit (0f568a5)	93066	25768	27.69%

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#3778)	426	336	78.87%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

¹ Codacy didn't receive coverage data for the commit, or there was an error processing the received data. Check your integration for errors and validate that your coverage setup is correct.

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}