Per-space revision cleanup strategies #3733

Context

With #1419, OpenCloud deletes old revisions (versions) of files according to a cleanup strategy:

• OC_REVISIONS_CLEANUP_STRATEGY configures one strategy for the whole instance, e.g. auto, 30d, 5, 30d or 7d, auto, 90d. It applies whenever a file gets a new revision, for the decomposed, decomposeds3 and posix storage drivers. The default, disabled, keeps all revisions.
• opencloud storage-users revisions cleanup cleans up whole spaces, folders or files on demand, optionally with a different strategy for that run (--strategy).

#1419 also mentions configuring the number of versions to keep per space. That was left out of the first implementation on purpose. This issue collects the options and the open design questions for a later implementation, as well as the related idea of strategies per file type (see below).

Out of scope: old revisions keep counting towards the quota of a space; that is intended and stays as it is.

Use cases

• Project spaces that keep more versions than personal spaces, or the other way around.
• A space with data that changes often (e.g. synced application folders, databases) that should keep only a few versions, while the rest of the instance keeps more.
• A space with important documents that should keep its versions longer than the instance-wide strategy.
• Few versions of large media files, disk images or database files, and many versions of office documents, independent of the space.

Options

1. A space attribute, set by an admin CLI

Store the strategy of a space as an attribute (metadata) on the space root, set with a command like:

opencloud storage-users revisions strategy set --space <id> --strategy "5, 90d"
opencloud storage-users revisions strategy unset --space <id>
opencloud storage-users revisions strategy list

The cleanup uses the strategy of the file's space if there is one, the instance-wide one otherwise.

• No change to the APIs or the web UI.
• The setting travels with the space, e.g. through backups, restores, or the files on disk with posix.
• Only admins (with access to the CLI) can set it.

2. A strategy per space type, in the configuration

E.g. OC_REVISIONS_CLEANUP_STRATEGY_PROJECT next to OC_REVISIONS_CLEANUP_STRATEGY (which would then apply to personal spaces and as the default).

• No setter, no new command, no permission questions.
• Probably covers the most common wish ("projects keep their versions longer").
• Can't target individual spaces.

3. A map of space ids to strategies in the configuration

• Simple, but needs a restart for every change, uses opaque space ids, and entries go stale when spaces are deleted.
• Not recommended.

4. A WebDAV property on the space root (PROPPATCH)

Store the strategy as a custom property of the space root, set with any WebDAV client.

• No new API at all.
• But: in decomposedfs, setting such properties only requires the permission to upload. Any editor of a shared space could set 0 and irrevocably delete all versions of the space. This would need a dedicated permission check for that property.
• Not recommended as it is.

5. The Graph API

A property of the drive, set with PATCH /graph/v1.0/drives/{id} like the quota or the description.

• The right long-term solution if space managers should set the strategy of their spaces themselves.
• Needs a change to the libre-graph-api schema, a permission model, and eventually support in the web UI.

6. No change: per-space cron jobs (possible today)

Configure the most lenient policy instance-wide, and run the on-demand cleanup with a stricter strategy for single spaces, e.g. from a cron job:

opencloud storage-users revisions cleanup --space <id> --strategy "3, 30d" --timeout 1h --shuffle

• Works today, only needs documentation.
• Only works for spaces that should keep less than the instance-wide strategy, and only applies when the job runs.

Open design questions

• Who owns the policy: admins or space managers? If only admins, a CLI (option 1) is enough. If space managers should decide for their spaces, the work is in the API and the permission model (option 5), and a CLI-only design would be a dead end. This decides what to build, so it should be answered first.
• Override, or within bounds? A per-space strategy that is more lenient than the instance-wide one can defeat what the admin wants to enforce, e.g. to limit the growth of the storage. Options: a plain override; only stricter strategies than the instance-wide one; or limits defined by the admin.
• Visibility. A strategy stored with a space is invisible unless there is a way to list them; a list command (or API) is needed from the start. With posix, the attribute can also be changed on disk, so an invalid value must not break the cleanup of the space: fall back to the instance-wide strategy and log it.
• Precedence for on-demand cleanups. --strategy of the command, then the strategy of the space, then the instance-wide one; to be documented.
• Demand. Options 2 and 6 cover the typical cases at almost no cost; it may be worth gathering feedback before building the full solution.

Notes on the implementation (option 1)

An estimate from the implementation of #1419:

• reva, roughly 300–400 lines including tests:
  • a new space attribute;
  • the strategy has to be resolved per file instead of once per cleanup service: read the attribute of the space root, cache the parsed strategy per distinct value, fall back to the instance-wide strategy;
  • getting, setting and listing the strategies of spaces in Decomposedfs (and its forwarding in the posix driver).
• The biggest part: today, the instance-wide strategy disabled means that the storage driver doesn't run a revision cleanup at all. With per-space strategies, the cleanup has to run whenever any space may have a strategy of its own, which changes the startup of the cleanup and its reva_revision_cleanup_enabled metric.
• On-demand cleanups currently use one strategy per run, and would need the same per-file resolution.
• opencloud, roughly 200–300 lines: the commands, their tests, the documentation in the README of the storage-users service, and acceptance tests.

Option 2 needs the same per-file resolution, but none of the setter, the command or the permission questions, so it is a much smaller step. Strategies per file type (see below) need it as well, so the per-file resolution should be built in a way that covers both.

Further idea: strategies per file type

Beyond spaces, the strategy could also depend on the type of the file, e.g. few versions of large media files, disk images or database files, and many versions of office documents.

• It needs the same per-file resolution of the strategy as per-space strategies, so both should be designed together, including their precedence (e.g. the file type rule within the strategy of the space).
• It would be configured as an ordered list of rules (e.g. by file extension or MIME type, first match wins), with the strategy of the space or the instance as the fallback.
• What "the type" is needs a decision: the extension is cheap but can be changed by renaming the file (also to avoid a strict rule), the MIME type stored with the file is more robust. A file's old revisions may also have had a different type than its current content.
• The size of a file may be a better criterion than its type for the most common goal, saving storage: e.g. keep fewer versions of files above a certain size.

Suggested plan

1. Document the per-space cron pattern (option 6).
2. Decide who owns the policy and whether per-space strategies may be more lenient than the instance-wide one.
3. Decide whether strategies per file type (or per file size) are wanted, and how they combine with the strategy of a space, so that the per-file resolution of the strategy is designed once for both.
4. Depending on that, implement option 2 as a first step, then option 1 (admins) or option 5 (space managers), and the rules per file type.