Skip to content

port the upload coordinator - #3839

Draft
butonic wants to merge 12 commits into
mainfrom
port-upload-coordinator
Draft

butonic wants to merge 12 commits into
mainfrom
port-upload-coordinator

Conversation

@butonic

@butonic butonic commented Oct 27, 2026

Copy link
Copy Markdown
Member

Switches opencloud to the reva upload coordinator (opencloud-eu/reva#922): uploads are initiated, received, postprocessed and committed by pkg/upload instead of the storage drivers.

  • reva is replaced with the coordinator branch. Drop the replace once port the upload coordinator reva#922 is merged and bump reva instead.
  • The ocm storage and data providers stage uploads in <storage root>/uploads, and the storage-users dataprovider gets the mount id.
  • storage-users passes its mount id to the drivers, and postprocessing sets the resource id on its result, so the coordinator ignores postprocessing results for other storages.
  • opencloud storage-users uploads sessions lists the sessions through the coordinator, since the drivers no longer keep them.
  • New --orphaned filter for uploads sessions: it lists the sessions whose node metadata can no longer be read, and --clean removes them and gives their quota back.

Held until opencloud-eu/reva#922 is ready to merge.

This PR was created with the help of AI (Claude Code).

LarsJurgensen and others added 8 commits October 27, 2026 15:29
Co-authored-by: Firas Frikha <firas.frikha@kiteworks.com>
Points reva at the upload coordinator branch, which moves uploads out of
the drivers into pkg/upload and removes the drivers' upload session
listing. Drop the replace once the reva branch is merged.

Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
Signed-off-by: Firas Frikha <firas.frikha@kiteworks.com>
Upload sessions whose node metadata can no longer be read can never
finish postprocessing, stay in processing state indefinitely and keep
consuming the quota of their space.

Wire up the Orphaned session filter so operators can list the affected
sessions and remove them with --clean:

    opencloud storage-users uploads sessions --orphaned
    opencloud storage-users uploads sessions --orphaned --clean

Signed-off-by: Julian Koberg <julian.koberg@kiteworks.com>
The driver no longer consumes postprocessing events: the upload
coordinator does, in the "dcfs" group, and the driver only consumes
DeleteRevision in "dcfs-revisions". The command still disables async
uploads so it does not join that group.

Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
…on PostprocessingFinished

The storage drops postprocessing events that belong to another storage,
which needs its mount id and the resource id on the event.
Postprocessing results are dropped when their storage id does not match
the driver's mount id. Only the decomposed driver was given a mount_id,
so under decomposeds3 there was nothing to compare against.

Also pass it in DecomposedNoEvents and DecomposedS3NoEvents.
The upload coordinator reads the mount id from the driver config to drop
postprocessing results that belong to another storage. The dataprovider
does not add it to the driver config the way the storageprovider does,
so without it the coordinator accepts every result for posix.

Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
@codacy-production

codacy-production Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 7 complexity

Metric Results
Complexity 7

View in Codacy

🟢 Coverage 36.73% diff coverage · +0.02% coverage variation

Metric Results
Coverage variation ✅ +0.02% coverage variation (-1.00%)
Diff coverage ✅ 36.73% diff coverage

View coverage diff in Codacy

Coverage variation details
Coverable lines Covered lines Coverage
Common ancestor commit (b8d51b2) 92640 25418 27.44%
Head commit (f1e3f7d) 92677 (+37) 25448 (+30) 27.46% (+0.02%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details
Coverable lines Covered lines Diff coverage
Pull request (#3839) 49 18 36.73%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@butonic butonic added the Type:Maintenance E.g. technical debt, packaging, etc. label Oct 27, 2026
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
The remote instance still sees the share with the role it was created
with, so the upload coordinator refuses the upload there instead of the
owner's instance failing it with 500.

Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
The upload coordinator now lives in a stack of PRs on
opencloud-eu/reva, so we require its tip directly instead of replacing
reva with the butonic fork. The tip only differs from the previous
pin by the orphaned upload rollback fix, which keeps nodes that existed
before the upload or that have content, and by dropping the OpenCloud
copyright from files written upstream.

related: https://github.com/opencloud-eu/reva/stack/952

Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Type:Maintenance E.g. technical debt, packaging, etc.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants