Repository navigation
Cloudsmith opencpn workspace and Plugin Repositories #1460
Description
We have a limit of 50 GB before we are shut down.
We are using about 40 GB having had a recent large spike in use.
We will be cut off at 50 gb, so we've got to do cleaning and pruning.
I've finished some bash scripts and have run them on the CS Opencpn REPOS alpha, beta, prod to determine how much space is being used by each plugin and approximately how much space can be saved by trimming and cleanup. I had already cleaned up alpha earlier, doing a script trial , saving a little over 1GB then.
alpha - 0.96 GB can be reclaimed. MAX_RETAIN : 60 MAX_DAYS: 60
beta - 3.65 GB can be reclaimed. MAX_RETAIN : 100 MAX_DAYS: 100
prod - 16.6 GB can be reclaimed. MAX_RETAIN : 150 MAX_DAYS: 100
Total = 21.26 GB
The three detailed log files are uploaded here. They take about 10 minutes to create.
The scripts are clean-alpha-dryrun.sh, clean-beta-dryrun.sh, and clean-prod-dryrun.sh. These scripts are identical, except renamed for each repro and given different settings. See below.
- ALPHA I have run "clean-alpha-dryrun.sh --deletion" once already and gained 1GB.
- BETA I'd like to run the beta script "clean-beta-dryrun --deletion" next, with the "exception" of Dashboard_Tactics (we have to keep this beta repo full because it was used for Prod). This will be pretty easy to do now, but developers should look at the dry runs carefully and let me know if you are in agreement to do this. (I don't have too much choice in this matter as we are approaching the limit) but I can make some accommodations for good reasons.
- PROD Next I've been thinking about the Prod Repo. This one is difficult.
- all the primary files over history. It is quite big. we could save 16gb with the settings I have put in.
- but that would not be a good result due to the 100 day max_time
- Max_Time is a problem, we want good primary files that span over a long time, not small changes.
- We could try filtering by version numbers. We want good primary versions.
- Some ideas for the filter.
- Remove all versions that do not have a 0 in the 4th place, then keep the rest for 4 years?
- Remove all version that do not have a 0 in the 3rd and 4th place, then keep the rest for 3 years?
- Keep only the last versions in a string, IE 1.2.78.0 and delete all 1.2 version before that?
- Keep only the last versions in a string, IE 1.2.5.62 and delete all 1.2.5 versions before that?
I am certain that every plugin's versioning use and policy is different, and perhaps we should be more discerning about how and when to increment versions in the future to easily keep the best ones and drop all the little changes.
Your suggestions will be appreciated. I think I will also put some of this in an Issue under Opencpn/Plugins
Just finished a script(s) for alpha, beta and prod to manage CS Opencpn repos and files. It has these settings:
- dryrun default or call delete.
- single or all plugin repos
- outputs to dated log files
Activity
rgleason commentedon Nov 1, 2026
ALL PLUGIN DEVS - HEADS UP!!!
Here are the Output Logs for Alpha, Beta, Prod.
Please take a look at the plugins files I would like to remove in the BETA repo. They are listed.
Please advise how best to remove unused and inconsequential files from PROD, and what filters to create.
20261031-234722-beta-dryrun.log
20261031-233320-prod-dryrun.log
20261031-235532-alpha-dryrun.log
jongough commentedon Nov 1, 2026
I have adjusted the limits for both ocpn_draw and testplugin to reduce the number of packages. I am just a little concerned about the retention period for production packages as some plugins do not get updated on a regular basis and may therefore fall off the end.
rgleason commentedon Nov 1, 2026
Thanks Jon, I agree. Do you recall what the settings were that you decided upon?
rgleason commentedon Nov 1, 2026
Some other thought about the PROD repo.
Is it safe to state that any file with a number is the fourth position 1.2.3.10 is probably not as important as 1.2.4.0
Same for 1.2.4.0 or 1.2**.4.98** is not as important as 1.2.5?
Based this concept, we could eliminate all file versions in a series up to the last 1.2.3.151 but keep this last in the series, for example.
Same for the 3rd position, keep the 1.2.4.0 version and the last in the that series, for example 1.2.4.57 If 57 is the last one in that series.
So, we would keep a sequence of files 1.2.3.0, 1.2.4.0, 1.2.5.0, 1.2.6.0
and another sequence 1.2.3.47, 1.2.4.5, 1.2.5.23, 1.2.6.89 which are all the last in the given series.
We would delete all other files.
This would preserve a larger time span of useful plugins for various versions of Opencpn.
Having culled that entire plugin repo, we could then set a max retainage policy 5 yrs.
When removing plugin files, they should be removed in pairs both xml and tarball.
We should have a script that finds all orphaned pairs and removes the existing file.
Does this seem reasonable?
jongough commentedon Nov 2, 2026
Major, Minor, Patch, Tweak. Normally Tweak does not seem to be used much (I do for my plugins, but...). Really the latest version, i.e. highest number, should be used if it is prod. It 'probably' should be used in beta, and in alpha it is just the latest distinguishable build but may or may not work well.
You should keep the highest number in prod, i.e 1.2.3.0 (del), 1.2.3.1 (del), 1.2.3.2 (keep as newest). If 1.3.1.0 appears then 1.2.3.2 should be kept for a while to allow 1.3.x,y stabilise.
bdbcat commentedon Nov 2, 2026
Rick....
I have an opinion on Prod:
I do not know if the Cloudsmith retention policy settings can support this, but...
Reasoning:
I highly recommend not to get involved in parsing revision numbers precisely. For a script, this could get pretty involved. Use publication dates, if possible.
rgleason commentedon Nov 2, 2026
I like this idea, and I think it can be implemented in script quite readily.
Now devs, won't have to wonder what version is being discussed quite so much, but there will be slow upgrade users.
This will save a tremendous amount of space.
Thanks for the advice about not doing versions, I have experienced that hole.
I would like to ask the general Plugin Dev community about this before implementing it
Results when running the script with these settings
prod - 16.6 GB can be reclaimed. MAX_RETAIN : 150 MAX_DAYS: 100
beta - 3.65 GB can be reclaimed. MAX_RETAIN : 100 MAX_DAYS: 100
alpha - 0.96 GB can be reclaimed. MAX_RETAIN : 60 MAX_DAYS: 60
Total Reclaimed = 21.26 GB
PROD two most recently published versions. And no more.
BETA MAX_RETAIN : 100 MAX_DAYS: 100
ALPA MAX_RETAIN : 60 MAX_DAYS: 60
@leamas as @jongough @Rasbats @mauroc @pob220 @TwoCanPlugIn @douwefokkema @seandepagnier @nohal
Please copy others I've missed so they know about this.
Also please provide your opinions on the best way the number versions, as Jon points out (major.minor.patch.tweak) and how best to use them.
bdbcat commentedon Nov 2, 2026
What does MAX_RETAIN mean to Cloudsmith?
TwoCanPlugIn commentedon Nov 2, 2026
Thanks for the heads-up although none of this affects me directly as I have my own Cloudsmith repo and I'm unlikely to hit any storage limits,
In terms of the "rule", maintaining n, n-1 versions of a plugin, this is something that I can achieve through keeping the plugin xml catalog up to date.
@bdbcat
from cloudsmith