Migrations by gammazero · Pull Request #327 · ipfs/distributions

gammazero · 2021-01-12T05:28:27Z

This PR creates individual distributions, for each IPFS fs repo migration. This enables migration tools to download a minimal set of migrations needed to go from one version of IPFS to another.

The individual migrations will be available from the ipfs network, but are not shown on the IPFS distributions web page**. Each migration has the same ipfs path structure as any other migration, so it can easily be found even in the absence of a link on the distributions web page. For example, the ipfs-10-to-11 migration can be found at https://dist.ipfs.io/ipfs-10-to-11 and will have the same contents as other migrations. NOTE: The migrations will not be published to IPFS until after approval/merge of this PR.

This PR requires, and is built from, #320

Addresses issue #326

Additional modifications to the build scripts:

Allow an optional sub-package to be specified when creating a new distribution
Do not build web content if distribution contains no-site file
Fix minor cleanup issues
Fix CI issue with jq not being able to read files directly

** NOTE: There is a no-site file in each migration distribution that prevents building the web page content for that distribution.

CID: QmVxxcTSuryJYdQJGcS8SyhzN7NBNLTqVPAxpu6gp2ZcrR

welcome · 2021-01-12T05:28:30Z

Thank you for submitting this PR!
A maintainer will be here shortly to review it.
We are super grateful, but we are also overloaded! Help us by making sure that:

The context for this PR is clear, with relevant discussion, decisions
and stakeholders linked/mentioned.
Your contribution itself is clear (code comments, self-review for the
rest) and in its best form. Follow the code contribution
guidelines
if they apply.

Getting other community members to do a review would be great help too on complex PRs (you can ask in the chats/forums). If you are unsure about something, just leave us a comment.
Next steps:

A maintainer will triage and assign priority to this PR, commenting on
any missing things and potentially assigning a reviewer for high
priority items.
The PR gets reviews, discussed and approvals as needed.
The PR is merged by maintainers when it has been approved and comments addressed.

We currently aim to provide initial feedback/triaging within two business days. Please keep an eye on any labelling actions, as these will indicate priorities and status of your contribution.
We are very grateful for your contribution!

jessicaschilling · 2021-01-19T17:03:31Z

@Stebalien Are you able to review this, or pass along to someone else who can? Thank you!

Stebalien

Looks reasonable to me. I didn't check every change in detail, but if it works, it works. And if something goes wrong, a manual "diff" should catch that.

jessicaschilling · 2021-01-20T20:39:03Z

@olizilla Are you wanting to wait to merge this until #320?

olizilla · 2021-01-20T21:24:22Z

This PR builds on top of #320, so we should merge #320 first. More generally for merge decisions on this repo, I defer to Adin as he has to use this to release go-ipfs and Steven as the most prolific contributor.

aschmahmann

LGTM. @gammazero it looks like you've also fixed and cleaned up a few unrelated things.

I left some questions/comments trying to clarify this to make sure A) I understand B) that if we check into why a change was made a few months from now we know why.

aschmahmann · 2021-02-16T16:39:58Z

-		sed "s/ABCGHREPOXYZ/$(sedEscapeArg "$repo")/g" templates/Makefile | sed "s/ABCDISTNAMEXYZ/$name/g" > "dists/$name/Makefile"
+		sed "s/github.com\/foo\/bar/$(sedEscapeArg "$repo")/g" templates/Makefile | sed "s/cmd\/bar/$subpkg/g" > "dists/$name/Makefile"


Is this just an ancient bug fix?

Yes. I was a bug leftover from when the string ABCDISTNAMEXYZ appeared in the Makefile template.

aschmahmann · 2021-02-16T17:08:01Z

+const event = new Date()
 const DIST_DOMAIN = 'dist.ipfs.io'
-const MFS_DIR = `/${DIST_DOMAIN}`
+const MFS_DIR = `/${DIST_DOMAIN}` + '_' + event.toISOString()


This means that people who run this will end up with one MFS reference per-build instead of one per dist website. Given that my understanding is we're pinning all the files we add anyway the only problem that could come up with this is just it polluting your MFS root space.

nit: WDYT about having some nesting to keep ipfs files ls / cleaner? An example might be /${DIST_DOMAIN}/${DIST_DOMAIN} + '_' + event.toISOString()`

I actually want to pollute MFS root. The is so I see that I have multiple dists and remember to remove those that I no longer need. This is not for the purpose of archiving dists, but to avoid stepping on a previous one. Once I am happy with a dist, I expect to remove previous ones.

@Stebalien suggested in the previous review, that I use a random string here. I decided that date was more useful to determine which was the most recent.

Honestly, keeping old ones should be fine as it shouldn't actually use any more data. It also makes it easier to delete them (delete the root).

But either way. I don't have strong opinions.

this is just a local thing for the builder we can always change later if it becomes a problem with no significant consequences, so we can just leave it as is and figure it out.

- Optional argument to dist.sh to specify sub-package - Fix Makefile to allow dist.sh to correctly edit variables - Do not generate web site data if distribution contains "no-site" file.

Each fs repo migration is build as a separate distribution

- Remove filtered_versions files - Do not create filtered_versions files

- Do not check that version files are sorted since that does not work with semver sorting.

Update migrations version to match current fs-repo-migrations.

- Fix build-go.sh to build using go modules, instead of GOPATH. - Update migration versions to 2.0.0

- Add build target "darwin arm64" for migrations - Update build matrix template

gammazero force-pushed the migrations branch from 201672e to 3d6d1d2 Compare January 12, 2021 06:00

This was referenced Jan 12, 2021

Fix/separate migration bins ipfs/kubo#7857

Merged

Give each migration its own binary ipfs/fs-repo-migrations#116

Merged

Migration downloads individual migration binaries ipfs/ipfs-update#134

Merged

gammazero force-pushed the migrations branch 4 times, most recently from 7a73c63 to 407fa69 Compare January 19, 2021 15:51

jessicaschilling assigned Stebalien Jan 19, 2021

gammazero force-pushed the migrations branch from 407fa69 to 2f04214 Compare January 19, 2021 17:55

Stebalien approved these changes Jan 20, 2021

View reviewed changes

Comment thread build-go.sh Outdated

Comment thread build-go.sh

Comment thread scripts/patch.js

gammazero mentioned this pull request Jan 20, 2021

Plan proposal for future of fs-repo-migrations ipfs/fs-repo-migrations#98

Closed

olizilla mentioned this pull request Jan 20, 2021

feat: only build versions that dont exist on dist #320

Merged

4 tasks

gammazero force-pushed the migrations branch from 2324f3a to ae1c17d Compare February 1, 2021 22:04

aschmahmann approved these changes Feb 16, 2021

View reviewed changes

gammazero force-pushed the migrations branch 2 times, most recently from 31ab6cd to 492ae26 Compare February 23, 2021 23:11

gammazero force-pushed the migrations branch from c7a9944 to 0aeea8c Compare March 26, 2021 06:41

gammazero added 8 commits March 31, 2021 11:20

Fix Makefile template and modify build scripts

8b5581b

- Optional argument to dist.sh to specify sub-package - Fix Makefile to allow dist.sh to correctly edit variables - Do not generate web site data if distribution contains "no-site" file.

Add distribution for each migration

ba237e5

Each fs repo migration is build as a separate distribution

Fix description file and Makefile

0904592

- Remove filtered_versions files - Do not create filtered_versions files

Remove 32-bit build for Mac since this is dropped in go1.15

fa0be5d

- Do not check that version files are sorted since that does not work with semver sorting.

fix lint error

9f4ffab

Fix CI issue with jq reading dist.json

f26086e

Review change: set and resolve DIST_PATH at beginning of script

73101f0

Create unique mfs root dir for dist.ipfs.ip

3240a3a

Update migrations version to match current fs-repo-migrations.

gammazero added 12 commits March 31, 2021 11:20

fix lint warning

f23a28e

review change: keep all dists in same mfs directory

edb12c2

Rename migrations from ipfs-x-to-y to fs-repo-x-to-y

a54e5d8

- Fix build-go.sh to build using go modules, instead of GOPATH. - Update migration versions to 2.0.0

Support sub-module version tagging

99cc11b

Update build matrices

8b0078e

migrations at v1.0.0

533df71

Sub-package version tag separate from versions

998ba94

Apply correct fix for building with go modules and g1.16

222f83e

Change variable name

8d348ea

Undo new version for ipfs-update

3b177a2

Linter fix

1cb8a21

Build for new Mac hardware

fbfcb7f

- Add build target "darwin arm64" for migrations - Update build matrix template

gammazero force-pushed the migrations branch from 0aeea8c to fbfcb7f Compare March 31, 2021 18:30

Do not change go-ipfs build_matrix

be321a5

aschmahmann merged commit eef9d68 into master Mar 31, 2021

gammazero deleted the migrations branch April 2, 2021 05:49

		sed "s/ABCGHREPOXYZ/$(sedEscapeArg "$repo")/g" templates/Makefile \| sed "s/ABCDISTNAMEXYZ/$name/g" > "dists/$name/Makefile"
		sed "s/github.com\/foo\/bar/$(sedEscapeArg "$repo")/g" templates/Makefile \| sed "s/cmd\/bar/$subpkg/g" > "dists/$name/Makefile"

Conversation

gammazero commented Jan 12, 2021 • edited by aschmahmann Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

welcome Bot commented Jan 12, 2021

Uh oh!

jessicaschilling commented Jan 19, 2021

Uh oh!

Stebalien left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jessicaschilling commented Jan 20, 2021

Uh oh!

olizilla commented Jan 20, 2021

Uh oh!

aschmahmann left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aschmahmann Feb 16, 2021

Choose a reason for hiding this comment

Uh oh!

gammazero Feb 17, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aschmahmann Feb 16, 2021

Choose a reason for hiding this comment

Uh oh!

gammazero Feb 17, 2021

Choose a reason for hiding this comment

Uh oh!

Stebalien Feb 17, 2021

Choose a reason for hiding this comment

Uh oh!

aschmahmann Feb 18, 2021

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

gammazero commented Jan 12, 2021 •

edited by aschmahmann

Loading