Discuss Goals

[ThomasWaldmann]

Ideas about potential goals for Borg

Borg is a fork of Attic and it was done to allow some different approaches to development, goals and policy (for details about how and why the fork happened, see jborg/attic#217 and the attic mailing list):

Openness

• Borg is intended to be an effort by "The Borg Collective"
  (see AUTHORS) and you can be assimilated into it, if you like.
• Welcome feature requests, discuss their general usefulness.
• Accept pull requests of good quality and coding style,
  give feedback on PRs that can't be accepted "as is".
• Openly discuss about stuff, don't work in the dark.

As simple as possible, but not simpler

• Nobody likes tools that are too complicated, ...
• ... but nobody likes everything to be fixed and inflexible either.
• Do the usually right thing by default, but offer other options.
• Accept the fact that the usually right defaults might be totally unfit
  for some users / use cases.

Compatibility - boon and bane of backup software

• Don't break it accidentally / without good reason / without warning.
• Break it if above does not apply. needs more thoughts/discussion
• As the fork is "new software" from the perspective of a Borg user or
  a Borg packaging distribution, there is no past we need to stay compatible
  with - we have the chance to break compatibility and change everything
  that we think needs changing.
• Over time, we'll have more users and incompatible changes get harder.
• Avoid getting into the "compatible forever" trap - we should maybe not
  assure compatibility of development versions nor spanning major releases.
• When used for long-term archiving, special considerations and care are required.
  E.g. a development snapshot of Borg might be not the right thing for this.
  Also, Borg exists to be able to change things. So if you don't like or can't
  live with a changing software, don't use it.

[silvio]

• Avoid getting into the "compatible forever" trap - we should maybe not assure compatibility of development versions nor spanning major releases.
• When used for long-term archiving, special care might be required.

I think extracting files of an archive should possible every time independent of the used version of borgbackup. At least with latest bb version should extracting of older version of archive possible. But not vice versa.
A converting operation could be the solution (thx, @joolswills)

[joolswills]

One thing I would like to see, is if/when the repository format changes to offer new features, the ability to convert in place existing repositories, rather than having to start again.

Agree with your points. Thanks for your efforts.

[ThomasWaldmann]

Converters are an option to think about when going from one release to an (incompatible) newer release.

But, someone would have to write that code and it is a burden and slows down development. Also, converting a large amount of historical archives might be a very time and space consuming affair, thus maybe impractical even if you had a converter.

Thus, I am still proposing just breaking compat now and then. Someone who wants conservative long time archiving might better off using tar (or attic) maybe, not with something that is being heavily developed. And the whole point of this fork is to accelerate development. :)

Also, of course nobody wants to write converters that convert between development snapshots and alpha/beta/rc releases.

[joolswills]

agreed in regards to dev, but if a new feature is added such as a new compression method it would be nice to do things like recompress etc. Also things like decrypting/encrypting repositories after set up would be useful. Just ideas anyway!

[anarcat]

i really like the promise of attic to keep backwards compatibility forever for storage.

who knows when i will need to restore this old backup? do we need to make such changes now anyways?

maybe an alternative here would be to stabilise versions at some point and treat those things as "API changes". so we could have a X.0 release that will be backwards-compatible with X-1.0 and forward-compatible with X+1.0 so that you could migrate your repo by upgrading incrementally?

[ThomasWaldmann]

@anarcat I thought about it quite a lot. Keeping something compatible forever sounds great, but from development and maintenance standpoint, it is a pain.

For example, look at flash player or windows - they try to be compatible forever (or at least over very long time) and it results in an accumulation of a lot of crap code (there was a talk by FX at a CCC conference about it, that went into quite some detail). They basically rewrote that thing multiple times (for reasons), but always kept the old code also. Thus you have now not just the bugs in the latest code, but all remaining bugs in all the old versions, too. Windows still has broken time stdlib functions, because they were broken in the same way on DOS.

I am not saying attic is quality like flash player. :) We are lucky that the code base is quite good, but I found some places where it is too hardcoded or where maybe even a layer is missing and blocking the development of a needed feature. Maybe we will find more over time, that has to be seen.

Your idea with +/- 1 version compatibility is somehow similar to a converter, right?

[ThomasWaldmann]

BTW, I pushed some code to the repo. It is not compatible with attic due to changes of the magic strings (like ATTIC_KEY, ATTICSEG, ATTICIDX). Other than that, it was mostly the content of the rather conservative "merge" branch + s/attic/borg/g. See CHANGES.txt for details.

[anarcat]

so i understand how hard it can be for windows to keep backwards compat with crap like DOS, or keep flash player stable. i think that's a different problem than what we are living here: as you said, attic is fairly well designed and implemented, and there are some tweaks we want to do, unless i misunderstand something deeper here.

a good example is changing the ATTIC strings: why would we do that at all, if not gratiously break backwards compatibility?

[aride]

@ThomasWaldmann it's true that backward compatibility can eventually turn into an inconvenience. But please keep in mind this is backup software we're talking about. You can't treat it like some UI or some other self-contained software. Its data is designed to last an indeterminate amount of time. Otherwise, it is not a backup. Please name one backup software that doesn't treat backup format with extreme care. All formats I can think of are very long-lived, especially those that have been successful think cpio, pax, tar... there may be format versions, but those are very few and still handled by recent versions. Proprietary backup solutions may show a bit more variability but still they can read old formats almost without exception.

Breaking compatibility with upstream attic is not a smart move, especially without a very good reason and a data migration path in place. To break it for some silly strings is simply absurd. This issue is called "Discuss Goals", so let's do that. What is the mission of backup software?: is it to have flexibility? is it to fix bugs fast? No, it is to keep data safe. Those things are nice, and we all want them, but they're not the software's mission. If backup software fails to keep data accessible, it fails as backup software, it's simply useless.

Besides, if so many changes are required to the backup format, then it is designed badly. Is the attic backup format design bad? Why? What are its shortcomings specifically?

Now, I can understand making no promises of data integrity or compatibility for development versions. That's just common sense. BUT development should strive from the start to stick to one and only one format (be it attic's or some variation on it). Format should be versioned, just in case there appears a real need to change it in the future, but if it's well designed it should support most features we might want. And if it doesn't, then either it wasn't well designed or we really should think hard whether such feature is really needed. And at a minimum, read-only backward compatibility is a must.

Just my opinion, of course.

[barsanuphe]

It's a fork. If changes are necessary, now is the time, especially if those changes make borg more resilient to future changes.
Also, the idea of backup software being able to keep files and format "forever" is a little ambitious (for borg or attic). Recently a msgpack bug was discovered: attic is not standalone, its very format is dependant on third party libraries. If those evolve or are abandonned, you will need to do something about your repositories.
But I agree read-only backward compatibility is a minimum; the ability to update the format (or change compression level/encryption) would definitely come in handy.

[aride]

@barsanuphe I didn't say "forever", I said an indeterminate amount of time. Meaning, "long enough". Is the time between borg releases long enough? I don't think so, not by a long shot. Tar files have lasted decades, I don't see a reason not to strive to reach that kind of quality. Users expect that kind of quality from backup software, not having to reencode archives at any upgrade.

You say "if changes are necessary", that's precisely my point. That necessity needs to be spelled out very clearly, for very good reasons, before format changes can be considered. And they haven't. I agree that now is the right time to discuss those needs. The sooner the format is established, the sooner borg development can begin.

[Ernest0x]

Converters are an option to think about when going from one release to an (incompatible) newer release.

But, someone would have to write that code and it is a burden and slows down development.

Maybe, but must be done.

Also, converting a large amount of historical archives might be a very time and space consuming affair, thus maybe impractical even if you had a converter.

Maybe or maybe not. You must provide the conversion functionality and let the user decide. He may choose to prune archives before converting and keep only a small subset of them. Or he may have the time and space and wants Everything.

[maltefiala]

Like anarcat and aride I too believe this fork should improve attic in a sensible way without braking too many things at once. Sure, nobody knows what bugs will be found in the future. However, I really don't see why changes in 3rd party libraries should brake compatibility with attic as attic needs to be upgraded as well in such a case.

An example would be the readability of attic's code. Variable names like "t0" and "st" are nice to write but don't make it very readable and brake PEP 0008 as they clearly aren't words:

lowercase with words separated by underscores as necessary to improve readability.

So should we change those variable names to something better to benefit readability or should we stay with them to benefit code compatibility? I would vote for the latter at the moment.