diff --git a/design/backup.md b/design/backup.md new file mode 100644 index 00000000000..6bcd22bb028 --- /dev/null +++ b/design/backup.md @@ -0,0 +1,116 @@ +### File Backup Design + + This implementation of `BackupAgentBase` implements mechanism to backup keyspace into file. File could be on the local + file system or a remote blob store. Basic idea is backup task would record KV ranges and mutation logs into a + container(directory!) to multiple files. Restore would read files from a container and apply KV Ranges and then + mutation logs. + + Just like any thing else in FoundationDB, Backup-Restore also depend on monotonically increasing versions. Simplest way + to describe backup would be record KVRanges at a version v0 and record mutation logs from v1 to vn. That gives us the + capability to restore to any version between v0 and vn. Restore to version vk (v0 <= vk <= vn) will be done by + * just load KV ranges recorded at v0. + * if (vk == v0) then no need to replay mutation logs else replay mutation logs from v1 to vk. + + There is a small flaw in the above simplistic design, versions in FoundationDB are short lived (expiring in 5 seconds). + There is no way to take snapshot. There is no way to record KV Ranges for the complete key space at a given version. For + a keyspace a-z, its not possible to record KV range (a-z, v0), if keyspace a-z is not small enough. Instead, we can record + KV ranges {(a-b, v0), (c-d, v1), (e-f, v2) ... (y-z, v10)}. With mutation log recorded all along, we can still use + the simple backup-restore scheme described above on sub keyspaces seperately. Assuming we did record mutation log from + v0 to vn, that allows us to restore + +* Keyspace a-b to any version between v0 and vn +* Keyspace c-d to any version between v1 and vn +* Keyspace y-z to any version between v10 and vn + +But, we are not interested in restoring sub keyspaces, we want to restore a-z. Well, we can restore a-z, to any +version between v10 and vn by restoring individual sub spaces seperately. + +#### Key Value Ranges + +Backing up KV ranges and restoring them just uses standard client API, tr->getRange() to read KV ranges and tr->set() to +restore them. + +#### Mutation Logs + +For a backup enabled cluster, proxy keeps a copy of all mutations in system keyspace for backup task to pick them +up and record. During restore, restore task puts the mutations in system keyspace, then proxy reads them and applies +to the database. + +#### Restore + +As we discussed above KV ranges are for subspaces, where as mutation log is combined for the full keyspace. If we +take the same example + +* KV ranges => {(a-b, v0), (c-d, v1), (e-f, v2) ... (y-z, v10)} +* Mutation log => [(a-z, v0), (a-z, v1), .. , (a-z, v10), .. , (a-z, vn)] + => (a-z, vx) represents all mutations on keys a-z at version x. + +Restoring to version vk (v10 < vk <= vn), needs KV ranges to be restored first and then replaying mutation logs. For +each KV range (k1-k2, vx) that is restored we need to replay mutation log [(k1-k2, vx+1), .., (k1-k2, vk)]. But, this +needs scanning complete muation log to get mutaions for k1-k2, that is sub-optimal, for any decent sized database +this will take forever. + +Instead looking at restore on key space, we can replay events on version space, that way we need to scan +mutation log only once. At each version vx, +* Wait for all the KV ranges recorded before vx to restore first. +* Replay (a-z, vx), but ignore mutations for the keys that are not yet restored by KV ranges. + +For the above example, it would look like +* KV ranges: + * Restore KV ranges in parallel +* Mutation Logs: + * At v0: + * Wait for all KV ranges recorded before v0 to restore - nothing + * Nothing to replay as no keys restored by KV ranges yet. + * At v1: + * Wait for all KV ranges recorded before v1 to restore - {(a-b, v0)} + * Replay (a-b, v1) + * At v10: + * Wait for all KV ranges recorded before v10 to restore - Everything except (y-z, v10) + * Replay (a-x, v10) + * At v11: + * Nothing to wait + * Replay (a-z, v11) + * At vk: + * Replay (a-z, vk) + +Even though, in the above description logging mutations is shown as continuous task, this is actually devided into + two logical parts. During KV ranges are being backedup, we start a mutation log backup in parallel. Until, we complete + KV range backup and the parallel mutation logs backup, cluster is not in restorable phase. Once these two tasks are + completed, now cluster can be restored back to the last version. To be able to restore after even after the KV range + backup, we continue to backup mutation logs called differental log backup. With differential backup, we can restore + to any version since KV range backup completed. It would look like below + +``` + Backup Too large x x + Diff backup ---------------- ----------------- + Mutations -------- --------- + KV Range -------- --------- + Start x x + Version space --------------------------------------------------------------- + a b c d e f g h +``` + + To explain sequence of events here + +|Version|Event| +|-----------|----------------------------------------------------------------------------------| +| a | Asked to start backup| +| b | Started backing up KV ranges and mutation logs| +| c | Completed backing up KV ranges and mutation logs also mark backup restorable| +| c+1 | Started differential backup of logs| +| d | Decided backup is too large already and discontinued differential backup| +| e | Asked to start backup| +| f | Started backing up KV ranges and mutation logs| +| g | Completed backing up KV ranges and mutation logs also mark backup restorable| +| g+1 | Started differential backup of logs| +| h | Decided backup is too large already and discontinued differential backup| + +With the above backup scenario, we could restore to any version from c to d or g to h. No other versions are + restorable. + +#### Continuous Backup + +Instead of going through the pain of monitoring for backup becoming too large and restarting backup, we could just +have backup running continuously with KVrange task restarting once in a while (with some policy ofcourse). Continuous +backup is already committed, its not discussed here yet.