Improving megolm key storage (meta)

[ara4n]

Yesterday had various discussions with @richvdh and @dbkr revisiting improving megolm key storage & sharing. This is a meta-bug to summarise and link to the various issues.

Scenarios where we need to better manage missing megolm keys:

1. Recovery of an account if a user only has one device, and deletes it without explicitly exporting their keys. (This is a scenario that several telco-scale Matrix users are worried about). (this is sort-of a variant of Users in short-lived incognito tabs may never be able to share megolm keys, unless they can dehydrate/rehydrate their device somehow. #3825, although the conclusion there looks flawed)

2. Fixing UISIs (due to races, decentralisation, etc) in general by somehow repopulating the missing keys. Currently handled by key-share requests, but this assumes you have the keys on one of your other devices. (https://github.com/vector-im/riot-web/issues/2286)

3. Fixing scenarios where devices run out of local storage for key data due to storage constraints. (we can run out of localstorage #3660)

4. Fixing missing keys for messages sent to you after you were invited but before you joined a room. (Let megolm session keys be available to devices added by invited users since the point they are invited #2713 or https://github.com/vector-im/riot-web/issues/3821\)

5. Fixing missing keys for messages in the distant past of a room before you joined it.

Conclusions of the conversation were:

• We can potentially fix scenarios 1,2,3 by storing encrypted keys on the server (https://github.com/vector-im/riot-web/issues/3661\). In order for scenario 1 to work, we'd need to encrypt the keys using some passphrase that the user can share between devices. We could store the keys in roomAccountData, but it feels like we might need to access them with more granularity than the blunt bulk roomAccountData API. This would replace the keyshare request stuff.
• Scenario 4 is probably best fixed by encrypting for devices as if they were joined, which means changing federation to track participating devices for invited users. Alternatively we could proactively keyshare the missing keys to the user once they do join (but this is racey, if the inviter is not online when the invitee joins).
• Scenario 5 may be best fixed out-of-band by making it a user problem (#6454). We let clients export megolm keys for a given room (or a given timeframe within a room), and let users negotiate with each other to share keys - it's the equivalent of asking a channel if someone wants to share their IRC logs. This avoids us having to somehow present a complex in-app UI for key share requests for ancient history and expect users to make a sensible decision in the face of an attacker, and instead make it a social problem. (We could alternatively make it a room configuration option as to whether users are allowed to transparently request keys from other users, but it's hard to see how these requests wouldn't be abused by an attacker).

Scenarios 4 & 5 could also be fixed by providing an in-app way to proactively share keys with users (as opposed to responding to a keyshare req): e.g. give the option when inviting a user to a room to also send them keys for past history (if the history visibility rules for the room allow it). This is an extension of #2713. Presumably we could use the existing keyshare toDevice mechanism to do this.

[ara4n]

Cross-linking to #2996 as the "encyclopaedia of UISI causes" bug, which complements this more use-case driven bug.

[ara4n]

A draft for a spec for incremental backup of encrypted keys on the server (#3661) now exists at https://docs.google.com/document/d/1MOoIA9qEKIhUQ3UmKZG-loqA8e0BzgWKKlKRUGMynVc/

[edit: superceded by MSC1219]

[cyphar]

Aside from a 👍 from myself (and it's great to see that this is going to be solved), I'd like to point out that in my experience this issue causes almost all of the push-back against using Riot for conversations with people I know. Having access to your conversation history is incredibly important for almost anyone you talk to, and Riot (unlike IRC clients or Signal) doesn't store a copy of the decrypted (or reencrypted-with-a-static-and-not-used-for-anything-else key) conversation anywhere. And since I don't currently host my own homeserver, encryption is mandatory (though even if I was running my own homeserver I would use encryption).

This means that important conversations that you had in the past might be lost through your session being killed in your browser (something that's happened to me in the past, meaning I can't read conversations from >2 years ago since key exporting wasn't implemented back then IIRC). I can think of many situations where this would cause an inconvenience (or even grief) to users who use Riot to communicate with everyone. It's definitely caused me some inconvenience...

IMHO this is such a major issue for users that wish to use encryption that the warning when you enable encryption in rooms should be modified to say something along the lines of

If you are not careful and log out in certain situations, you may lose the ability to read old messages permanently. If this will cause an inconvenience, please reconsider using encryption with Matrix -- or read [this guide] on how to manage your keys.

Because currently this issue is not documented anywhere outside of GitHub issues, as far as I can tell!

[ara4n]

this would also help solve #2671

[richvdh]

Looks like all the linked issues have been resolved. I don't think this issue has much value nowadays.