Declaring End-to-end Encryption stable and turning it on by default for private rooms.

[ara4n]

Somehow we seem to be missing a high-level tracking bug for the endgame of E2E. (The starting point was matrix-org/matrix-spec-proposals#501, but that's a spec bug and feels a bit weird to hijack it for this).

In order to declare E2E stable (and by extension Matrix), we need to:

• Cross-sign devices at login to eliminate verification warnings (We could cross-sign devices to aid trust when new ones join a room #2714)
• Optionally incrementally backup encrypted E2E keys to your HS so if you lose all your devices you can restore your history via a recovery key that you own. (Improving megolm key storage (meta) #5675)
• Improve verification UX (comparing per-verification mnemonic passphrases rather than public keys) (Implement E2E key verification process #2142)
• Hook up key sharing to let history be visible from before you joined a room (but after you were invited to it), history visibility allowing. (Let megolm session keys be available to devices added by invited users since the point they are invited #2713) - mainly fixed, other than for edge case where users add devices after being invited to a E2E room
• Have less than 0.0x% unresolved unable-to-decrypt errors reported through telemetry ("Unable to decrypt: The sender's device has not sent us the keys for this message." (The UISI bug) #2996, Telemetry to report incidence of UISIs #6390)
• Fix device lists getting out of sync over federation (E2E device lists can get out of sync over federation when users delete their devices. #6974)
• Support full-text search within E2E rooms (Client-side search for E2E #2548, solved by github.com/matrix-org/seshat)
• Support push and in-app notifications in E2E rooms at least by locally calculating them. We could also support setting metadata on the messages to bing users, especially if we had pseudonymous MXIDs.
• Support NotifPanel in E2E rooms (Notif panel needs to know how to deal with encrypted room #6874)
• Turn it on by default for private rooms. All rooms created with the intent of use as a 1:1 or a private group chat must be E2E (and homeservers would enforce that, as best they can, c.f. Support only E2E or public rooms on my homeserver matrix-org/synapse#3269), although obviously a broken client could still try emitting unencrypted events into such a room). The code exists for this already over at https://github.com/matrix-org/synapse/pull/3426/files.

Ideally:

• Provide some kind of E2E daemon/proxy to stop non-E2E clients/bots/bridges/curl-users being unceremoniously dumped out in the cold. (#6778)
• Improve the UX for managing the trust of the devices which are in a given room (E2E device/key management dashboard to replace UnknownDeviceDialog #4522). In practice if verification is working nicely this might not be so important.
• Optionally hook up key sharing to let history be visible from before you were invited to a room (history visibility allowing) (#2286)
• Get an end-to-end audit of crypto spanning a set of reference implementations - e.g. matrix-{web,ios,android}-sdk + synapse.

[aaronraimist]

#6959 should probably be in here at least under ideally

[dbkr]

I think '* Optionally hook up key sharing to let history be visible from before you were invited to a room' needs to be non-optional, or if not then change the history visibility settings for e2e rooms to line up with what's actually possible, ie. remove 'anyone' and 'Members only (since the point in time of selecting this option)')

[ilmaisin]

It was originally promised that e2e encryption would be enabled by default when it was out of beta. That didn't happen.

But anyway, as I understand, there are basically three things to be done before e2ee can be default: cross-signing, local search and notifications. Are there any on-going efforts to achieve those?

[ara4n]

yes, of course. cross-signing is in the final stages; notifications just got largely fixed on riot/web and already worked on mobile; local search development is kicking off again as of tomorrow. we also want to fix element-hq/element-meta#80 before turning it on by default.

[ilmaisin]

Well, cross-signing has been "in the final stages" for half a year now. A honest, detailed explaination on what is really happening would be nice.

[ara4n]

a quick but honest detailed explanation is:

• @uhoreg has been working solidly on it for ages
• he shipped working SDK & server code about 2-3 months ago, as per https://scitech.video/videos/watch/1c8a0262-dee9-4927-b9e9-9856a7f4d8bf and https://scitech.video/videos/watch/d1ef04a8-397a-4570-a9a9-cef143ea637c
• we then made an architectural decision to switch to using 1:1s rather than toDevice messages for verification, so you get history of verifications persisted nicely in your DM with the person who is verifying you (or vice versa).
• this work has now been implemented too (as of last week or so)
• meanwhile, much of the Riot/Web team got pulled onto dealing with Privacy stuff, thanks to https://matrix.org/blog/2019/06/30/tightening-up-privacy-in-matrix
• this work is now finished and shipping next week (hopefully); meanwhile, work has commenced again on hooking up the E2E UX for cross-signing in Riot.
• The current proposed UX changes are: https://matrix.org/_matrix/media/r0/download/matrix.org/iYotsQUzFQTgsZxaiNyBEfzF (warning; 7MB JPEG)
• There is a lot of work to land this on Riot across all 3 platforms, as they share no code, so it will have to be implemented 3 times over. We'll do Web first, and probably have to iterate a bit, and then mobile will follow.

Sorry it's taken a while; turns out that this stuff is hard, and we've been juggling a lot of stuff thanks to privacy dramas etc.

[ara4n]

meanwhile, e2e search has been progressing first via https://github.com/matrix-org/pantalaimon (and works well; i use it daily, but it's not integrated tightly with Riot), and now by https://github.com/matrix-org/seshat, which I believe works with Riot/Desktop although I haven't tested it yet.

[ara4n]

turning it on by default is formally proposed as an MSC now, as part of the Canonical DMs proposal: (point 5 of https://github.com/matrix-org/matrix-doc/blob/travis/msc/immutable-dms/proposals/2199-canonical-dms.md#creating-dms)

[ara4n]

...and UISI errors (aka UTDs) are almost unheard of, in my experience. We fixed a major remaining cause of them back in July (matrix-org/synapse#5693) and have built out a whole new project for adding OpenTracing to Synapse so that whenever find further ones, we can pull up a full visualisation of precisely what went wrong to diagnose it and so snuff them out conclusively one by one if/when we see them: https://github.com/matrix-org/synapse/pulls?q=is%3Apr+opentracing+is%3Aclosed.

I think those were the only pending points.

[ilmaisin]

#11125 is supposed to implement the e2e search, but it seems to have been sitting idle for two weeks now. By the way, Conversations has been doing e2ee search for ages.

The progress still looks like being really slow to me, though I absolutely may have missed something as I am not an expert on this stuff.

[ara4n]

You’ve missed that the e2e search is 3 layers deep: riot-web, matrix-react-sdk and seshat itself (which unlike Conversations is designed to be crossplatform, written in Rust). The react-sdk layer was active less than a week ago: https://github.com/matrix-org/matrix-react-sdk/tree/poljar/seshat-rebase and the seshat layer was active a few hours ago: https://github.com/matrix-org/seshat/tree/sqlcipher. The feature works great; i’ve been using it for a few weeks. The only issue is that it stores its data currently in plaintext on disk, which is not ideal - so we are reencrypting it when it sits on disk.

[ilmaisin]

Is there any way to donate specifically to e2ee work? Looking at those "referenced" notices, it seems that I am not the only one who would like to have this as a much higher priority. I don't want my donations to be spent to the fiddling with emojis or any other such bells and whistles.

Should I open a bountysource entry for this issue or something?