Repository navigation
Performance of device list updates could be improved (aka /login is unusably slow) #7721
Description
I was investigating why logging in with a new device was leading up to poor performances on my account. It seems like both sending the device list update to other homeservers and other homeservers in turn hitting GET /_matrix/federation/v1/user/devices/{userId}.
Looking a bit further, it looks like both code paths end up calling _get_e2e_device_keys_txn in the database code, which seems to be pretty expensive, and is called multiple time in a row, yet isn't cached. This results in quite bad performances:
On the bottom right graph the _get_e2e_device_keys_txn transaction (in blue) is originating from sending out the m.device_list_update EDUs, while the get_devices_with_keys_by_user one (in purple) is originating from responding to GET /_matrix/federation/v1/user/devices/{userId}. Both transactions involve calling the _get_e2e_device_keys_txn function (in fact, the first one does only that).
On the bottom left graph these two transactions are colored respectively in green and purple.
Activity
erikjohnston commentedon Jun 18, 2020
I think the next steps here are to add a cache 👍
[-]Performances for device list updates could be improved[/-][+]Performance of device list updates could be improved[/+]hifi commentedon Feb 21, 2022
In #12048 I had an issue where logging in with a new device pretty much kills synapse for a while because of the device notify on login. Just batching up and slowing down sending the notifies would help a lot.
richvdh commentedon Mar 2, 2022
It's worth mentioning that this effectively prevents logging in with Hydrogen (cf element-hq/hydrogen-web#699)
richvdh commentedon Mar 2, 2022
oh, and fluffychat, which seems to do the same thing
erikjohnston commentedon Mar 2, 2022
Note that there seems to be two related but distinct problems:
_get_e2e_device_keys_txnis slow as mentioned in the issue descriptionnotify_device_updatesis slow@richvdh ooi what were you seeing? Or was it a combination of the two?
richvdh commentedon Mar 2, 2022
I was seeing all requests to my HS responding with a 502, due to my reverse-proxy timing out. The mean reactor tick time is about 15 seconds, which means Synapse is essentially unable to respond to anything. I think it's very busy sending out federation updates.
richvdh commentedon Mar 2, 2022
related: #5373
hifi commentedon Mar 2, 2022
@erikjohnston In #12048 I disabled
notify_device_updatestemporarily and I could login just fine without delays and Synapse was chugging along without stalling.erikjohnston commentedon Mar 4, 2022
#12132 will hopefully help a bit here, as it smears out sending out device list updates a bit. However, it won't help for the other issues that we're seeing.
My current thinking is that we should try and get rid of the use of
store.get_users_who_share_room_with_userinnotify_device_update, as that is expensive for larger accounts (both in CPU and memory terms). We can instead pull out the rooms for the user, which is enough to wake up the notifier. Then, instead of recording the device update per remote server, we store the device update per room1. When the federation sender pulls out the device updates it can find the hosts in the room.This does lead to the same amount of work being done (as
self.store.get_users_who_share_room_with_usereffectively just doesget_rooms_for_userfollowed by a loop ofget_users_in_room), but has the advantage of:/login(or whatever) request sooner; andWe can also do a vaguely similar thing for presence, i.e. get rid of
store.get_users_who_share_room_with_user, which should help the perf overhead of presence.1. This can be done in a backwards compatible way be recording in the DB both device list updates by server and by room at the same time, then in a future release stop doing the former and bump the minimum DB schema version.
ara4n commentedon Mar 19, 2022
I literally can't log in any more thanks to this. It would be very much appreciated if this could be prioritised...
[-]Performance of device list updates could be improved[/-][+]Performance of device list updates could be improved (aka /login is unusably slow)[/+]Track device list updates per room. (#12321)
erikjohnston commentedon Apr 27, 2022
1.58.0rc1 should now include the fix described in #7721 (comment), so I think this is fixed from a user perspective. Logging in on a large account still takes a lot of CPU and DB though, so we may want to investigate that, but for now I think we can close this issue.
chagai95 commentedon Jun 6, 2022
I still can not log in with an account with many rooms, I have 8 GB RAM and 8 cores, is there anything in the logs that I can keep an eye out for? I tried playing around with
get_users_who_share_room_with_userandget_users_in_roombut had no success. Anything else I am missing perhaps?