This repository was archived by the owner on Apr 26, 2024. It is now read-only.
Repository navigation
This repository was archived by the owner on Apr 26, 2024. It is now read-only.
Purge Room API fails to remove all rows from state_groups_state #6566
Description
Description
Invoking the /_synapse/admin/v1/purge_room endpoint on a room removes many, but not all, rows from state_groups_state (other tables not investigated).
Steps to reproduce
select count(*) as num_rows from state_groups_state where room_id = '!PcCidzjUKXHKImhTrB:matrix.org';=> 159049493curl -XPOST "https://example.com/_synapse/admin/v1/purge_room?access_token=$ACCESS_TOKEN" -d "{ \"room_id\": \"!PcCidzjUKXHKImhTrB:matrix.org\" }"=>{}select count(*) as num_rows from state_groups_state where room_id = '!PcCidzjUKXHKImhTrB:matrix.org';=> 997223
Version information
- Homeserver: ralith.com
If not matrix.org:
-
Version: 1.6.1
-
Install method: NixOS module
-
Platform: NixOS on dedicated atom server
Activity
Ralith commentedon Dec 18, 2019
Furthermore, multiple attempts to purge
!QtykxKocfZaZOUrTwp:matrix.orgfail to remove any rows, even as each attempt indicates success. There are more than 4 million!richvdh commentedon Dec 24, 2019
I assume it has managed to remove all the rows from
state_groups?The problem is that
state_groups_statelacks an index onroom_idso we can't just delete the rows fromstate_groups_state; instead we look up the state groups to delete fromstate_groupsand then delete the relevant groups.So that explains why it still doesn't do anything on a second attempt. What it doesn't explain is why it fails on the first attempt. There are two possibilities:
state_groupsfor some reason, so we don't know that we have to purge the rows fromstate_groups_statestate_groups, but then failed to remove the rows fromstate_groups_state.Logs from the first purge attempt would be useful in determining which of these is the problem.
richvdh commentedon Mar 17, 2020
closing this pending the information needed to investigate.
Ralith commentedon Mar 17, 2020
What recourse is available for cleaning up the millions of orphaned rows?
richvdh commentedon Jul 9, 2020
@Ralith : are the orphaned rows in question in (a)
state_groupsor (b)state_groups_state?For case (a), https://github.com/erikjohnston/synapse-find-unreferenced-state-groups is a thing that helps find unreferenced state groups and clear them out.
For case (b) (the rows have gone from
state_groupsbut there are still orphaned rows instate_groups_state), you might be able to build a list of such rows with aSELECTquery and then run a series ofDELETEs.Ralith commentedon Jul 9, 2020
I have rows for both ostensibly-purged rooms in
state_groups, but a comparatively trivial number: only 365 for!PcCidzjUKXHKImhTrB:matrix.org, and 1531 for'!QtykxKocfZaZOUrTwp:matrix.org'. I'll try Erik's tool, thanks! From the warning, I gather it's safe to run deletes for orphaned rooms, or when synapse is shut down, but generally not for live rooms when synapse is running?richvdh commentedon Jul 9, 2020
yes.
rettichschnidi commentedon Jun 29, 2021
For all victims of the spam wave in «Matrix HQ»: Using synapse-find-unreferenced-state-groups (still) works!
After closing down Matrix HQ (!OGEhHVWSdvArJzumhm:matrix.org) using Synapse Admin on my homeserver, I was left with 2.5M rows for this room in the state_groups_state stable. Using synapse-find-unreferenced-state-groups deleted all of those. Did not shut down Synapse while doing so, as the room got deleted before already.
lnicola commentedon May 20, 2022
I'm seeing about 700K rows in
current_state_delta_streamand more than 40M rows instate_groups_statefor rooms that should have been purged (ref. #12730). At some point purging failed because of a serialization error.Unfortunately,
synapse-find-unreferenced-state-groupsdoesn't seem to work. I can delete the rows manually, but I don't know exactly what tables I should be looking at.