This repository was archived by the owner on Apr 26, 2024. It is now read-only.
Repository navigation
This repository was archived by the owner on Apr 26, 2024. It is now read-only.
Forward extremity event missing from database, leading to No state group for unknown or outlier event #12507
Description
Description
After upgrading to Synapse v1.57.0, one direct room is not working anymore. Cannot post new messages from any client or member.
Cannot roll back history more than the screen, then its stuck.
Cannot leave room "Internal Server Error 500"
Steps to reproduce
Tried rebooting the server, even tried to remove the room with Synapse-Admin - still get an error, room cannot be deleted.
Version information
- Homeserver:
If not matrix.org:
matrix.flyar.net
-
Version:
{"server_version":"1.57.0","python_version":"3.9.2"} -
Install method:
Debian repo
- Platform:
Debian 11.3 VM x64
This is how the logfile looks like:
2022-04-20 10:03:34,130 - synapse.http.server - 100 - ERROR - GET-0- Failed handle request via 'RoomMessageListRestServlet': <XForwardedForRequest at 0x7f28d07d3730 method>
Traceback (most recent call last):
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
result = current_context.run(gen.send, result)
StopIteration: [{'event_id': '$1650209062194EQOlf:matrix.flyar.net', 'state_group': 1277497}]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
result = current_context.run(gen.send, result)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/state.py", line 331, in _get_state_group_for_events
raise RuntimeError("No state group for unknown or outlier event %s" % e)
RuntimeError: No state group for unknown or outlier event $1638282339138PTbRx:matrix.flyar.net
2022-04-20 10:03:34,508 - synapse.http.server - 100 - ERROR - GET-1- Failed handle request via 'RoomMessageListRestServlet': <XForwardedForRequest at 0x7f28d07e3be0 method>
Traceback (most recent call last):
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
result = current_context.run(gen.send, result)
StopIteration: [{'event_id': '$1650209062194EQOlf:matrix.flyar.net', 'state_group': 1277497}]
The part after "During handling" is repeated. It seems one event is blocking everything? How can I get rid of this?
Activity
[-]Direct room broken - [/-][+]Direct room broken - Failed handle request via 'RoomMessageListRestServlet'[/+]DMRobertson commentedon Apr 20, 2022
Error message is from #12191. @richvdh, one for you?
[-]Direct room broken - Failed handle request via 'RoomMessageListRestServlet'[/-][+]Direct room broken - `No state group for unknown or outlier event`[/+]richvdh commentedon Apr 20, 2022
Possibly, but there's not really enough info in those logs to help. @esackbauer: is that the complete log? there is nothing logged at INFO?
esackbauer commentedon Apr 20, 2022
[edited by @richvdh to fix formatting]
There is actually nothing more. the room which is not working is
!DJCGiBKsgowqkkhkfS:matrix.flyar.netrichvdh commentedon Apr 20, 2022
[please wrap your logs in triple-backticks (```) for legibility]
well, that's frustrating. Could you maybe enable DEBUG logging (change the
levelto DEBUG in your log config file)?esackbauer commentedon Apr 20, 2022
Ok, sorry. The debug output is rather large, I did a clean start of synapse and logged everything after I tried to post something in that room.
Because its so large, I attached the homeserver.log
richvdh commentedon Apr 20, 2022
Right, well, that sheds a bit of light on the situation, but not much.
I'd like to ask you to run some queries on your database - could you DM me at
@richvdh:sw1v.org?esackbauer commentedon Apr 20, 2022
richvdh commentedon Apr 20, 2022
esackbauer commentedon Apr 20, 2022
richvdh commentedon Apr 20, 2022
It seems that a particular event (
$1638282339138PTbRx:matrix.flyar.net) is marked as a forward extremity, but that event is not present in the database.Attempts to delete the room are failing, because the delete operation tries to generate leave events for each of the members of the room - which requires the forward extremities to exist.
The event id suggests that the event was created in November 2021, so this problem has probably been waiting undetected since then, and it only became a real problem when Synapse 1.57 added the extra validation.
I'm at a bit of a loss to explain how this could have happened. There's essentially no way for an entry to be added to
event_forward_extremiteswithout there also being an entry inevents. It could have been related to #11521, but since this room isn't federated, I don't think it can be.[-]Direct room broken - `No state group for unknown or outlier event`[/-][+]Forward extremity event missing from database, leading to `No state group for unknown or outlier event`[/+]16 remaining items
travisghansen commentedon Apr 25, 2022
My install is fully isolated currently.
I've never enabled that, so I'm guessing the default is no. Glad to now know it exists however! That's partly why I wasn't federating anything as I knew the data would grow massively on public rooms etc :)
No.
No.
richvdh commentedon Apr 26, 2022
@MparkG: as the earlier comments make very clear: "If it doesn't [return any rows] you likely have a different problem and should open a separate issue".
richvdh commentedon Apr 26, 2022
@travisghansen thanks for the answers, though honestly they don't help me get any closer to understanding what can have caused this.
Be wary of retention: it's not a well-used feature and the reason I asked about it is that it is likely to cause corruption bugs like this.
mweinelt commentedon May 31, 2022
We're seeing exceptions like these as well and those rooms can't be written to. This is a room on the hackint IRC bridge, so lots of federated users, no usage of delete room or purge history API endpoints and we are using
rust-synapse-compress-stateregularly.We're on Synapse
1.58.01.59.1 right now.Feel free to contact me at
@hexa:lossy.networkand we can debug this together.Traceback
synapse.http.server: [PUT-672070] Failed handle request via 'RoomSendEventRestServlet': <XForwardedForRequest at 0x7fb87ab32a30 method='PUT' uri='/_matrix/client/r0/rooms/!lxUgQdCOpWnOrYNVxb%3Ahax404.de/send/m.room.message/1653997592776__inc2?user_id=%40hexa-%3Ahackint.org' clientproto='HTTP/1.1' site='8008'> Traceback (most recent call last): File "/nix/store/ks8r355nmkcx9q23s4m3nm1y767rfrln-python3.9-Twisted-22.4.0/lib/python3.9/site-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks result = current_context.run(gen.send, result) StopIteration: [{'event_id': '$Ps2Ot0YLz20kbjS8lRFmK_Yo4t2fXPAvq3IU0CQTMoU', 'state_group': 820535}] During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/nix/store/ks8r355nmkcx9q23s4m3nm1y767rfrln-python3.9-Twisted-22.4.0/lib/python3.9/site-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks result = current_context.run(gen.send, result) File "/nix/store/lan1pgij0701dzi9nsvvy6nn62qk93b5-matrix-synapse-1.58.0/lib/python3.9/site-packages/synapse/storage/databases/main/state.py", line 332, in _get_state_group_for_events raise RuntimeError("No state group for unknown or outlier event %s" % e) RuntimeError: No state group for unknown or outlier event $FXKIdG6LkVKVsOEROx4IBer-eqPsQ-HCWdWpRuqyRFkRetention
We are indeed using a pretty aggressive retention rule, since we only want to relay the bridged content, but not store it persistenly.
Query result
Not the event id mentioned in the traceback above, but I guess that's still worth debugging/fixing.
No corruption log lines
We don't see any corruption log events from #12620
Removed events according to #12507 (comment)
2660 - 7 = 2655 … okay. 🤔
richvdh commentedon Aug 8, 2022
@mweinelt: it is unsurprising to me that you would see these symptoms when retention is enabled, particularly with such aggressive numbers (see #13476). This issue is specific to people who do not have retention enabled, meaning there is no explanation for the missing
event_forward_extremities.14mRh4X0r commentedon Jan 24, 2023
I noticed it recently when restarting a bot.
Yes. The event is my own homeserver's though.
No.
Yes, only purge history; it has happened that Synapse was terminated during mass purging, it may have been while purging this room.
No.
squahtx commentedon Jan 25, 2023
@14mRh4X0r Could you post the logs for your issue?
14mRh4X0r commentedon Jan 25, 2023
Sure (Synapse 1.66.0 from Debian backports):
richvdh commentedon Jan 30, 2023
I'm going to assume @14mRh4X0r's issue is caused by their use of the history purge function and is hence a duplicate of #13476.
As far as I can tell from the history in this issue, everybody who has seen this (apart from @travisghansen) has, at some point in the past, used one of the history deletion functions. @travisghansen's symptoms remain unexplained, but in the absence of further reports, it's unlikely to be worth further investigation.
Accordingly, I'm closing this as a duplicate of #13476.