This repository was archived by the owner on Apr 26, 2024. It is now read-only.
Repository navigation
This repository was archived by the owner on Apr 26, 2024. It is now read-only.
"Failed to calculate hosts in room" error from outbound federation #12509
Description
Presumably introduced by #12191. Suspect this means that outbound federation is broken in some rooms? Or it might be transient?
2022-04-20 00:27:42,101 - synapse.federation.sender - 392 - ERROR - process_event_queue_for_federation-6803872 - Failed to calculate hosts in room for event: <event_id>
Capture point (most recent call last):
File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/synapse/src/synapse/app/federation_sender.py", line 22, in <module>
start(sys.argv[1:])
File "/home/synapse/src/synapse/app/generic_worker.py", line 505, in start
_base.start_worker_reactor("synapse-generic-worker", config)
File "/home/synapse/src/synapse/app/_base.py", line 125, in start_worker_reactor
run_command=run_command,
File "/home/synapse/src/synapse/app/_base.py", line 180, in start_reactor
run()
File "/home/synapse/src/synapse/app/_base.py", line 162, in run
run_command()
File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/base.py", line 1315, in run
self.mainLoop()
File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/base.py", line 1325, in mainLoop
reactorBaseSelf.runUntilCurrent()
File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/base.py", line 964, in runUntilCurrent
f(*a, **kw)
File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 661, in callback
self._startRunCallbacks(result)
File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 763, in _startRunCallbacks
self._runCallbacks()
File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 858, in _runCallbacks
current.result, *args, **kwargs
File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 1750, in gotResult
current_context.run(_inlineCallbacks, r, gen, status)
File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 1663, in _inlineCallbacks
status.deferred.callback(getattr(e, "value", None))
File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 661, in callback
self._startRunCallbacks(result)
File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 763, in _startRunCallbacks
self._runCallbacks()
File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 858, in _runCallbacks
current.result, *args, **kwargs
File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 661, in callback
self._startRunCallbacks(result)
File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 763, in _startRunCallbacks
self._runCallbacks()
File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 858, in _runCallbacks
current.result, *args, **kwargs
File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 1750, in gotResult
current_context.run(_inlineCallbacks, r, gen, status)
File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 1740, in _inlineCallbacks
status.deferred.errback()
File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 700, in errback
self._startRunCallbacks(fail)
File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 763, in _startRunCallbacks
self._runCallbacks()
File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 858, in _runCallbacks
current.result, *args, **kwargs
File "/home/synapse/src/synapse/util/caches/descriptors.py", line 505, in errback_all
d1.errback(f)
File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 700, in errback
self._startRunCallbacks(fail)
File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 763, in _startRunCallbacks
self._runCallbacks()
File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 858, in _runCallbacks
current.result, *args, **kwargs
File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 1750, in gotResult
current_context.run(_inlineCallbacks, r, gen, status)
File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 1657, in _inlineCallbacks
cast(Failure, result).throwExceptionIntoGenerator, gen
File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/python/failure.py", line 489, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
Traceback (most recent call last):
File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
result = current_context.run(gen.send, result)
StopIteration: []
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/synapse/src/synapse/federation/sender/__init__.py", line 387, in handle_event
event.room_id, event_ids=event.prev_event_ids()
File "/home/synapse/src/synapse/state/__init__.py", line 257, in get_hosts_in_room_at_events
entry = await self.resolve_state_groups_for_events(room_id, event_ids)
File "/home/synapse/src/synapse/util/metrics.py", line 106, in measured_func
r = await func(self, *args, **kwargs)
File "/home/synapse/src/synapse/state/__init__.py", line 425, in resolve_state_groups_for_events
room_id, event_ids
File "/home/synapse/src/synapse/storage/state.py", line 582, in get_state_groups_ids
event_to_groups = await self.stores.main._get_state_group_for_events(event_ids)
File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
result = current_context.run(gen.send, result)
File "/home/synapse/src/synapse/storage/databases/main/state.py", line 331, in _get_state_group_for_events
raise RuntimeError("No state group for unknown or outlier event %s" % e)
RuntimeError: No state group for unknown or outlier event <different event id>
Activity
richvdh commentedon Apr 20, 2022
See https://sentry.matrix.org/sentry/synapse-matrixorg/issues/246739
reivilibre commentedon Apr 21, 2022
If we suspect this is breaking outbound federation for some rooms, it seems like we might want to investigate this one urgently.
Ezwen commentedon Apr 21, 2022
I shared a slightly similar error log in this other issue (more precisely, the message is similar, even though the stack is different): #12507 (comment)
richvdh commentedon Apr 27, 2022
Ok, so this is being caused by:
Neither of these are meant to be sent over federation, so the exception does not reflect a functional problem. However, we clearly need to do something about the log spam.
Exclude OOB memberships from the federation sender
Exclude OOB memberships from the federation sender (#12570)