This repository was archived by the owner on Apr 26, 2024. It is now read-only.
Repository navigation
This repository was archived by the owner on Apr 26, 2024. It is now read-only.
Rate Limiter: RecursionError: maximum recursion depth exceeded #14480
Description
This can't be good (observed in the federation-reader logs on libera.chat):
2022-11-17 10:48:27,448 - twisted - 274 - CRITICAL - sentinel- Unhandled error in Deferred:
2022-11-17 10:48:27,474 - twisted - 274 - CRITICAL - sentinel-
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/synapse/http/server.py", line 307, in _async_render_wrapper
callback_return = await self._async_render(request)
File "/usr/local/lib/python3.9/site-packages/synapse/http/server.py", line 513, in _async_render
callback_return = await raw_callback_return
File "/usr/local/lib/python3.9/site-packages/synapse/federation/transport/server/_base.py", line 342, in new_func
await d
File "/usr/local/lib/python3.9/site-packages/twisted/internet/defer.py", line 993, in send
raise result.value
synapse.api.errors.LimitExceededError: 429: Too Many Requests
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/synapse/util/ratelimitutils.py", line 249, in ratelimit
yield ret
File "/usr/local/lib/python3.9/site-packages/synapse/federation/transport/server/_base.py", line 344, in new_func
logger.warning(
File "/usr/local/lib/python3.9/logging/__init__.py", line 1458, in warning
self._log(WARNING, msg, args, **kwargs)
File "/usr/local/lib/python3.9/logging/__init__.py", line 1577, in _log
fn, lno, func, sinfo = self.findCaller(stack_info, stacklevel)
File "/usr/local/lib/python3.9/logging/__init__.py", line 1519, in findCaller
f = currentframe()
File "/usr/local/lib/python3.9/logging/__init__.py", line 160, in <lambda>
currentframe = lambda: sys._getframe(3)
RecursionError: maximum recursion depth exceeded while calling a Python object
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/synapse/http/server.py", line 307, in _async_render_wrapper
callback_return = await self._async_render(request)
File "/usr/local/lib/python3.9/site-packages/synapse/http/server.py", line 513, in _async_render
callback_return = await raw_callback_return
File "/usr/local/lib/python3.9/site-packages/synapse/federation/transport/server/_base.py", line 349, in new_func
response = await func(
File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.9/site-packages/synapse/util/ratelimitutils.py", line 251, in ratelimit
self._on_exit(request_id)
File "/usr/local/lib/python3.9/site-packages/synapse/util/ratelimitutils.py", line 366, in _on_exit
logger.debug("Ratelimit(%s) [%s]: Processed req", self.host, id(request_id))
RecursionError: maximum recursion depth exceeded while calling a Python object
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/synapse/http/site.py", line 309, in processing
yield
File "/usr/local/lib/python3.9/site-packages/synapse/http/server.py", line 229, in wrapped_async_request_handler
await h(self, request)
File "/usr/local/lib/python3.9/site-packages/synapse/http/server.py", line 317, in _async_render_wrapper
self._send_error_response(f, request)
File "/usr/local/lib/python3.9/site-packages/synapse/http/server.py", line 525, in _send_error_response
return_json_error(f, request, self.hs.config)
File "/usr/local/lib/python3.9/site-packages/synapse/http/server.py", line 124, in return_json_error
logger.error(
File "/usr/local/lib/python3.9/logging/__init__.py", line 1475, in error
self._log(ERROR, msg, args, **kwargs)
File "/usr/local/lib/python3.9/logging/__init__.py", line 1577, in _log
fn, lno, func, sinfo = self.findCaller(stack_info, stacklevel)
File "/usr/local/lib/python3.9/logging/__init__.py", line 1519, in findCaller
f = currentframe()
File "/usr/local/lib/python3.9/logging/__init__.py", line 160, in <lambda>
currentframe = lambda: sys._getframe(3)
RecursionError: maximum recursion depth exceeded while calling a Python object
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/synapse/http/site.py", line 312, in processing
logger.exception(
File "/usr/local/lib/python3.9/logging/__init__.py", line 1481, in exception
self.error(msg, *args, exc_info=exc_info, **kwargs)
File "/usr/local/lib/python3.9/logging/__init__.py", line 1475, in error
self._log(ERROR, msg, args, **kwargs)
File "/usr/local/lib/python3.9/logging/__init__.py", line 1577, in _log
fn, lno, func, sinfo = self.findCaller(stack_info, stacklevel)
File "/usr/local/lib/python3.9/logging/__init__.py", line 1519, in findCaller
f = currentframe()
RecursionError: maximum recursion depth exceeded
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/twisted/internet/defer.py", line 1696, in _inlineCallbacks
result = context.run(gen.send, result)
File "/usr/local/lib/python3.9/site-packages/synapse/http/server.py", line 229, in wrapped_async_request_handler
await h(self, request)
File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.9/site-packages/synapse/http/site.py", line 328, in processing
self._finished_processing()
File "/usr/local/lib/python3.9/site-packages/synapse/http/site.py", line 428, in _finished_processing
usage = self.logcontext.get_resource_usage()
File "/usr/local/lib/python3.9/site-packages/synapse/logging/context.py", line 481, in get_resource_usage
res = self._resource_usage.copy()
File "/usr/local/lib/python3.9/site-packages/synapse/logging/context.py", line 136, in copy
return ContextResourceUsage(copy_from=self)
RecursionError: maximum recursion depth exceeded
Activity
richvdh commentedon Nov 17, 2022
There's some very weird stuff going on with the ratelimiter on this process.
It seems like lots of requests stack up, and then all suddenly get released at once:
Some requests are going missing altogether:
(note that it is never responded to)
Also, er:
(that's actually a different worker. federation-reader-1, from which the above logs are taken, also shows an increase, though less extreme)
turt2live commentedon Nov 17, 2022
re: the graph: a few servers in the wild have been reporting backlogs in matrix.org->them traffic (their server falls behind) - is it possible something happened in the wild, or synapse changed something to do with transaction handling?
progval commentedon Nov 25, 2022
Sorry if this is off-topic, but could https://status.matrix.org/ be updated to reflect this issue?
DMRobertson commentedon Nov 28, 2022
I hadn't realised we also tracked the status of the libera bridge there. Status page updated.
DMRobertson commentedon Dec 1, 2022
Libera seems happier these days. I'll leave this issue open in case we see this failure mode again.
[-]`RecursionError: maximum recursion depth exceeded`[/-][+]Rate Limiter: `RecursionError: maximum recursion depth exceeded`[/+]squahtx commentedon Dec 20, 2022
This is basically #2532 but for the per-host rate limiters.
The federation rate limiter is per-host and its usage looks like
synapse/synapse/federation/transport/server/_base.py
Lines 341 to 352 in d8cc86e
await dblocks until the request is allowed to start. When the requesting homeserver gives up before the rate limiter allows a request to start,client disconnected before we started processing requestis printed, and the rate limiter context manager is exited synchronously.When the rate limiter context manager is __exit__ed, the next request is allowed to continue execution out of
await dwith the previous request still above it on the stack.synapse/synapse/util/ratelimitutils.py
Lines 365 to 375 in d8cc86e
If the requesting homeserver has also given up on the next request, the next request after that is resumed, and so on. Each level of abandoned request adds 17 stack frames*. At around 50 abandoned requests, that comes out to 850 stack frames, which is close enough to Python's default limit that
RecursionErrors are likely.* determined through experiment
Experiment
scripts-dev/federation_client.pyto sign a requestsquahtx commentedon Dec 20, 2022
In short, this failure mode happens when close to 50 (all) of the requests in the rate limiter queue have been abandoned.
H-Shay commentedon Jan 9, 2023
Is there a way to clear the abandoned requests from the stack?
squahtx commentedon Jan 9, 2023
Yep, we can either
with PreserveLoggingContext(): ...hs.get_reactor().callFromThread()thewith PreserveLoggingContext(): ...both of which resume the next request later.
The first option adds another reactor tick to the response time, which I don't like.
The second option needs a tiny bit of refactoring to thread
hsor the reactor through intoPerHostRateLimiter.Fix stack overflow in `_PerHostRatelimiter` due to synchronous requests
Fix stack overflow in `_PerHostRatelimiter` due to synchronous reques…