This repository was archived by the owner on Apr 26, 2024. It is now read-only.
Repository navigation
This repository was archived by the owner on Apr 26, 2024. It is now read-only.
Synchrotron stopped relaying new messages #7345
Description
I noticed today that my messages had stopped arriving after some time, but /sync was happily running as normal (no clientside errors). I checked the synchrotron and noticed that I was hitting the long poll timeout (30s) for my user pretty much reliably, and the sync token wasn't advancing.
And to be sure, I checked that my federation workers were all working, and they seemed to be processing incoming traffic just fine. I've also checked my certificates, and they are okay too.
2020-04-24 23:36:10,512 - synapse.state - 497 - INFO - GET-109649- Resolving state for !tdtPuWhSgVBBsTHMhG:matrix.org with 2 groups
2020-04-24 23:36:10,512 - synapse.state - 520 - INFO - GET-109649- Resolving conflicted state for '!tdtPuWhSgVBBsTHMhG:matrix.org'
2020-04-24 23:36:10,513 - synapse.state.v1 - 80 - INFO - GET-109649- Asking for 190/190 conflicted events
2020-04-24 23:36:10,522 - synapse.state - 497 - INFO - GET-109649- Resolving state for !FTFmvhuoHbAAzjYtZe:matrix.org with 8 groups
2020-04-24 23:36:10,522 - synapse.state - 520 - INFO - GET-109649- Resolving conflicted state for '!FTFmvhuoHbAAzjYtZe:matrix.org'
2020-04-24 23:36:10,527 - synapse.state.v1 - 80 - INFO - GET-109649- Asking for 447/447 conflicted events
2020-04-24 23:36:10,542 - synapse.metrics - 464 - INFO - - Collecting gc 1
2020-04-24 23:36:10,586 - synapse.state.v1 - 112 - INFO - GET-109649- Asking for 3/3 auth events
2020-04-24 23:36:10,676 - synapse.state.v1 - 112 - INFO - GET-109649- Asking for 4/4 auth events
2020-04-24 23:36:11,349 - synapse.access.http.8018 - 301 - INFO - GET-109649- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 1.083sec/0.000sec (0.165sec, 0.008sec) (0.078sec/2.126sec/57) 15178B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=30000&since=s36879948_272948375_38408_18234089_321641_1391_199653_1182298_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDr
oid; MatrixAndroidSDK_X 0.0.1)" [660 dbevts]
2020-04-24 23:36:15,103 - synapse.access.http.8018 - 301 - INFO - GET-109650- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.152sec/0.000sec (0.031sec, 0.001sec) (0.036sec/0.348sec/17) 14608B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879950_272948375_38420_18234097_321641_1391_199653_1182298_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15
Chrome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:36:15,374 - synapse.access.http.8018 - 301 - INFO - GET-109651- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.037sec/0.000sec (0.017sec, 0.000sec) (0.001sec/0.018sec/1) 1384B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38427_18234109_321641_1391_199653_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 C
hrome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:36:16,972 - synapse.access.http.8018 - 301 - INFO - GET-109653- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.059sec/0.000sec (0.019sec, 0.000sec) (0.001sec/0.036sec/2) 2707B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=30000&since=s36879969_272948375_38425_18234106_321641_1391_199653_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroi
d; MatrixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:36:18,871 - synapse.access.http.8018 - 301 - INFO - GET-109652- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 3.378sec/0.000sec (0.011sec, 0.000sec) (0.001sec/0.038sec/2) 559B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38428_18234112_321641_1391_199653_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 Ch
rome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:36:21,919 - synapse.access.http.8018 - 301 - INFO - GET-109654- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 2.983sec/0.000sec (0.007sec, 0.000sec) (0.000sec/0.000sec/0) 759B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38428_18234112_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 Ch
rome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:36:22,853 - synapse.access.http.8018 - 301 - INFO - GET-109655- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.860sec/0.000sec (0.012sec, 0.000sec) (0.001sec/0.018sec/1) 848B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38429_18234112_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 Ch
rome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:36:25,040 - synapse.access.http.8018 - 301 - INFO - GET-109656- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 2.045sec/0.000sec (0.010sec, 0.000sec) (0.000sec/0.000sec/0) 803B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38429_18234113_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 Ch
rome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:36:27,238 - synapse.access.http.8018 - 301 - INFO - GET-109657- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 2.120sec/0.000sec (0.041sec, 0.000sec) (0.001sec/0.045sec/2) 848B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38430_18234113_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 Ch
rome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:36:27,795 - synapse.access.http.8018 - 301 - INFO - GET-109658- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.476sec/0.000sec (0.014sec, 0.000sec) (0.000sec/0.018sec/1) 851B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38430_18234114_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 Ch
rome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:36:28,130 - synapse.access.http.8018 - 301 - INFO - GET-109659- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.252sec/0.000sec (0.017sec, 0.000sec) (0.007sec/0.018sec/1) 853B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38430_18234115_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 Ch
rome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:36:28,130 - synapse.metrics - 464 - INFO - - Collecting gc 1
2020-04-24 23:36:28,139 - synapse.metrics - 464 - INFO - - Collecting gc 2
2020-04-24 23:36:39,442 - synapse.access.http.8018 - 301 - INFO - GET-109661- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.145sec/0.000sec (0.034sec, 0.000sec) (0.001sec/0.105sec/4) 2394B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=0&since=s36879970_272948375_38428_18234112_321641_1391_199653_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroid; M
atrixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:36:58,590 - synapse.access.http.8018 - 301 - INFO - GET-109660- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.013sec/0.000sec (0.014sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 C
hrome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:37:10,582 - synapse.access.http.8018 - 301 - INFO - GET-109662- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.010sec/0.000sec (0.008sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroi
d; MatrixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:37:10,840 - synapse.access.http.8018 - 301 - INFO - GET-109664- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.012sec/0.000sec (0.009sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=0&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroid; Ma
trixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:37:28,678 - synapse.access.http.8018 - 301 - INFO - GET-109663- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.008sec/0.000sec (0.006sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 C
hrome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:37:41,678 - synapse.access.http.8018 - 301 - INFO - GET-109665- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.008sec/0.000sec (0.005sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroi
d; MatrixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:37:41,856 - synapse.access.http.8018 - 301 - INFO - GET-109667- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.007sec/0.000sec (0.004sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=0&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroid; Ma
trixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:37:58,751 - synapse.access.http.8018 - 301 - INFO - GET-109666- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.009sec/0.000sec (0.004sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 C
hrome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:38:12,332 - synapse.access.http.8018 - 301 - INFO - GET-109668- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.009sec/0.000sec (0.004sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroi
d; MatrixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:38:12,458 - synapse.access.http.8018 - 301 - INFO - GET-109670- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.008sec/0.000sec (0.008sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=0&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroid; Ma
trixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:38:20,666 - synapse.storage.data_stores.main.event_push_actions - 499 - INFO - event_push_action_stream_orderings-221- Searching for stream ordering 1 month ago
2020-04-24 23:38:20,866 - synapse.storage.data_stores.main.event_push_actions - 504 - INFO - event_push_action_stream_orderings-221- Found stream ordering 1 month ago: it's 36262982
2020-04-24 23:38:20,867 - synapse.storage.data_stores.main.event_push_actions - 506 - INFO - event_push_action_stream_orderings-221- Searching for stream ordering 1 day ago
2020-04-24 23:38:21,025 - synapse.storage.data_stores.main.event_push_actions - 511 - INFO - event_push_action_stream_orderings-221- Found stream ordering 1 day ago: it's 36857827
2020-04-24 23:38:28,839 - synapse.access.http.8018 - 301 - INFO - GET-109669- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.012sec/0.000sec (0.008sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 C
hrome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:38:42,594 - synapse.access.http.8018 - 301 - INFO - GET-109671- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.008sec/0.000sec (0.005sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroi
d; MatrixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:38:42,781 - synapse.access.http.8018 - 301 - INFO - GET-109673- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.007sec/0.000sec (0.007sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=0&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroid; Ma
trixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:38:58,920 - synapse.access.http.8018 - 301 - INFO - GET-109672- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.013sec/0.000sec (0.008sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 C
hrome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:39:13,273 - synapse.access.http.8018 - 301 - INFO - GET-109674- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.009sec/0.000sec (0.008sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroi
d; MatrixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:39:13,487 - synapse.access.http.8018 - 301 - INFO - GET-109676- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.013sec/0.000sec (0.011sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=0&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroid; Ma
trixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:39:29,040 - synapse.access.http.8018 - 301 - INFO - GET-109675- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.016sec/0.000sec (0.013sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 C
hrome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:39:43,976 - synapse.access.http.8018 - 301 - INFO - GET-109677- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.008sec/0.000sec (0.003sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroid; MatrixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:39:44,237 - synapse.access.http.8018 - 301 - INFO - GET-109679- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.007sec/0.000sec (0.007sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=0&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroid; MatrixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:39:59,121 - synapse.access.http.8018 - 301 - INFO - GET-109678- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.013sec/0.000sec (0.013sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 Chrome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:40:14,737 - synapse.access.http.8018 - 301 - INFO - GET-109680- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.014sec/0.000sec (0.011sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroid; MatrixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:40:29,227 - synapse.access.http.8018 - 301 - INFO - GET-109681- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.014sec/0.000sec (0.010sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 Chrome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:40:33,459 - synapse.access.http.8018 - 301 - INFO - GET-109683- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.013sec/0.000sec (0.011sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=0&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroid; MatrixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:40:59,315 - synapse.access.http.8018 - 301 - INFO - GET-109682- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.010sec/0.000sec (0.005sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 Chrome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:41:03,993 - synapse.access.http.8018 - 301 - INFO - GET-109684- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.015sec/0.000sec (0.010sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroid; MatrixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:41:29,497 - synapse.access.http.8018 - 301 - INFO - GET-109685- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.009sec/0.000sec (0.005sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 C:
It has been going like this from 23:38 last night up until now.
Activity
Half-Shot commentedon Apr 25, 2020
Restarting the worker seemed to get it running again.
Half-Shot commentedon Apr 25, 2020
Ah, and it quickly fell apart. The master process appears to be screaming a lot about:
babolivier commentedon Apr 25, 2020
This last error seems to be #7308, which should have been fixed by #7337 although apparently other
AssertionErrors can still happen despite that :/Half-Shot commentedon Apr 25, 2020
Oh, it's worth pointing out that I might not have been quite up to date...I'll try forcing myself to latest develop and see if that helps.
Half-Shot commentedon Apr 25, 2020
it did not
richvdh commentedon Apr 25, 2020
Yup there are still bugs here as per #7308. Half-Shot can you share a stacktrace with latest develop?
Half-Shot commentedon Apr 25, 2020
richvdh commentedon Apr 27, 2020
sorry @Half-Shot can you include the ERROR line above the traceback?
richvdh commentedon Apr 27, 2020
(this looks closely related to #7308 but a slightly different code path)
Half-Shot commentedon Apr 27, 2020
@richvdh my logs have since rotated but the traceback was cased by the error below it (it was looping the same error and traceback in the logs over and over).
I can try to run develop again and capture fresh logs, if useful.
richvdh commentedon Apr 27, 2020
ah don't worry. Will have a stare at the code and try and track it down.
richvdh commentedon Apr 27, 2020
I'm struggling to figure out which stream could be throwing that assertion. I'll add some better logging and then get you to try again.
richvdh commentedon Apr 27, 2020
Oh wait, it says:
but then... that makes no sense, because as of #7337, the events stream doesn't go via
tcp/streams/_base.py:178. @Half-Shot : are you sure that stack trace came from develop?Half-Shot commentedon Apr 28, 2020
fresh logs from develop
richvdh commentedon Apr 28, 2020
thanks, looks like a dup of #7308