Synchrotron stopped relaying new messages #7345

Half-Shot · 2020-04-25T15:28:27Z

I noticed today that my messages had stopped arriving after some time, but /sync was happily running as normal (no clientside errors). I checked the synchrotron and noticed that I was hitting the long poll timeout (30s) for my user pretty much reliably, and the sync token wasn't advancing.

And to be sure, I checked that my federation workers were all working, and they seemed to be processing incoming traffic just fine. I've also checked my certificates, and they are okay too.

2020-04-24 23:36:10,512 - synapse.state - 497 - INFO - GET-109649- Resolving state for !tdtPuWhSgVBBsTHMhG:matrix.org with 2 groups
2020-04-24 23:36:10,512 - synapse.state - 520 - INFO - GET-109649- Resolving conflicted state for '!tdtPuWhSgVBBsTHMhG:matrix.org'
2020-04-24 23:36:10,513 - synapse.state.v1 - 80 - INFO - GET-109649- Asking for 190/190 conflicted events
2020-04-24 23:36:10,522 - synapse.state - 497 - INFO - GET-109649- Resolving state for !FTFmvhuoHbAAzjYtZe:matrix.org with 8 groups
2020-04-24 23:36:10,522 - synapse.state - 520 - INFO - GET-109649- Resolving conflicted state for '!FTFmvhuoHbAAzjYtZe:matrix.org'
2020-04-24 23:36:10,527 - synapse.state.v1 - 80 - INFO - GET-109649- Asking for 447/447 conflicted events
2020-04-24 23:36:10,542 - synapse.metrics - 464 - INFO - - Collecting gc 1
2020-04-24 23:36:10,586 - synapse.state.v1 - 112 - INFO - GET-109649- Asking for 3/3 auth events
2020-04-24 23:36:10,676 - synapse.state.v1 - 112 - INFO - GET-109649- Asking for 4/4 auth events
2020-04-24 23:36:11,349 - synapse.access.http.8018 - 301 - INFO - GET-109649- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 1.083sec/0.000sec (0.165sec, 0.008sec) (0.078sec/2.126sec/57) 15178B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=30000&since=s36879948_272948375_38408_18234089_321641_1391_199653_1182298_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDr
oid; MatrixAndroidSDK_X 0.0.1)" [660 dbevts]
2020-04-24 23:36:15,103 - synapse.access.http.8018 - 301 - INFO - GET-109650- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.152sec/0.000sec (0.031sec, 0.001sec) (0.036sec/0.348sec/17) 14608B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879950_272948375_38420_18234097_321641_1391_199653_1182298_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15
 Chrome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:36:15,374 - synapse.access.http.8018 - 301 - INFO - GET-109651- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.037sec/0.000sec (0.017sec, 0.000sec) (0.001sec/0.018sec/1) 1384B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38427_18234109_321641_1391_199653_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 C
hrome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:36:16,972 - synapse.access.http.8018 - 301 - INFO - GET-109653- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.059sec/0.000sec (0.019sec, 0.000sec) (0.001sec/0.036sec/2) 2707B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=30000&since=s36879969_272948375_38425_18234106_321641_1391_199653_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroi
d; MatrixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:36:18,871 - synapse.access.http.8018 - 301 - INFO - GET-109652- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 3.378sec/0.000sec (0.011sec, 0.000sec) (0.001sec/0.038sec/2) 559B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38428_18234112_321641_1391_199653_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 Ch
rome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:36:21,919 - synapse.access.http.8018 - 301 - INFO - GET-109654- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 2.983sec/0.000sec (0.007sec, 0.000sec) (0.000sec/0.000sec/0) 759B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38428_18234112_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 Ch
rome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:36:22,853 - synapse.access.http.8018 - 301 - INFO - GET-109655- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.860sec/0.000sec (0.012sec, 0.000sec) (0.001sec/0.018sec/1) 848B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38429_18234112_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 Ch
rome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:36:25,040 - synapse.access.http.8018 - 301 - INFO - GET-109656- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 2.045sec/0.000sec (0.010sec, 0.000sec) (0.000sec/0.000sec/0) 803B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38429_18234113_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 Ch
rome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:36:27,238 - synapse.access.http.8018 - 301 - INFO - GET-109657- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 2.120sec/0.000sec (0.041sec, 0.000sec) (0.001sec/0.045sec/2) 848B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38430_18234113_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 Ch
rome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:36:27,795 - synapse.access.http.8018 - 301 - INFO - GET-109658- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.476sec/0.000sec (0.014sec, 0.000sec) (0.000sec/0.018sec/1) 851B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38430_18234114_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 Ch
rome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:36:28,130 - synapse.access.http.8018 - 301 - INFO - GET-109659- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.252sec/0.000sec (0.017sec, 0.000sec) (0.007sec/0.018sec/1) 853B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38430_18234115_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 Ch
rome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:36:28,130 - synapse.metrics - 464 - INFO - - Collecting gc 1
2020-04-24 23:36:28,139 - synapse.metrics - 464 - INFO - - Collecting gc 2
2020-04-24 23:36:39,442 - synapse.access.http.8018 - 301 - INFO - GET-109661- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.145sec/0.000sec (0.034sec, 0.000sec) (0.001sec/0.105sec/4) 2394B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=0&since=s36879970_272948375_38428_18234112_321641_1391_199653_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroid; M
atrixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:36:58,590 - synapse.access.http.8018 - 301 - INFO - GET-109660- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.013sec/0.000sec (0.014sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 C
hrome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:37:10,582 - synapse.access.http.8018 - 301 - INFO - GET-109662- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.010sec/0.000sec (0.008sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroi
d; MatrixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:37:10,840 - synapse.access.http.8018 - 301 - INFO - GET-109664- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.012sec/0.000sec (0.009sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=0&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroid; Ma
trixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:37:28,678 - synapse.access.http.8018 - 301 - INFO - GET-109663- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.008sec/0.000sec (0.006sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 C
hrome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:37:41,678 - synapse.access.http.8018 - 301 - INFO - GET-109665- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.008sec/0.000sec (0.005sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroi
d; MatrixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:37:41,856 - synapse.access.http.8018 - 301 - INFO - GET-109667- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.007sec/0.000sec (0.004sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=0&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroid; Ma
trixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:37:58,751 - synapse.access.http.8018 - 301 - INFO - GET-109666- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.009sec/0.000sec (0.004sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 C
hrome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:38:12,332 - synapse.access.http.8018 - 301 - INFO - GET-109668- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.009sec/0.000sec (0.004sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroi
d; MatrixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:38:12,458 - synapse.access.http.8018 - 301 - INFO - GET-109670- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.008sec/0.000sec (0.008sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=0&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroid; Ma
trixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:38:20,666 - synapse.storage.data_stores.main.event_push_actions - 499 - INFO - event_push_action_stream_orderings-221- Searching for stream ordering 1 month ago
2020-04-24 23:38:20,866 - synapse.storage.data_stores.main.event_push_actions - 504 - INFO - event_push_action_stream_orderings-221- Found stream ordering 1 month ago: it's 36262982
2020-04-24 23:38:20,867 - synapse.storage.data_stores.main.event_push_actions - 506 - INFO - event_push_action_stream_orderings-221- Searching for stream ordering 1 day ago
2020-04-24 23:38:21,025 - synapse.storage.data_stores.main.event_push_actions - 511 - INFO - event_push_action_stream_orderings-221- Found stream ordering 1 day ago: it's 36857827
2020-04-24 23:38:28,839 - synapse.access.http.8018 - 301 - INFO - GET-109669- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.012sec/0.000sec (0.008sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 C
hrome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:38:42,594 - synapse.access.http.8018 - 301 - INFO - GET-109671- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.008sec/0.000sec (0.005sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroi
d; MatrixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:38:42,781 - synapse.access.http.8018 - 301 - INFO - GET-109673- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.007sec/0.000sec (0.007sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=0&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroid; Ma
trixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:38:58,920 - synapse.access.http.8018 - 301 - INFO - GET-109672- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.013sec/0.000sec (0.008sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 C
hrome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:39:13,273 - synapse.access.http.8018 - 301 - INFO - GET-109674- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.009sec/0.000sec (0.008sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroi
d; MatrixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:39:13,487 - synapse.access.http.8018 - 301 - INFO - GET-109676- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.013sec/0.000sec (0.011sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=0&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroid; Ma
trixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:39:29,040 - synapse.access.http.8018 - 301 - INFO - GET-109675- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.016sec/0.000sec (0.013sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 C
hrome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:39:43,976 - synapse.access.http.8018 - 301 - INFO - GET-109677- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.008sec/0.000sec (0.003sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroid; MatrixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:39:44,237 - synapse.access.http.8018 - 301 - INFO - GET-109679- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.007sec/0.000sec (0.007sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=0&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroid; MatrixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:39:59,121 - synapse.access.http.8018 - 301 - INFO - GET-109678- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.013sec/0.000sec (0.013sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 Chrome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:40:14,737 - synapse.access.http.8018 - 301 - INFO - GET-109680- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.014sec/0.000sec (0.011sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroid; MatrixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:40:29,227 - synapse.access.http.8018 - 301 - INFO - GET-109681- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.014sec/0.000sec (0.010sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 Chrome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:40:33,459 - synapse.access.http.8018 - 301 - INFO - GET-109683- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.013sec/0.000sec (0.011sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=0&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroid; MatrixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:40:59,315 - synapse.access.http.8018 - 301 - INFO - GET-109682- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.010sec/0.000sec (0.005sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 Chrome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:41:03,993 - synapse.access.http.8018 - 301 - INFO - GET-109684- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.015sec/0.000sec (0.010sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroid; MatrixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:41:29,497 - synapse.access.http.8018 - 301 - INFO - GET-109685- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.009sec/0.000sec (0.005sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 C:

It has been going like this from 23:38 last night up until now.

The text was updated successfully, but these errors were encountered:

Half-Shot · 2020-04-25T15:30:46Z

Restarting the worker seemed to get it running again.

Half-Shot · 2020-04-25T15:41:25Z

Ah, and it quickly fell apart. The master process appears to be screaming a lot about:

2020-04-25 15:14:28,809 - synapse.http.client - 318 - INFO - as-sender-105d7bcabb777ad734c4db8675bb5896c18beb4e1ea45d5c4a3a411e0b3a9f8e-3609- Received response to PUT http://localhost:12450/transactions/367266?access_token=<redacted>: 200
2020-04-25 15:14:28,816 - synapse.replication.tcp.resource - 169 - INFO - replication_notifier-137927- Failed to handle stream events
2020-04-25 15:14:28,816 - synapse.metrics.background_process_metrics - 215 - ERROR - replication_notifier-137927- Background process 'replication_notifier' threw an exception
Traceback (most recent call last):
  File "/opt/synapse/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: [(36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@mfaure:matrix.org', '$15449765203163589VTibv:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_htoc^:matrix.org', '$15498973977237451lRpuU:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@mike_new:mikenew.io', '$1549469192151SUawy:mikenew.io'), (36879971, '!QtykxKocfZaZOUrTwp:ma
trix.org', 'm.room.member', '@duncanturk:synapse.duncanturk.com', '$15471013481428EBVoH:synapse.duncanturk.com'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@aelequin:matrix.org', '$15628785501751584YcIrd:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@mj:matrix.ucw.cz', '$15609802944QuyZy:matrix.ucw.cz'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@ant
onov:halogen.city', '$1554751835268PiitW:halogen.city'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_yanosz:matrix.org', '$1539093345170475beZRk:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_dc_irc_bridge:matrix.org', '$15525394782757049ftBmX:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@snk:nobeta.org', '$155068652752ywF
CP:nobeta.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_ManyRaptors:matrix.org', '$15350844691475147zHalG:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Mutter:matrix.org', '$1550691114253095FNVVn:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_grin[m]:matrix.org', '$15605059211912460COSIn:matrix.org'), (368799
71, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_adru_:matrix.org', '$15352505244429584zhOZK:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_psychnosis:matrix.org', '$1536830715595391cnThL:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@midar:matrix.org', '$1568548157474580xeSJe:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 
'm.room.member', '@freenode_JonOsterman:matrix.org', '$15686366511803808kJqIO:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_sandra24xi:matrix.org', '$15734300293241205Jfoyj:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Solprefixer:matrix.org', '$1534422199123209SMpzq:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member',
 '@freenode_hernan604:matrix.org', '$1532138248298124ODYiD:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_KibaFox:matrix.org', '$1566108001467241LdsgV:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_eido1on:matrix.org', '$15344958761140681uGGit:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_graphene:matri
x.org', '$1558541094151819VxbRC:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_kalamaja_:matrix.org', '$15477765452033083isFOa:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@philip:trailingwhite.space', '$15494866193968UpZzl:trailingwhite.space'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Tobias928:matrix.org', '$15576200
355949126QEpNm:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@okapii:matrix.org', '$1551201211128322aSZRy:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@G4sD0c:matrix.org', '$1567959819445692FmmRg:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_jrl:matrix.org', '$1569107689949399LPFjv:matrix.org'), (36879971, '!QtykxKocfZa
ZOUrTwp:matrix.org', 'm.room.member', '@freenode_hilario:matrix.org', '$15713532531374805DaqUy:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_detran:matrix.org', '$1554145233336312vEaIl:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_dc_irc_bridge1:matrix.org', '$15492545341289686IfEEE:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 
'm.room.member', '@freenode_stee3:matrix.org', '$1532467670323396LFFAS:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_ks1en:matrix.org', '$15625500441686201DVVnf:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_TheBloodyMaster1:matrix.org', '$1569991165219560iORRp:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@phil
rw:matrix.rosenberg-watt.com', '$15605467771262PXbFR:matrix.rosenberg-watt.com'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_xvnvx-:matrix.org', '$15431988362961092DYzDF:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_clarjon_1:matrix.org', '$15346019861242697ILSKq:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@dylanrod:mat
rix.org', '$15456240461126230msBaS:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_tlaxkit:matrix.org', '$15400810221822899mSckR:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@simonpatapon:matrix.patapon.lol', '$15539913543qDcwy:matrix.patapon.lol'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_coumbole_:matrix.org', '$1541058276906845hFFny:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_georgie:matrix.org', '$1568427530489262EZhGh:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Whidou_:matrix.org', '$15691897691468494qKpew:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@kechpaja:matrix.org', '$1566287710370480bmjrE:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_hehehehe:matrix.org', '$1543016200866329zOqec:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@jm.strauss:matrix.org', '$1586684896564238FMSmG:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Morpheus_B:matrix.org', '$15297493881036032cVXQT:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@h90-:matrix.org', '$1536358558702889iAZYq:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_banc:matrix.org', '$15293441861636921FptLB:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@ml_mx_test:matrix.org', '$1546187749689590vpUvg:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@test.user.12381209:matrix.org', '$15426473271297140vUWDM:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Wouldnlite:matrix.org', '$1555452282355523RDYUv:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@fcoimbrar:matrix.org', '$1553765486366766bsxgz:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_djanos-:matrix.org', '$15539405272916163ydEmk:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@sixecho:matrix.org', '$157383207334247lbBPA:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@sameer.isa27:matrix.org', '$1549289269569101nfBme:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_i1nfusion:matrix.org', '$15575319025190317hqShz:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_malSet_:matrix.org', '$1569086482793534uBGYc:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@darksiders:darksiders.me', '$155697162144VhOXl:darksiders.me'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@felix:chat.matrix4me.de', '$15877713704mLoRT:chat.matrix4me.de'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_e0f:matrix.org', '$15431911342731002srBrv:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@eibhear:matrix.org', '$1552397095432828RHZJu:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Not_White_Rabbit:matrix.org', '$1536735890654282TIlad:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_jistr|mtgs:matrix.org', '$1571691219128941kaCjo:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@matroid:matrix.org', '$15573687911918058Pzeim:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Guest35:matrix.org', '$15516513584184196YITis:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_CcxWrk:matrix.org', '$15397982216810427UXule:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Vikulin:matrix.org', '$15360438862585714rleKj:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_fxrs:matrix.org', '$1534794797262796hDurT:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@avere:banter.city', '$15485076691016UauhC:banter.city'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_SoniEx2:matrix.org', '$15613568361503853LHrjP:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_rachelfi1h:matrix.org', '$155930606172089AWaha:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_evilem:matrix.org', '$15418771602244393VqNXX:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_tracyiot376:matrix.org', '$1536702779399191WlsRo:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Roybent:matrix.org', '$15420864544215478CMFWY:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_milkt:matrix.org', '$15635795131004581tOOsI:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@bjoern.schanz:matrix.org', '$155258946877500RNUvB:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_SocksPls:matrix.org', '$1539137650661188QRQPV:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_fredcy_:matrix.org', '$15615631251479604RhswL:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Sh|m0d4:matrix.org', '$15674761332336217xbGWQ:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@tristan:kretzer.it', '$15500931970TOMKd:kretzer.it'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_dabeowulf:matrix.org', '$1539142122711430kxUIi:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_zetorian:matrix.org', '$1547074379571416hkZeS:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@_x_half-shot=20=28xmpp=29=2f=23matrix=23matrix.org=40half-shot.uk:half-shot.uk', '$15724593328uHJKw:half-shot.uk'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@wtp2018:matrix.org', '$1530870816277887DMtQv:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@wuhlheide:matrix.org', '$15498960082623309fZzsf:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@benoit0816:matrix.org', '$1568888854213905dMMhw:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_lazydude[m]:matrix.org', '$15528197872685794mPayS:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@mujx:feneas.org', '$15550217676651QsrzQ:feneas.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@risc-j508:matrix.org', '$15687419811091673hBcYn:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_MorpheusB:matrix.org', '$15719022421162306hMajg:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@riegel:matrix.org', '$155529774083701xdEQj:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Notkea:matrix.org', '$15363462961872490iKoMd:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@condar:matrix.org', '$1562265305416808IglRw:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@anoboy666:matrix.org', '$155631056952733uDYYu:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@hawkowl:atleastfornow.net', '$15495418452tTgfA:atleastfornow.net'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_thunders1ore:matrix.org', '$1534422164122485rHvRm:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_crivrc:matrix.org', '$15706186421306928WYsfu:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_sparklefarkle:matrix.org', '$15359746391833656XPoBi:matrix.org')]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/synapse/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: [(36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@mfaure:matrix.org', '$15449765203163589VTibv:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_htoc^:matrix.org', '$15498973977237451lRpuU:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@mike_new:mikenew.io', '$1549469192151SUawy:mikenew.io'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@duncanturk:synapse.duncanturk.com', '$15471013481428EBVoH:synapse.duncanturk.com'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@aelequin:matrix.org', '$15628785501751584YcIrd:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@mj:matrix.ucw.cz', '$15609802944QuyZy:matrix.ucw.cz'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@antonov:halogen.city', '$1554751835268PiitW:halogen.city'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_yanosz:matrix.org', '$1539093345170475beZRk:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_dc_irc_bridge:matrix.org', '$15525394782757049ftBmX:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@snk:nobeta.org', '$155068652752ywFCP:nobeta.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_ManyRaptors:matrix.org', '$15350844691475147zHalG:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Mutter:matrix.org', '$1550691114253095FNVVn:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_grin[m]:matrix.org', '$15605059211912460COSIn:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_adru_:matrix.org', '$15352505244429584zhOZK:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_psychnosis:matrix.org', '$1536830715595391cnThL:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@midar:matrix.org', '$1568548157474580xeSJe:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_JonOsterman:matrix.org', '$15686366511803808kJqIO:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_sandra24xi:matrix.org', '$15734300293241205Jfoyj:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Solprefixer:matrix.org', '$1534422199123209SMpzq:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_hernan604:matrix.org', '$1532138248298124ODYiD:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_KibaFox:matrix.org', '$1566108001467241LdsgV:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_eido1on:matrix.org', '$15344958761140681uGGit:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_graphene:matrix.org', '$1558541094151819VxbRC:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_kalamaja_:matrix.org', '$15477765452033083isFOa:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@philip:trailingwhite.space', '$15494866193968UpZzl:trailingwhite.space'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Tobias928:matrix.org', '$15576200355949126QEpNm:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@okapii:matrix.org', '$1551201211128322aSZRy:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@G4sD0c:matrix.org', '$1567959819445692FmmRg:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_jrl:matrix.org', '$1569107689949399LPFjv:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_hilario:matrix.org', '$15713532531374805DaqUy:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_detran:matrix.org', '$1554145233336312vEaIl:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_dc_irc_bridge1:matrix.org', '$15492545341289686IfEEE:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_stee3:matrix.org', '$1532467670323396LFFAS:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_ks1en:matrix.org', '$15625500441686201DVVnf:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_TheBloodyMaster1:matrix.org', '$1569991165219560iORRp:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@philrw:matrix.rosenberg-watt.com', '$15605467771262PXbFR:matrix.rosenberg-watt.com'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_xvnvx-:matrix.org', '$15431988362961092DYzDF:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_clarjon_1:matrix.org', '$15346019861242697ILSKq:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@dylanrod:matrix.org', '$15456240461126230msBaS:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_tlaxkit:matrix.org', '$15400810221822899mSckR:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@simonpatapon:matrix.patapon.lol', '$15539913543qDcwy:matrix.patapon.lol'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_coumbole_:matrix.org', '$1541058276906845hFFny:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_georgie:matrix.org', '$1568427530489262EZhGh:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Whidou_:matrix.org', '$15691897691468494qKpew:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@kechpaja:matrix.org', '$1566287710370480bmjrE:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_hehehehe:matrix.org', '$1543016200866329zOqec:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@jm.strauss:matrix.org', '$1586684896564238FMSmG:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Morpheus_B:matrix.org', '$15297493881036032cVXQT:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@h90-:matrix.org', '$1536358558702889iAZYq:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_banc:matrix.org', '$15293441861636921FptLB:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@ml_mx_test:matrix.org', '$1546187749689590vpUvg:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@test.user.12381209:matrix.org', '$15426473271297140vUWDM:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Wouldnlite:matrix.org', '$1555452282355523RDYUv:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@fcoimbrar:matrix.org', '$1553765486366766bsxgz:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_djanos-:matrix.org', '$15539405272916163ydEmk:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@sixecho:matrix.org', '$157383207334247lbBPA:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@sameer.isa27:matrix.org', '$1549289269569101nfBme:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_i1nfusion:matrix.org', '$15575319025190317hqShz:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_malSet_:matrix.org', '$1569086482793534uBGYc:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@darksiders:darksiders.me', '$155697162144VhOXl:darksiders.me'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@felix:chat.matrix4me.de', '$15877713704mLoRT:chat.matrix4me.de'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_e0f:matrix.org', '$15431911342731002srBrv:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@eibhear:matrix.org', '$1552397095432828RHZJu:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Not_White_Rabbit:matrix.org', '$1536735890654282TIlad:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_jistr|mtgs:matrix.org', '$1571691219128941kaCjo:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@matroid:matrix.org', '$15573687911918058Pzeim:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Guest35:matrix.org', '$15516513584184196YITis:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_CcxWrk:matrix.org', '$15397982216810427UXule:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Vikulin:matrix.org', '$15360438862585714rleKj:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_fxrs:matrix.org', '$1534794797262796hDurT:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@avere:banter.city', '$15485076691016UauhC:banter.city'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_SoniEx2:matrix.org', '$15613568361503853LHrjP:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_rachelfi1h:matrix.org', '$155930606172089AWaha:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_evilem:matrix.org', '$15418771602244393VqNXX:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_tracyiot376:matrix.org', '$1536702779399191WlsRo:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Roybent:matrix.org', '$15420864544215478CMFWY:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_milkt:matrix.org', '$15635795131004581tOOsI:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@bjoern.schanz:matrix.org', '$155258946877500RNUvB:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_SocksPls:matrix.org', '$1539137650661188QRQPV:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_fredcy_:matrix.org', '$15615631251479604RhswL:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Sh|m0d4:matrix.org', '$15674761332336217xbGWQ:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@tristan:kretzer.it', '$15500931970TOMKd:kretzer.it'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_dabeowulf:matrix.org', '$1539142122711430kxUIi:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_zetorian:matrix.org', '$1547074379571416hkZeS:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@_x_half-shot=20=28xmpp=29=2f=23matrix=23matrix.org=40half-shot.uk:half-shot.uk', '$15724593328uHJKw:half-shot.uk'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@wtp2018:matrix.org', '$1530870816277887DMtQv:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@wuhlheide:matrix.org', '$15498960082623309fZzsf:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@benoit0816:matrix.org', '$1568888854213905dMMhw:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_lazydude[m]:matrix.org', '$15528197872685794mPayS:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@mujx:feneas.org', '$15550217676651QsrzQ:feneas.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@risc-j508:matrix.org', '$15687419811091673hBcYn:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_MorpheusB:matrix.org', '$15719022421162306hMajg:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@riegel:matrix.org', '$155529774083701xdEQj:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Notkea:matrix.org', '$15363462961872490iKoMd:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@condar:matrix.org', '$1562265305416808IglRw:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@anoboy666:matrix.org', '$155631056952733uDYYu:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@hawkowl:atleastfornow.net', '$15495418452tTgfA:atleastfornow.net'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_thunders1ore:matrix.org', '$1534422164122485rHvRm:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_crivrc:matrix.org', '$15706186421306928WYsfu:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_sparklefarkle:matrix.org', '$15359746391833656XPoBi:matrix.org')]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/synapse/lib/python3.7/site-packages/synapse/metrics/background_process_metrics.py", line 213, in run
    return (yield result)
  File "/opt/synapse/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/opt/synapse/lib/python3.7/site-packages/synapse/replication/tcp/resource.py", line 166, in _run_notifier_loop
    updates, current_token, limited = await stream.get_updates()
  File "/opt/synapse/lib/python3.7/site-packages/synapse/replication/tcp/streams/_base.py", line 134, in get_updates
    self.last_token, current_token
  File "/opt/synapse/lib/python3.7/site-packages/synapse/replication/tcp/streams/_base.py", line 159, in get_updates_since
    from_token, upto_token, limit,
  File "/opt/synapse/lib/python3.7/site-packages/synapse/replication/tcp/streams/_base.py", line 178, in update_function
    assert len(updates) <= limit
AssertionError
2020-04-25 15:14:28,858 - synapse.push.httppusher - 195 - INFO - httppush.process-2284- Processing 1 unprocessed push actions for @Half-Shot:half-shot.uk/im.vector.app.android/cw24so2GjJs:APA91bGnJYSh8nHSoHV28c-IHKZolYqc76SCbFkaDzgVYt0n2Li2IwQNqrMDC0i4ssX6_AdGnmxyd-ZQE_J-tErjwi8K1zrZ__C71U9b9lVQgrpZ-Win2gGrZCPoprLat1mPATyPfVX2 starting at stream_ordering 36888411

babolivier · 2020-04-25T17:15:42Z

This last error seems to be #7308, which should have been fixed by #7337 although apparently other AssertionErrors can still happen despite that :/

Half-Shot · 2020-04-25T18:42:05Z

Oh, it's worth pointing out that I might not have been quite up to date...I'll try forcing myself to latest develop and see if that helps.

Half-Shot · 2020-04-25T20:42:56Z

it did not

richvdh · 2020-04-25T21:32:54Z

Yup there are still bugs here as per #7308. Half-Shot can you share a stacktrace with latest develop?

Half-Shot · 2020-04-25T21:44:44Z

Traceback (most recent call last):
  File "/opt/synapse/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: [(36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_Pempri=2fdisroot=40chat.disroot.org:bau-ha.us', '$1572513494483ARvOY:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_newbie=2fdisroot=40chat.disroot.org:bau-ha.us', '$15797231134407DbccO:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_michelc:matrix.org', '$15774803232684446OXYIq:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_33kk=2fdisroot=40chat.disroot.org:bau-ha.us', '$1581691057869qwfyT:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_erbzn=20=2fdisroot=40chat.disroot.org:bau-ha.us', '$15694168238668ejgDi:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_golanv=2fdisroot=40chat.disroot.org:bau-ha.us', '$1572527774846TekpE:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_lukas=2fdisroot=40chat.disroot.org:bau-ha.us', '$15798981442818TWtxY:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_Neurognostic_:matrix.org', '$15816546711199735iVoNs:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_domayor7=2fdisroot=40chat.disroot.org:bau-ha.us', '$1581685121744fFtRX:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_AngryArt=2fdisroot=40chat.disroot.org:bau-ha.us', '$1572540843397QyBuz:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_wikwg9=2fdisroot=40chat.disroot.org:bau-ha.us', '$15715141501877vSqWP:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_baldeagle=2fdisroot=40chat.disroot.org:bau-ha.us', '$1581578706889GznWF:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_Manson=2fdisroot=40chat.disroot.org:bau-ha.us', '$1581683033702fSjRq:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_sintrenton=2fdisroot=40chat.disroot.org:bau-ha.us', '$1570022688514Mzcny:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_willpate=2fdisroot=40chat.disroot.org:bau-ha.us', '$1572532753269afhHl:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_bonber=2fdisroot=40chat.disroot.org:bau-ha.us', '$1570024587559htYII:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@xmpp__maryjane_disroot@chat.disroot.org:disroot.org', '$148320022659tPVvI:disroot.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@ako:disroot.org', '$14828968831161RIEmH:disroot.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_coldbuddy=2fdisroot=40chat.disroot.org:bau-ha.us', '$15797292574589ldplO:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_cc__:matrix.org', '$1581477965334072nLqHq:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_outloudvi[m]:matrix.org', '$1568414692354648ZjaxW:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_icMikaela:matrix.org', '$15527884382355807cglWS:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_hekanju=2fdisroot=40chat.disroot.org:bau-ha.us', '$15817564342148nRfWM:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@Ciblia:matrix.org', '$14875843601338755aoxIF:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_tulsi=2fdisroot=40chat.disroot.org:bau-ha.us', '$15597705089011kdknj:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_expresionvocaloid=2fdisroot=40chat.disroot.org:bau-ha.us', '$1581552021240JdAlH:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@xmpp__mupp-xmpp_disroot@chat.disroot.org:disroot.org', '$14817577361HHxIb:disroot.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_cc_:matrix.org', '$15795036653073557bazgf:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_ozwald1=2fdisroot=40chat.disroot.org:bau-ha.us', '$157848178218806BPZSC:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_basedude:matrix.org', '$1581981956206514NWSuC:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_jh=2fdisroot=40chat.disroot.org:bau-ha.us', '$15701727904496PzlpJ:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_userdebug=2fdisroot=40chat.disroot.org:bau-ha.us', '$156011843485naJvu:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_maryjane=2fdisroot=40chat.disroot.org:bau-ha.us', '$156215398215wRLpH:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_muntashir=2fdisroot=40chat.disroot.org:bau-ha.us', '$15736709891108iMkGh:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_oxo=2fdisroot=40chat.disroot.org:bau-ha.us', '$157776490011238ndfKx:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_mars=2fdisroot=40chat.disroot.org:bau-ha.us', '$15610171791521lDEws:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_anam_:matrix.org', '$15794610782848027gKObS:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_awal=2fdisroot=40chat.disroot.org:bau-ha.us', '$15619195565420gtHuw:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_clackety:matrix.org', '$1581981967206617EIcHL:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_coffeebird=2fdisroot=40chat.disroot.org:bau-ha.us', '$15799600881294bqUpd:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_learningc:matrix.org', '$1581982036207290rMdAj:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_amam5=2fdisroot=40chat.disroot.org:bau-ha.us', '$15819078625064wUKbr:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_techbolt:matrix.org', '$1581982171208302UeyVR:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_nly:matrix.org', '$1581982073207557qvsLO:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_linuxturtle=2fdisroot=40chat.disroot.org:bau-ha.us', '$15788870471004NioHg:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_anon003=2fdisroot=40chat.disroot.org:bau-ha.us', '$15649291498697huXxU:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_Nuts_:matrix.org', '$1581981924206165BasRn:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_ironmonkey=2fdisroot=40chat.disroot.org:bau-ha.us', '$15596783686136AMFRx:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_diego.m51=2fdisroot=40chat.disroot.org:bau-ha.us', '$15764786081764JPrLN:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_toktr=2fdisroot=40chat.disroot.org:bau-ha.us', '$15787462713806rdUEJ:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_anam1:matrix.org', '$15856066292028898SqoHp:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_intuxikated[m]:matrix.org', '$1581982011207014mcTRp:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_crayon=2fdisroot=40chat.disroot.org:bau-ha.us', '$15748640221597PeIWX:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_freelancehero=2fdisroot=40chat.disroot.org:bau-ha.us', '$15766942317764sfedM:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_steve-_-:matrix.org', '$1581982168208278hkrXe:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_gmar=2fdisroot=40chat.disroot.org:bau-ha.us', '$15648527237255EdYZr:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_shahab:matrix.org', '$1581982151208154eNyoR:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_dingen:matrix.org', '$1581981981206735ZuDaN:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_coffeecat=2fdisroot=40chat.disroot.org:bau-ha.us', '$1559930340460LzsPu:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_massimiliano:matrix.org', '$1581982050207399jPLNK:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_draxaris=2fdisroot=40chat.disroot.org:bau-ha.us', '$15792705211025ggfbc:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_insanity=2fdisroot=40chat.disroot.org:bau-ha.us', '$15764764951668zsoor:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_vigilian=2fdisroot=40chat.disroot.org:bau-ha.us', '$15818530323802whQfB:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_ajeremias:matrix.org', '$1581981945206392RveoE:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_azhar=2fdisroot=40chat.disroot.org:bau-ha.us', '$1570033096739sEVkN:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_skrzyp1:matrix.org', '$15856066262028868MwMuk:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_perflyst:matrix.org', '$1581982088207679SVFJp:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_bou_:matrix.org', '$1581981962206567sHRuC:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_naimishavasi=2fdisroot=40chat.disroot.org:bau-ha.us', '$15597542318481auPjG:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_=5bAlien=5d=2fdisroot=40chat.disroot.org:bau-ha.us', '$15827418482574PilQq:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_steve=2fdisroot=40chat.disroot.org:bau-ha.us', '$155989704923411BIxaS:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_gongzisun=2fdisroot=40chat.disroot.org:bau-ha.us', '$15754102044760MCAZn:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_lurch:matrix.org', '$1581982042207347taXPq:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_ricardog08=2fdisroot=40chat.disroot.org:bau-ha.us', '$1576731958466PFaJB:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_jvuitton=2fdisroot=40chat.disroot.org:bau-ha.us', '$15799569861208HKMYK:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_oier=2fdisroot=40chat.disroot.org:bau-ha.us', '$15797935136359PBfdS:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_atilano=2fdisroot=40chat.disroot.org:bau-ha.us', '$15799653941474xgemx:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_perflyst[x]:matrix.org', '$1581982090207694hnQPR:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_golanv[m]:matrix.org', '$1581981996206881vYiiJ:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_sum_one=2fdisroot=40chat.disroot.org:bau-ha.us', '$1572541412433TJxHz:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_mitfree[m]:matrix.org', '$1581982062207478jWarf:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_antilopa:matrix.org', '$15792581791680716dYvHn:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_andy=2fdisroot=40chat.disroot.org:bau-ha.us', '$15649236058575nbson:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_Sasha1=2fdisroot=40chat.disroot.org:bau-ha.us', '$15764646841487DrxLQ:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_TheAceOfHearts:matrix.org', '$1581981933206280UVbIt:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_tlaxkit:matrix.org', '$1568414704354840RNUAF:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_salonitanicus=2fdisroot=40chat.disroot.org:bau-ha.us', '$15793799921860JssEl:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_strypey=2fdisroot=40chat.disroot.org:bau-ha.us', '$1581669173452EWNlB:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_andresflamenco=2fdisroot=40chat.disroot.org:bau-ha.us', '$15818221063313CfDpM:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_adn=2fdisroot=40chat.disroot.org:bau-ha.us', '$15747491651878rTzes:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_meaz=2fdisroot=40chat.disroot.org:bau-ha.us', '$15638995173197JHscv:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_GyroW_:matrix.org', '$15856066302028921zeAAc:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_ignacio=2fdisroot=40chat.disroot.org:bau-ha.us', '$1576159219813QmNwQ:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_mikelpz_:matrix.org', '$15856066302028928hBcOk:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_ryzendeath=2fdisroot=40chat.disroot.org:bau-ha.us', '$15596287384626aTuJH:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_pinguingitkeyz=2fdisroot=40chat.disroot.org:bau-ha.us', '$15763980482945xJCAP:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_masnavan=2fdisroot=40chat.disroot.org:bau-ha.us', '$15647598165568WtmKA:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_aerro=2fdisroot=40chat.disroot.org:bau-ha.us', '$15817812352571RnloL:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_pempri=2fdisroot=40chat.disroot.org:bau-ha.us', '$15764762471660JALHF:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_oscar=2fdisroot=40chat.disroot.org:bau-ha.us', '$15648337076842BqDjP:bau-ha.us')]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/synapse/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: [(36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_Pempri=2fdisroot=40chat.disroot.org:bau-ha.us', '$1572513494483ARvOY:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_newbie=2fdisroot=40chat.disroot.org:bau-ha.us', '$15797231134407DbccO:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_michelc:matrix.org', '$15774803232684446OXYIq:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_33kk=2fdisroot=40chat.disroot.org:bau-ha.us', '$1581691057869qwfyT:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_erbzn=20=2fdisroot=40chat.disroot.org:bau-ha.us', '$15694168238668ejgDi:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_golanv=2fdisroot=40chat.disroot.org:bau-ha.us', '$1572527774846TekpE:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_lukas=2fdisroot=40chat.disroot.org:bau-ha.us', '$15798981442818TWtxY:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_Neurognostic_:matrix.org', '$15816546711199735iVoNs:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_domayor7=2fdisroot=40chat.disroot.org:bau-ha.us', '$1581685121744fFtRX:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_AngryArt=2fdisroot=40chat.disroot.org:bau-ha.us', '$1572540843397QyBuz:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_wikwg9=2fdisroot=40chat.disroot.org:bau-ha.us', '$15715141501877vSqWP:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_baldeagle=2fdisroot=40chat.disroot.org:bau-ha.us', '$1581578706889GznWF:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_Manson=2fdisroot=40chat.disroot.org:bau-ha.us', '$1581683033702fSjRq:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_sintrenton=2fdisroot=40chat.disroot.org:bau-ha.us', '$1570022688514Mzcny:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_willpate=2fdisroot=40chat.disroot.org:bau-ha.us', '$1572532753269afhHl:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_bonber=2fdisroot=40chat.disroot.org:bau-ha.us', '$1570024587559htYII:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@xmpp__maryjane_disroot@chat.disroot.org:disroot.org', '$148320022659tPVvI:disroot.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@ako:disroot.org', '$14828968831161RIEmH:disroot.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_coldbuddy=2fdisroot=40chat.disroot.org:bau-ha.us', '$15797292574589ldplO:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_cc__:matrix.org', '$1581477965334072nLqHq:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_outloudvi[m]:matrix.org', '$1568414692354648ZjaxW:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_icMikaela:matrix.org', '$15527884382355807cglWS:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_hekanju=2fdisroot=40chat.disroot.org:bau-ha.us', '$15817564342148nRfWM:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@Ciblia:matrix.org', '$14875843601338755aoxIF:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_tulsi=2fdisroot=40chat.disroot.org:bau-ha.us', '$15597705089011kdknj:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_expresionvocaloid=2fdisroot=40chat.disroot.org:bau-ha.us', '$1581552021240JdAlH:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@xmpp__mupp-xmpp_disroot@chat.disroot.org:disroot.org', '$14817577361HHxIb:disroot.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_cc_:matrix.org', '$15795036653073557bazgf:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_ozwald1=2fdisroot=40chat.disroot.org:bau-ha.us', '$157848178218806BPZSC:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_basedude:matrix.org', '$1581981956206514NWSuC:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_jh=2fdisroot=40chat.disroot.org:bau-ha.us', '$15701727904496PzlpJ:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_userdebug=2fdisroot=40chat.disroot.org:bau-ha.us', '$156011843485naJvu:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_maryjane=2fdisroot=40chat.disroot.org:bau-ha.us', '$156215398215wRLpH:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_muntashir=2fdisroot=40chat.disroot.org:bau-ha.us', '$15736709891108iMkGh:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_oxo=2fdisroot=40chat.disroot.org:bau-ha.us', '$157776490011238ndfKx:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_mars=2fdisroot=40chat.disroot.org:bau-ha.us', '$15610171791521lDEws:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_anam_:matrix.org', '$15794610782848027gKObS:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_awal=2fdisroot=40chat.disroot.org:bau-ha.us', '$15619195565420gtHuw:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_clackety:matrix.org', '$1581981967206617EIcHL:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_coffeebird=2fdisroot=40chat.disroot.org:bau-ha.us', '$15799600881294bqUpd:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_learningc:matrix.org', '$1581982036207290rMdAj:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_amam5=2fdisroot=40chat.disroot.org:bau-ha.us', '$15819078625064wUKbr:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_techbolt:matrix.org', '$1581982171208302UeyVR:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_nly:matrix.org', '$1581982073207557qvsLO:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_linuxturtle=2fdisroot=40chat.disroot.org:bau-ha.us', '$15788870471004NioHg:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_anon003=2fdisroot=40chat.disroot.org:bau-ha.us', '$15649291498697huXxU:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_Nuts_:matrix.org', '$1581981924206165BasRn:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_ironmonkey=2fdisroot=40chat.disroot.org:bau-ha.us', '$15596783686136AMFRx:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_diego.m51=2fdisroot=40chat.disroot.org:bau-ha.us', '$15764786081764JPrLN:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_toktr=2fdisroot=40chat.disroot.org:bau-ha.us', '$15787462713806rdUEJ:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_anam1:matrix.org', '$15856066292028898SqoHp:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_intuxikated[m]:matrix.org', '$1581982011207014mcTRp:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_crayon=2fdisroot=40chat.disroot.org:bau-ha.us', '$15748640221597PeIWX:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_freelancehero=2fdisroot=40chat.disroot.org:bau-ha.us', '$15766942317764sfedM:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_steve-_-:matrix.org', '$1581982168208278hkrXe:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_gmar=2fdisroot=40chat.disroot.org:bau-ha.us', '$15648527237255EdYZr:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_shahab:matrix.org', '$1581982151208154eNyoR:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_dingen:matrix.org', '$1581981981206735ZuDaN:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_coffeecat=2fdisroot=40chat.disroot.org:bau-ha.us', '$1559930340460LzsPu:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_massimiliano:matrix.org', '$1581982050207399jPLNK:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_draxaris=2fdisroot=40chat.disroot.org:bau-ha.us', '$15792705211025ggfbc:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_insanity=2fdisroot=40chat.disroot.org:bau-ha.us', '$15764764951668zsoor:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_vigilian=2fdisroot=40chat.disroot.org:bau-ha.us', '$15818530323802whQfB:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_ajeremias:matrix.org', '$1581981945206392RveoE:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_azhar=2fdisroot=40chat.disroot.org:bau-ha.us', '$1570033096739sEVkN:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_skrzyp1:matrix.org', '$15856066262028868MwMuk:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_perflyst:matrix.org', '$1581982088207679SVFJp:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_bou_:matrix.org', '$1581981962206567sHRuC:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_naimishavasi=2fdisroot=40chat.disroot.org:bau-ha.us', '$15597542318481auPjG:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_=5bAlien=5d=2fdisroot=40chat.disroot.org:bau-ha.us', '$15827418482574PilQq:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_steve=2fdisroot=40chat.disroot.org:bau-ha.us', '$155989704923411BIxaS:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_gongzisun=2fdisroot=40chat.disroot.org:bau-ha.us', '$15754102044760MCAZn:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_lurch:matrix.org', '$1581982042207347taXPq:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_ricardog08=2fdisroot=40chat.disroot.org:bau-ha.us', '$1576731958466PFaJB:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_jvuitton=2fdisroot=40chat.disroot.org:bau-ha.us', '$15799569861208HKMYK:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_oier=2fdisroot=40chat.disroot.org:bau-ha.us', '$15797935136359PBfdS:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_atilano=2fdisroot=40chat.disroot.org:bau-ha.us', '$15799653941474xgemx:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_perflyst[x]:matrix.org', '$1581982090207694hnQPR:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_golanv[m]:matrix.org', '$1581981996206881vYiiJ:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_sum_one=2fdisroot=40chat.disroot.org:bau-ha.us', '$1572541412433TJxHz:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_mitfree[m]:matrix.org', '$1581982062207478jWarf:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_antilopa:matrix.org', '$15792581791680716dYvHn:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_andy=2fdisroot=40chat.disroot.org:bau-ha.us', '$15649236058575nbson:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_Sasha1=2fdisroot=40chat.disroot.org:bau-ha.us', '$15764646841487DrxLQ:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_TheAceOfHearts:matrix.org', '$1581981933206280UVbIt:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_tlaxkit:matrix.org', '$1568414704354840RNUAF:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_salonitanicus=2fdisroot=40chat.disroot.org:bau-ha.us', '$15793799921860JssEl:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_strypey=2fdisroot=40chat.disroot.org:bau-ha.us', '$1581669173452EWNlB:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_andresflamenco=2fdisroot=40chat.disroot.org:bau-ha.us', '$15818221063313CfDpM:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_adn=2fdisroot=40chat.disroot.org:bau-ha.us', '$15747491651878rTzes:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_meaz=2fdisroot=40chat.disroot.org:bau-ha.us', '$15638995173197JHscv:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_GyroW_:matrix.org', '$15856066302028921zeAAc:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_ignacio=2fdisroot=40chat.disroot.org:bau-ha.us', '$1576159219813QmNwQ:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_mikelpz_:matrix.org', '$15856066302028928hBcOk:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_ryzendeath=2fdisroot=40chat.disroot.org:bau-ha.us', '$15596287384626aTuJH:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_pinguingitkeyz=2fdisroot=40chat.disroot.org:bau-ha.us', '$15763980482945xJCAP:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_masnavan=2fdisroot=40chat.disroot.org:bau-ha.us', '$15647598165568WtmKA:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_aerro=2fdisroot=40chat.disroot.org:bau-ha.us', '$15817812352571RnloL:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_pempri=2fdisroot=40chat.disroot.org:bau-ha.us', '$15764762471660JALHF:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_oscar=2fdisroot=40chat.disroot.org:bau-ha.us', '$15648337076842BqDjP:bau-ha.us')]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/synapse/lib/python3.7/site-packages/synapse/metrics/background_process_metrics.py", line 213, in run
    return (yield result)
  File "/opt/synapse/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/opt/synapse/lib/python3.7/site-packages/synapse/replication/tcp/resource.py", line 166, in _run_notifier_loop
    updates, current_token, limited = await stream.get_updates()
  File "/opt/synapse/lib/python3.7/site-packages/synapse/replication/tcp/streams/_base.py", line 134, in get_updates
    self.last_token, current_token
  File "/opt/synapse/lib/python3.7/site-packages/synapse/replication/tcp/streams/_base.py", line 159, in get_updates_since
    from_token, upto_token, limit,
  File "/opt/synapse/lib/python3.7/site-packages/synapse/replication/tcp/streams/_base.py", line 178, in update_function
    assert len(updates) <= limit
AssertionError
2020-04-25 18:10:14,980 - synapse.access.http.9092 - 301 - INFO - GET-9613- ::ffff:127.0.0.1 - 9092 - {None} Processed request: 0.053sec/-0.000sec (0.051sec, 0.000sec) (0.000sec/0.000sec/0) 783423B 200 "GET /_synapse/metrics/master HTTP/1.0" "Prometheus/2.17.1" [0 dbevts]
2020-04-25 18:10:15,242 - synapse.logging.context - 355 - WARNING - renew_attestations-4- Re-starting finished log context renew_attestations-4
2020-04-25 18:10:15,244 - synapse.http.matrixfederationclient - 408 - INFO - renew_attestations-4- {POST-O-432} [krtdex.com] Sending request: POST matrix://krtdex.com/_matrix/federation/v1/groups/%2Bcops%3Ahalf-shot.uk/renew_attestation/%40seequ%3Akrtdex.com; timeout 60.000000s
2020-04-25 18:10:15,246 - synapse.http.federation.matrix_federation_agent - 242 - INFO - renew_attestations-4- Connecting to m.krtdex.com:8448
2020-04-25 18:10:15,949 - synapse.replication.http.federation - 160 - INFO - POST-9614- Got 'm.receipt' edu from matrix.org
2020-04-25 18:10:16,112 - synapse.replication.http.federation - 116 - INFO - POST-9615- Got 1 events from federation
2020-04-25 18:10:16,194 - synapse.access.http.9593 - 301 - INFO - POST-9614- 127.0.0.1 - 9593 - {None} Processed request: 0.245sec/0.000sec (0.004sec, 0.000sec) (0.005sec/0.235sec/5) 4B 200 "POST /_synapse/replication/fed_send_edu/m.receipt/kteZMhPYbI HTTP/1.1" "Synapse/1.12.4rc1" [0 dbevts]
2020-04-25 18:10:16,214 - synapse.logging.context - 355 - WARNING - renew_attestations-4- Re-starting finished log context renew_attestations-4
2020-04-25 18:10:16,215 - synapse.http.matrixfederationclient - 408 - INFO - renew_attestations-4- {POST-O-431} [krtdex.com] Sending request: POST matrix://krtdex.com/_matrix/federation/v1/groups/%2Bteam-scrollup%3Ahalf-shot.uk/renew_attestation/%40seequ%3Akrtdex.com; timeout 60.000000s
2020-04-25 18:10:16,217 - synapse.http.federation.matrix_federation_agent - 242 - INFO - renew_attestations-4- Connecting to m.krtdex.com:8448
2020-04-25 18:10:16,231 - synapse.replication.tcp.resource - 169 - INFO - replication_notifier-5473- Failed to handle stream events
2020-04-25 18:10:16,232 - synapse.metrics.background_process_metrics - 215 - ERROR - replication_notifier-5473- Background process 'replication_notifier' threw an exception
...etc etc...

richvdh · 2020-04-27T09:05:55Z

sorry @Half-Shot can you include the ERROR line above the traceback?

richvdh · 2020-04-27T09:06:58Z

(this looks closely related to #7308 but a slightly different code path)

Half-Shot · 2020-04-27T11:34:11Z

@richvdh my logs have since rotated but the traceback was cased by the error below it (it was looping the same error and traceback in the logs over and over).

I can try to run develop again and capture fresh logs, if useful.

richvdh · 2020-04-27T13:30:02Z

ah don't worry. Will have a stare at the code and try and track it down.

richvdh · 2020-04-27T21:28:45Z

I'm struggling to figure out which stream could be throwing that assertion. I'll add some better logging and then get you to try again.

richvdh · 2020-04-27T21:40:34Z

Oh wait, it says:

2020-04-25 18:10:16,231 - synapse.replication.tcp.resource - 169 - INFO - replication_notifier-5473- Failed to handle stream events

but then... that makes no sense, because as of #7337, the events stream doesn't go via tcp/streams/_base.py:178. @Half-Shot : are you sure that stack trace came from develop?

Half-Shot · 2020-04-28T17:16:24Z

2020-04-28 17:15:02,264 - synapse.access.http.9593 - 301 - INFO - POST-6164- 127.0.0.1 - 9593 - {None} Processed request: 0.258sec/0.000sec (0.004sec, 0.001sec) (0.003sec/0.250sec/5) 4B 200 "POST /_synapse/replication/fed_send_edu/m.receipt/pApQEXzcZG HTTP/1.1" "Synapse/1.12.4" [0 dbevts]
2020-04-28 17:15:02,268 - synapse.replication.http.send_event - 119 - INFO - POST-6166- Got event to send with ID: $7EhWbiwsfRAJnGlaZ-BUyGkX-ueNNAghtQbMDmqQQ1c into room: !TdAwENXmXuMrCrFEFX:maunium.net
2020-04-28 17:15:02,289 - synapse.replication.tcp.resource - 166 - INFO - replication_notifier-3139- Failed to handle stream events
2020-04-28 17:15:02,289 - synapse.metrics.background_process_metrics - 215 - ERROR - replication_notifier-3139- Background process 'replication_notifier' threw an exception
Traceback (most recent call last):
  File "/opt/synapse/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: [(36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@lucio776:matrix.org', '$1557941942501905Sfzou:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@theloudspeaker:matrix.org', '$1577126616423568yPobd:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@babel:matrix.org', '$155887605030618Idvhk:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_mdjanic:matrix.org', '$15787689151746958uFEpF:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_traumsch1le:matrix.org', '$15817340261773906pCrKc:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_joffamac=2fdisroot=40chat.disroot.org:bau-ha.us', '$156832202421036GaODX:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_soila=2fdisroot=40chat.disroot.org:bau-ha.us', '$15610770572287kNVpI:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_userdebug=2fdisroot=40chat.disroot.org:bau-ha.us', '$156011843485naJvu:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_outloudvi[m]:matrix.org', '$15527884852356229JIVyi:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_maryjane=2fdisroot=40chat.disroot.org:bau-ha.us', '$15708084934410ATnIV:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_adho=2fdisroot=40chat.disroot.org:bau-ha.us', '$15757335581994YrcXV:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_uggo=2fdisroot=40chat.disroot.org:bau-ha.us', '$15623700031264ubARR:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_pwr=2fdisroot=40chat.disroot.org:bau-ha.us', '$15620757306854vDIBY:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_awal=2fdisroot=40chat.disroot.org:bau-ha.us', '$15618775895009KnrMA:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_wikwg9=2fdisroot=40chat.disroot.org:bau-ha.us', '$15714053802598hnJCR:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_errorabend=2fdisroot=40chat.disroot.org:bau-ha.us', '$15688424278967FZakH:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_amam5=2fdisroot=40chat.disroot.org:bau-ha.us', '$15734361532161BbYFO:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_geuse=2fdisroot=40chat.disroot.org:bau-ha.us', '$1575234594504DTPaY:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_marmulak=2fdisroot=40chat.disroot.org:bau-ha.us', '$15747969853227oYCqq:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_mmzz=2fdisroot=40chat.disroot.org:bau-ha.us', '$1563753312441XqjtN:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_dhanesh95|xmpp:matrix.org', '$1568414689354606MdVjS:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_debacle=2fdisroot=40chat.disroot.org:bau-ha.us', '$15595831783257CAoKs:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_diego.m51=2fdisroot=40chat.disroot.org:bau-ha.us', '$15649050868112JhRHH:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@ako:disroot.org', '$149684567160411BCmrr:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_tobi=2fdisroot=40chat.disroot.org:bau-ha.us', '$15616482092884rSwis:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_sabisil=2fdisroot=40chat.disroot.org:bau-ha.us', '$15606345081050TaKHE:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_post2me=2fdisroot=40chat.disroot.org:bau-ha.us', '$15608360532818lRamH:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_abdhessuk=2fdisroot=40chat.disroot.org:bau-ha.us', '$15610712722234qEfyD:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_perflyst_=2fdisroot=40chat.disroot.org:bau-ha.us', '$15608333262786vnMDI:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_bcome=2fdisroot=40chat.disroot.org:bau-ha.us', '$15610749252269kFLfr:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_alexesc=2fdisroot=40chat.disroot.org:bau-ha.us', '$15688338008706RADlo:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_neonbukowskii=2fdisroot=40chat.disroot.org:bau-ha.us', '$15734145301732aNRaI:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_dasj19_:matrix.org', '$1565646951342278vjPvO:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_tuxita=2fdisroot=40chat.disroot.org:bau-ha.us', '$15617342993781lRlBx:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_bartender=2fdisroot=40chat.disroot.org:bau-ha.us', '$15608420272895wVKnX:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_valeford=2fdisroot=40chat.disroot.org:bau-ha.us', '$15649203998498dOqvO:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@xmpp__maryjane_disroot@chat.disroot.org:disroot.org', '$149684567760514XBBLD:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_cr5t=2fdisroot=40chat.disroot.org:bau-ha.us', '$1573889623721yIEfs:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_szymon.wierzbicki=2fdisroot=40chat.disroot.org:bau-ha.us', '$15734011631363QWLez:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_zuc=2fdisroot=40chat.disroot.org:bau-ha.us', '$15610700932229tffQN:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_tblu=2fdisroot=40chat.disroot.org:bau-ha.us', '$15734287442071LYdxF:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_steve-_-:matrix.org', '$15549365014950517rEeDx:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@Ciblia:matrix.org', '$1497678116800470KNbQf:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_nrice=2fdisroot=40chat.disroot.org:bau-ha.us', '$1560581808481BPuIZ:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_coffeecat=2fdisroot=40chat.disroot.org:bau-ha.us', '$1559930340460LzsPu:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_ale=2fdisroot=40chat.disroot.org:bau-ha.us', '$15618649674903jBEFq:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@mikaela:privacytools.io', '$15814444571375IMnnM:privacytools.io'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_alhassanaraouf=2fdisroot=40chat.disroot.org:bau-ha.us', '$15767749621007QqQxk:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_joselp=2fdisroot=40chat.disroot.org:bau-ha.us', '$157480289413xGagW:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_ebi=2fdisroot=40chat.disroot.org:bau-ha.us', '$15596080624202apyJP:bau-ha.us')]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/synapse/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: [(36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@lucio776:matrix.org', '$1557941942501905Sfzou:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@theloudspeaker:matrix.org', '$1577126616423568yPobd:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@babel:matrix.org', '$155887605030618Idvhk:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_mdjanic:matrix.org', '$15787689151746958uFEpF:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_traumsch1le:matrix.org', '$15817340261773906pCrKc:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_joffamac=2fdisroot=40chat.disroot.org:bau-ha.us', '$156832202421036GaODX:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_soila=2fdisroot=40chat.disroot.org:bau-ha.us', '$15610770572287kNVpI:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_userdebug=2fdisroot=40chat.disroot.org:bau-ha.us', '$156011843485naJvu:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_outloudvi[m]:matrix.org', '$15527884852356229JIVyi:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_maryjane=2fdisroot=40chat.disroot.org:bau-ha.us', '$15708084934410ATnIV:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_adho=2fdisroot=40chat.disroot.org:bau-ha.us', '$15757335581994YrcXV:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_uggo=2fdisroot=40chat.disroot.org:bau-ha.us', '$15623700031264ubARR:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_pwr=2fdisroot=40chat.disroot.org:bau-ha.us', '$15620757306854vDIBY:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_awal=2fdisroot=40chat.disroot.org:bau-ha.us', '$15618775895009KnrMA:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_wikwg9=2fdisroot=40chat.disroot.org:bau-ha.us', '$15714053802598hnJCR:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_errorabend=2fdisroot=40chat.disroot.org:bau-ha.us', '$15688424278967FZakH:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_amam5=2fdisroot=40chat.disroot.org:bau-ha.us', '$15734361532161BbYFO:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_geuse=2fdisroot=40chat.disroot.org:bau-ha.us', '$1575234594504DTPaY:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_marmulak=2fdisroot=40chat.disroot.org:bau-ha.us', '$15747969853227oYCqq:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_mmzz=2fdisroot=40chat.disroot.org:bau-ha.us', '$1563753312441XqjtN:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_dhanesh95|xmpp:matrix.org', '$1568414689354606MdVjS:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_debacle=2fdisroot=40chat.disroot.org:bau-ha.us', '$15595831783257CAoKs:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_diego.m51=2fdisroot=40chat.disroot.org:bau-ha.us', '$15649050868112JhRHH:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@ako:disroot.org', '$149684567160411BCmrr:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_tobi=2fdisroot=40chat.disroot.org:bau-ha.us', '$15616482092884rSwis:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_sabisil=2fdisroot=40chat.disroot.org:bau-ha.us', '$15606345081050TaKHE:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_post2me=2fdisroot=40chat.disroot.org:bau-ha.us', '$15608360532818lRamH:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_abdhessuk=2fdisroot=40chat.disroot.org:bau-ha.us', '$15610712722234qEfyD:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_perflyst_=2fdisroot=40chat.disroot.org:bau-ha.us', '$15608333262786vnMDI:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_bcome=2fdisroot=40chat.disroot.org:bau-ha.us', '$15610749252269kFLfr:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_alexesc=2fdisroot=40chat.disroot.org:bau-ha.us', '$15688338008706RADlo:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_neonbukowskii=2fdisroot=40chat.disroot.org:bau-ha.us', '$15734145301732aNRaI:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_dasj19_:matrix.org', '$1565646951342278vjPvO:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_tuxita=2fdisroot=40chat.disroot.org:bau-ha.us', '$15617342993781lRlBx:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_bartender=2fdisroot=40chat.disroot.org:bau-ha.us', '$15608420272895wVKnX:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_valeford=2fdisroot=40chat.disroot.org:bau-ha.us', '$15649203998498dOqvO:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@xmpp__maryjane_disroot@chat.disroot.org:disroot.org', '$149684567760514XBBLD:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_cr5t=2fdisroot=40chat.disroot.org:bau-ha.us', '$1573889623721yIEfs:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_szymon.wierzbicki=2fdisroot=40chat.disroot.org:bau-ha.us', '$15734011631363QWLez:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_zuc=2fdisroot=40chat.disroot.org:bau-ha.us', '$15610700932229tffQN:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_tblu=2fdisroot=40chat.disroot.org:bau-ha.us', '$15734287442071LYdxF:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_steve-_-:matrix.org', '$15549365014950517rEeDx:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@Ciblia:matrix.org', '$1497678116800470KNbQf:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_nrice=2fdisroot=40chat.disroot.org:bau-ha.us', '$1560581808481BPuIZ:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_coffeecat=2fdisroot=40chat.disroot.org:bau-ha.us', '$1559930340460LzsPu:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_ale=2fdisroot=40chat.disroot.org:bau-ha.us', '$15618649674903jBEFq:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@mikaela:privacytools.io', '$15814444571375IMnnM:privacytools.io'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_alhassanaraouf=2fdisroot=40chat.disroot.org:bau-ha.us', '$15767749621007QqQxk:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_joselp=2fdisroot=40chat.disroot.org:bau-ha.us', '$157480289413xGagW:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_ebi=2fdisroot=40chat.disroot.org:bau-ha.us', '$15596080624202apyJP:bau-ha.us')]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/synapse/lib/python3.7/site-packages/synapse/metrics/background_process_metrics.py", line 213, in run
    return (yield result)
  File "/opt/synapse/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/opt/synapse/lib/python3.7/site-packages/synapse/replication/tcp/resource.py", line 163, in _run_notifier_loop
    updates, current_token, limited = await stream.get_updates()
  File "/opt/synapse/lib/python3.7/site-packages/synapse/replication/tcp/streams/_base.py", line 138, in get_updates
    self.last_token, current_token
  File "/opt/synapse/lib/python3.7/site-packages/synapse/replication/tcp/streams/_base.py", line 163, in get_updates_since
    from_token, upto_token, _STREAM_UPDATE_TARGET_ROW_COUNT,
  File "/opt/synapse/lib/python3.7/site-packages/synapse/replication/tcp/streams/events.py", line 180, in _update_function
    assert len(state_rows) < target_row_count
AssertionError
2020-04-28 17:15:02,465 - synapse.access.http.9593 - 301 - INFO - POST-6166- 127.0.0.1 - 9593 - {@Half-Shot:half-shot.uk} Processed request: 0.197sec/0.000sec (0.003sec, 0.000sec) (0.001sec/0.018sec/1) 2B 200 "POST /_synapse/replication/send_event/%247EhWbiwsfRAJnGlaZ-BUyGkX-ueNNAghtQbMDmqQQ1c/nWxGBrDBCH HTTP/1.1" "Synapse/1.12.4" [0 dbevts]

fresh logs from develop

richvdh · 2020-04-28T21:11:52Z

thanks, looks like a dup of #7308

eMPee584 mentioned this issue Apr 25, 2020

Synapse workers spantaleev/matrix-docker-ansible-deploy#456

Merged

richvdh added the operation gemini label Apr 27, 2020

richvdh added the release blocker label Apr 27, 2020

richvdh self-assigned this Apr 27, 2020

richvdh closed this as completed Apr 28, 2020

richvdh added the A-Workers Problems related to running Synapse in Worker Mode (or replication) label Feb 16, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Synchrotron stopped relaying new messages #7345

Synchrotron stopped relaying new messages #7345

Half-Shot commented Apr 25, 2020 •

edited

Loading

Half-Shot commented Apr 25, 2020

Half-Shot commented Apr 25, 2020

babolivier commented Apr 25, 2020

Half-Shot commented Apr 25, 2020

Half-Shot commented Apr 25, 2020

richvdh commented Apr 25, 2020

Half-Shot commented Apr 25, 2020

richvdh commented Apr 27, 2020

richvdh commented Apr 27, 2020

Half-Shot commented Apr 27, 2020

richvdh commented Apr 27, 2020

richvdh commented Apr 27, 2020

richvdh commented Apr 27, 2020

Half-Shot commented Apr 28, 2020

richvdh commented Apr 28, 2020

Synchrotron stopped relaying new messages #7345

Synchrotron stopped relaying new messages #7345

Comments

Half-Shot commented Apr 25, 2020 • edited Loading

Half-Shot commented Apr 25, 2020

Half-Shot commented Apr 25, 2020

babolivier commented Apr 25, 2020

Half-Shot commented Apr 25, 2020

Half-Shot commented Apr 25, 2020

richvdh commented Apr 25, 2020

Half-Shot commented Apr 25, 2020

richvdh commented Apr 27, 2020

richvdh commented Apr 27, 2020

Half-Shot commented Apr 27, 2020

richvdh commented Apr 27, 2020

richvdh commented Apr 27, 2020

richvdh commented Apr 27, 2020

Half-Shot commented Apr 28, 2020

richvdh commented Apr 28, 2020

Half-Shot commented Apr 25, 2020 •

edited

Loading