users mysteriously ejected from HQ #7742
Comments
|
we have logs from lieuwe which include the moderately interesting: ... which suggests that state resolution resulted in their server deciding they were no longer in the room. We think those logs are in This sounds very much like #6774. It would be interesting to try to figure out:
|
|
This happened to me too, the logs are somewhat fun (and match lieuw):
|
|
On jki.re looks like event |
|
One interesting thing here is that if you ask servers for the state of {
"pdu_ids": [
"$MaPUZG2fNleIE_gUYDmOy-nj0Zom8mQ5FNM7gEAVI6I", "$gAwT-xGHhlqqg9bnWaP_AyT7j6qjyeWsvCyN2dHYpHw", "$3Z5K-4YJq0uYYAIplFHm7V8vRZgfjCoRV74N8wVWiWw", "$Mu9qujTV7no2PtHgEILrPOib5TJ7YMyH92lWIBPB_sg", "$KzA3bPapYE2mCW4EpSnlAaY1xJ0LrNJjxsPLq4hmqoQ", "$9s0TWzih_V7KH4TJq4RVK8fwXS6ladAoRF3zjh3j_PQ"
],
"auth_chain_ids": [
"$MzrJBHz4bPtW5p-m9mlIjk6_FfXSBrVoEqC9qPa78IQ", "$VTSRTz8yrsbhmpMTBDjl3NntWClCvoihwb1eO-yiHqw", "$0v15XhSH1i3UNZgST6wN5VMf4ItOI3CE0_PTsqlD1cE", "$KzA3bPapYE2mCW4EpSnlAaY1xJ0LrNJjxsPLq4hmqoQ", "$gArjO1RTILcA5p8pYpmc0mlGx581NMj90gEG--89DBw", "$3Z5K-4YJq0uYYAIplFHm7V8vRZgfjCoRV74N8wVWiWw", "$2YGE5c9okZ4jTkbdn5xnGAvaIAlYYBQCzp-ckeyCfEA", "$u2XxFBinIjuK-xBS9cgMiLpOcPw5TCpqQFzNC2uEVFU", "$MaPUZG2fNleIE_gUYDmOy-nj0Zom8mQ5FNM7gEAVI6I", "$WvqflEV-JrulPCpJcyG2LYuiNObeiYEeUAP_NgQwvHY", "$fKDKYB2rIc1VilsiYJoDE-ShiLLMA4MWo3cQOH_fz1c", "$yq8UADQMFWuZVR1wcx0_va7SpJTKv8fJXkI4ax0AxIg", "$9s0TWzih_V7KH4TJq4RVK8fwXS6ladAoRF3zjh3j_PQ", "$eMqfjsGLmwAnmcRNELX5mT6FDIkZiYMLnlzlqfklE8k", "$UYWIGiJzKJEDoGas39f_ArFfdU06Ygzs0dXzJrVQ3TY", "$gAwT-xGHhlqqg9bnWaP_AyT7j6qjyeWsvCyN2dHYpHw"
]
}
|
|
I (@joepie91:pixie.town) am no longer receiving events for Matrix HQ, but I still appear to be joined from my side, and there's no "you got kicked" screen or anything like that. The last visible event dates from June 24, 20:13:37 local time (Netherlands). |
|
almost certainly homeserver-wide for pixie.town, and trying to send a message results in |
|
my synapse logs only go back to yesterday, but lists a lot of PDU's being rejected for HQ (as we are no longer in the room), |
My timezone is GMT+2 (Netherlands) |
|
I seem to be back in HQ now after: |
|
Yup, it's worth noting that this is exacerbated by the issue that state resets don't get sent down |
Worth noting that I didn't explicitly leave the room, but HQ is still fixed for me (on the same homeserver as @f0x52). |
|
|
Saw this again. Room Ok, so:
Later on, if we try and get the current state for |
|
fixed in 96e9afe / synapse 1.15.2. |
|
Since the changelog speaks about "malicious homeserver[s]", were the ejections in Matrix HQ done on purpose? |
|
The changelog also states:
so no. |
|
@richvdh In the interest of other homeserver implementers not making the same mistake, is there any further documentation on what design/implementation decision caused the issue and how it is now being prevented? |
|
in short: if you're going to check that you have the state at a particular point in the DAG, you better check that you actually have the state at that point in the DAG. When you first join a room, you get the state of that room, which might be 1000 events. We don't need to know the state of the room at each of those events - it is sufficient to know that the exist, that they are correctly signed, and what they contain. We therefore store these events as "outliers" in synapse: ie, we have the event itself, but no information about the state at that point in the DAG. We can also get similar outlier events whenever we end up with new gaps in the DAG. The code that was fixed was incorrectly assuming that if we had an event in the database, we also knew what the state of the room was at that event; if it was an outlier, when we tried to use that state, we'd get an empty state set, and obviously that's going to go badly. We think it's a security issue because we think it's possible to engineer that situation. We have no reason to believe that the ejections from HQ were deliberate, though. |
|
I see, thanks! |
|
Today I was once again thrown out of Matrix HQ. My synapse log file: https://1bit.se/files/homeserver.log.1 At 11:05:47 in the logfile I see an error resolving state for the room matrix hq. I should mention that I use synapse 1.17.0 I made a new bug report here #7913 in case this is another issue |
last night, at least 3 users (grin, sbug, lieuwe) found themselves no longer part of matrix HQ and had to rejoin. It would be good to dig into what happened
The text was updated successfully, but these errors were encountered: