You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.
lant.uk synapse was accepting messages, but they weren't making it to matrix.org.
Anecdotally this seemed to start at the same time I logged out of Riot Android to test another account (not on lant.uk), then tried to log in to my main account again. This login attempt never succeeded.
Synapse was logging the following:
Mar 28 00:11:11 thor matrix-synapse[31968]: 2020-03-27 23:11:11,815 - synapse.federation.sender.per_destination_queue - 332 - ERROR - federation_transaction_transmission_loop-0 - TX [matrix.org] Failed to send transaction
Mar 28 00:11:11 thor matrix-synapse[31968]: Traceback (most recent call last):
Mar 28 00:11:11 thor matrix-synapse[31968]: File "/usr/local/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
Mar 28 00:11:11 thor matrix-synapse[31968]: result = g.send(result)
Mar 28 00:11:11 thor matrix-synapse[31968]: StopIteration: 0
Mar 28 00:11:11 thor matrix-synapse[31968]: During handling of the above exception, another exception occurred:
Mar 28 00:11:11 thor matrix-synapse[31968]: Traceback (most recent call last):
Mar 28 00:11:11 thor matrix-synapse[31968]: File "/usr/local/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
Mar 28 00:11:11 thor matrix-synapse[31968]: result = g.send(result)
Mar 28 00:11:11 thor matrix-synapse[31968]: StopIteration: 0
Mar 28 00:11:11 thor matrix-synapse[31968]: During handling of the above exception, another exception occurred:
Mar 28 00:11:11 thor matrix-synapse[31968]: Traceback (most recent call last):
Mar 28 00:11:11 thor matrix-synapse[31968]: File "/usr/local/lib/python3.7/site-packages/synapse/federation/sender/per_destination_queue.py", line 201, in _transaction_transmission_loop
Mar 28 00:11:11 thor matrix-synapse[31968]: limit
Mar 28 00:11:11 thor matrix-synapse[31968]: File "/usr/local/lib/python3.7/site-packages/synapse/federation/sender/per_destination_queue.py", line 368, in _get_device_update_edus
Mar 28 00:11:11 thor matrix-synapse[31968]: self._destination, last_device_list, limit=limit
Mar 28 00:11:11 thor matrix-synapse[31968]: File "/usr/local/lib/python3.7/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
Mar 28 00:11:11 thor matrix-synapse[31968]: result = result.throwExceptionIntoGenerator(g)
Mar 28 00:11:11 thor matrix-synapse[31968]: File "/usr/local/lib/python3.7/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
Mar 28 00:11:11 thor matrix-synapse[31968]: return g.throw(self.type, self.value, self.tb)
Mar 28 00:11:11 thor matrix-synapse[31968]: File "/usr/local/lib/python3.7/site-packages/synapse/storage/data_stores/main/devices.py", line 233, in get_device_updates_by_remote
Mar 28 00:11:11 thor matrix-synapse[31968]: destination, from_stream_id, query_map
Mar 28 00:11:11 thor matrix-synapse[31968]: File "/usr/local/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
Mar 28 00:11:11 thor matrix-synapse[31968]: result = g.send(result)
Mar 28 00:11:11 thor matrix-synapse[31968]: File "/usr/local/lib/python3.7/site-packages/synapse/storage/data_stores/main/devices.py", line 325, in _get_device_update_edus_by_remote
Mar 28 00:11:11 thor matrix-synapse[31968]: result["keys"].setdefault("signatures", {}).setdefault(
Mar 28 00:11:11 thor matrix-synapse[31968]: KeyError: 'keys'
My db state was this:
synapse=# select * from devices where user_id ='@tom:lant.uk' and hidden = False;
user_id | device_id | display_name | last_seen | ip | user_agent | hidden
--------------+------------------+-------------------------------+---------------+----------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+--------
@tom:lant.uk | Mautrix-Facebook | Generic Matrix Client · Linux | 1585351781327 | - | Python/3.8 aiohttp/3.6.2 | f
@tom:lant.uk | ABICGMVRNF | Riot · Desktop · macOS | 1584985239789 | XXXXXXXX | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) RiotNightly/2020032301 Chrome/80.0.3987.134 Electron/8.0.3 Safari/537.36 | f
@tom:lant.uk | HWVQMHGYDM | Riot · Firefox · macOS | 1585351825192 | XXXXXXXXX | Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:76.0) Gecko/20100101 Firefox/76.0 | f
@tom:lant.uk | EYDLDOKXIK | | | | | f
@tom:lant.uk | BKOEIBIDJW | Mobile | 1585349903283 | XXXXXXXX | Riot.im/0.9.9-dev (Linux; U; Android 9; CLT-L09 Build/HUAWEICLT-L0; Flavour GooglePlay; MatrixAndroidSDK 0.9.31-dev) | f
(5 rows)
synapse=# select * from e2e_device_keys_json where user_id = '@tom:lant.uk';;
user_id | device_id | ts_added_ms | key_json
--------------+------------+---------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
@tom:lant.uk | ABICGMVRNF | 1584632687735 | {"algorithms":["m.olm.v1.curve25519-aes-sha2","m.megolm.v1.aes-sha2"],"device_id":"ABICGMVRNF","keys":{LIST_OF_KEYS},"signatures":{"@tom:lant.uk":{SIGNATURE_STUFF}},"user_id":"@tom:lant.uk"}
@tom:lant.uk | HWVQMHGYDM | 1584973633940 | {"algorithms":["m.olm.v1.curve25519-aes-sha2","m.megolm.v1.aes-sha2"],"device_id":"HWVQMHGYDM","keys":{LIST_OF_KEYS},"signatures":{"@tom:lant.uk":{SIGNATURE_STUFF}},"user_id":"@tom:lant.uk"}
(2 rows)
I guess the stuff in e2e_device_keys_json isn't sensitive but have redacted anyway. I have a copy if it's useful.
The text was updated successfully, but these errors were encountered:
This is suspected to be caused by a client uploading signatures of their device keys, but failing to upload the keys themselves, and then some Synapse code handling that situation badly.
The Synapse code is fixed in v1.12.1 (of which there is currently an RC for. Unfortunately, if you're affected by this bug, other servers will need to upgrade before they can continue to send your server events.
Deleting devices may or may not be a valid workaround at this time. The best suggestion is to wait for the full v1.12.1 release and then to urge others to upgrade. This bug was introduced in v1.11.0, so servers on a version before that will not be affected.
lant.uk synapse was accepting messages, but they weren't making it to matrix.org.
Anecdotally this seemed to start at the same time I logged out of Riot Android to test another account (not on lant.uk), then tried to log in to my main account again. This login attempt never succeeded.
Synapse was logging the following:
My db state was this:
I guess the stuff in e2e_device_keys_json isn't sensitive but have redacted anyway. I have a copy if it's useful.
The text was updated successfully, but these errors were encountered: