Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Split different RDATA streams into different redis channels #12461

Open
Tracked by #12463 ...
erikjohnston opened this issue Apr 13, 2022 · 0 comments
Open
Tracked by #12463 ...

Split different RDATA streams into different redis channels #12461

erikjohnston opened this issue Apr 13, 2022 · 0 comments
Assignees
Labels
A-Workers Problems related to running Synapse in Worker Mode (or replication) T-Task Refactoring, removal, replacement, enabling or disabling functionality, other engineering tasks.

Comments

@erikjohnston
Copy link
Member

Similar to #12460, we should have a separate pub/sub channel for each RDATA stream so that workers only need to subscribe to the channels they actually need.

This is a bit tricky to do safely, i.e. that workers do correctly subscribe to streams they need, especially for caching purposes. I think the best approach is instead of having the replication client call into the various handlers and storage classes:

async def on_rdata(
self, stream_name: str, instance_name: str, token: int, rows: list
) -> None:
"""Called to handle a batch of replication data with a given stream token.
By default this just pokes the slave store. Can be overridden in subclasses to
handle more.
Args:
stream_name: name of the replication stream for this batch of rows
instance_name: the instance that wrote the rows.
token: stream token for this batch of rows
rows: a list of Stream.ROW_TYPE objects as returned by Stream.parse_row.
"""
self.store.process_replication_rows(stream_name, instance_name, token, rows)

instead each class registers with the replication client in their __init__ each stream they care about with a callback to handle that particular streams rows. That way a worker only subscribes to a stream if a class that requires is actually instantiated.

The downside of this is that it's a bit magic, and pulling in a class can have the side effect of causing a stream to suddenly be subscribed to.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
A-Workers Problems related to running Synapse in Worker Mode (or replication) T-Task Refactoring, removal, replacement, enabling or disabling functionality, other engineering tasks.
Projects
None yet
Development

No branches or pull requests

3 participants