Allow start many replication pollers in one shard #3790
Conversation
| processor.Stop() | ||
| delete(r.taskProcessors, perShardTaskProcessorKey) | ||
| } | ||
| if clusterInfo := newClusterMetadata[clusterName]; clusterInfo != nil && clusterInfo.Enabled { |
There was a problem hiding this comment.
what if a cluster exists in both old & new cluster metadata? stop & restart?
what if a cluster only exists in both new cluster metadata? noop?
There was a problem hiding this comment.
if a cluster exists in both old and new, it means the metadata is updated. So stop the processor and restart it to load the new metadata. If a cluster only exists in new cluster. It means this cluster is newly added, so we just start a new processor.
| @@ -401,6 +403,7 @@ func (p *taskProcessorImpl) convertTaskToDLQTask( | |||
| // NOTE: last event vs next event, next event ID is exclusive | |||
| nextEventID := lastEvent.GetEventId() + 1 | |||
|
|
|||
| // TODO: GetShardID will break GetDLQReplicationMessages we need to handle DLQ for cross shard replication. | |||
There was a problem hiding this comment.
Why will it break? Using local shardID for storing DLQ seems the right behavior to me.
There was a problem hiding this comment.
Yes. Here is ok. But when hydrate the replication task from the source cluster, it needs to calculate the shard id instead of using this. We will not hit this case in a short term.
| WithMaximumInterval(config.ReplicationTaskProcessorErrorRetryMaxInterval(shardID)). | ||
| WithMaximumAttempts(config.ReplicationTaskProcessorErrorRetryMaxAttempts(shardID)). | ||
| WithExpirationInterval(config.ReplicationTaskProcessorErrorRetryExpiration(shardID)) | ||
| taskRetryPolicy := backoff.NewExponentialRetryPolicy(config.ReplicationTaskProcessorErrorRetryWait(pollingShardID)). |
There was a problem hiding this comment.
This config cannot be based on pollingShardID, right? or why would it be?
There was a problem hiding this comment.
if you have 3 clusters and the shard counts are all different, then what will you specify in the config?
There was a problem hiding this comment.
Yes. You are right. I am going to revert this.
* Allow start many replication poller in one shard
What changed?
Allow start many replication pollers in one shard
Why?
Allow start many replication pollers in one shard to poll replication tasks from the source cluster.
How did you test it?
Unit tests.
Potential risks
Is hotfix candidate?