Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DM-worker should not retry on long time ddls directly #5962

Closed
lichunzhu opened this issue Jun 21, 2022 · 2 comments
Closed

DM-worker should not retry on long time ddls directly #5962

lichunzhu opened this issue Jun 21, 2022 · 2 comments
Labels
area/dm Issues or PRs related to DM. type/enhancement The issue or PR belongs to an enhancement.

Comments

@lichunzhu
Copy link
Contributor

What did you do?

Use DM-worker to replicate some long time costing ddls to downstream TiDB. For example, adding index ddls.

What did you expect to see?

DM-worker can replicate ddls successfully.

What did you see instead?

DM-worker fails because of i/o timeout and keeps retrying to replicate these ddls again.

Versions of the cluster

master

current status of DM cluster (execute query-status <task-name> in dmctl)

(paste current status of DM cluster here)
@lichunzhu lichunzhu added type/enhancement The issue or PR belongs to an enhancement. area/dm Issues or PRs related to DM. labels Jun 21, 2022
@lichunzhu
Copy link
Contributor Author

lichunzhu commented Jun 21, 2022

Root Cause

When DM executes some time-costing ddls, especially for adding index, dm-worker may fail to get result because it won't return any result until it succeeds. If this time cost is larger than syncer's read-timeout, dm-worker will fail and try to execute ddl again. This will make the situation even worse.

Workaround

For these long time costing ddls, we can use a special connection to execute them and keep watching the result asynchronously through admin show ddl jobs.

@lance6716
Copy link
Contributor

duplicated with #4689

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/dm Issues or PRs related to DM. type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Development

No branches or pull requests

2 participants