Skip to content
This repository has been archived by the owner on Aug 21, 2023. It is now read-only.

rebuild mysql conn when retry failed chunks and support --transactional-consistency parameter #199

Merged
merged 7 commits into from
Nov 17, 2020

Conversation

lichunzhu
Copy link
Contributor

@lichunzhu lichunzhu commented Nov 12, 2020

What problem does this PR solve?

#178
When we retry on failed chunks, if mysql connection is broken, the retry will fail directly and will take no effect.

What is changed and how it works?

  1. Add --transactional-consistency option. The default value is true. If it is false, dumpling won't release locks until the dumping progress is finished.
  2. When dumpling retrys on failed chunks, we should rebuild MySQL connection when it was broken.
  • For consistency none and consistency snapshot, we can rebuild MySQL connection directly since it doesn't assure consistent snapshot. But for consistency none we should update metadata after conn for DM when we build new MySQL conn.
  • For consistency flush and consistency lock, we can rebuild MySQL connection only when dumpling hold locks during the whole dumping progress (--transactional-consistency = false). Before rebuilding MySQL connection we should also check that whether the mysql lock connection is still alive.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
    Test kill the socket mysql connection. The connection can be rebuilt and dump data correctly.

Side effects

  • Increased code complexity

Related changes

  • Need to cherry-pick to the release branch
  • Need to update the documentation

Release note

  • Support --transactional-consistency parameter and support rebuild mysql connections during retry.

@lichunzhu
Copy link
Contributor Author

Should be merged after #190

@kennytm
Copy link
Collaborator

kennytm commented Nov 13, 2020

i suggest we avoid the obscure abbreviation trx.

since --trx-consistency-only in mydumper defaults to "false", while here we default to "true", i think we can pick a different name. --transactional-consistency=0 or --transactional-snapshot=0 or something.

@lichunzhu
Copy link
Contributor Author

i suggest we avoid the obscure abbreviation trx.

since --trx-consistency-only in mydumper defaults to "false", while here we default to "true", i think we can pick a different name. --transactional-consistency=0 or --transactional-snapshot=0 or something.

I think changing the name is okay, but we should keep this behavior the same as before, which means --transactional-consistency defaults to 1.

@kennytm
Copy link
Collaborator

kennytm commented Nov 16, 2020

Yes the default should be 1.

kennytm
kennytm previously approved these changes Nov 16, 2020
Copy link
Collaborator

@kennytm kennytm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lichunzhu
Copy link
Contributor Author

fix some typo in 91dcef2

@lichunzhu lichunzhu merged commit fe21f01 into pingcap:master Nov 17, 2020
@lichunzhu lichunzhu deleted the rebuildMySQLConn branch November 17, 2020 02:30
@lichunzhu lichunzhu added status/LGT1 One reviewer approved (LGTM1) and removed status/PTAL labels Nov 17, 2020
@lichunzhu lichunzhu mentioned this pull request Nov 17, 2020
@lichunzhu lichunzhu changed the title rebuild mysql conn when retry failed chunks and support trx-consistency-only parameter rebuild mysql conn when retry failed chunks and support --transactional-consistency parameter Nov 17, 2020
tisonkun pushed a commit to tisonkun/dumpling that referenced this pull request Oct 20, 2021
…l-consistency` parameter (pingcap#199)

* support rebuild mysql connection to retry failed chunks

* refine consistency variables

* add --transactional-consistency and support rebuilding mysql conn to retry
tisonkun pushed a commit to tisonkun/dumpling that referenced this pull request Oct 20, 2021
…l-consistency` parameter (pingcap#199)

* support rebuild mysql connection to retry failed chunks

* refine consistency variables

* add --transactional-consistency and support rebuilding mysql conn to retry
tisonkun pushed a commit to tisonkun/dumpling that referenced this pull request Oct 20, 2021
…l-consistency` parameter (pingcap#199)

* support rebuild mysql connection to retry failed chunks

* refine consistency variables

* add --transactional-consistency and support rebuilding mysql conn to retry
tisonkun pushed a commit to tisonkun/dumpling that referenced this pull request Oct 20, 2021
…l-consistency` parameter (pingcap#199)

* support rebuild mysql connection to retry failed chunks

* refine consistency variables

* add --transactional-consistency and support rebuilding mysql conn to retry
tisonkun pushed a commit to tisonkun/tidb that referenced this pull request Oct 20, 2021
…l-consistency` parameter (pingcap/dumpling#199)

* support rebuild mysql connection to retry failed chunks

* refine consistency variables

* add --transactional-consistency and support rebuilding mysql conn to retry
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
status/LGT1 One reviewer approved (LGTM1)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants