Skip to content
This repository has been archived by the owner on Aug 21, 2023. It is now read-only.

add retry mechanism #182

Merged
merged 13 commits into from
Nov 6, 2020
Merged

add retry mechanism #182

merged 13 commits into from
Nov 6, 2020

Conversation

lichunzhu
Copy link
Contributor

@lichunzhu lichunzhu commented Nov 3, 2020 •
edited
Loading

What problem does this PR solve?

#178
#180

What is changed and how it works?

  1. Retry dumping on failed chunks.
  2. Increase maxIdleConnsPerHost when config.Threads is set. Reference: https://github.com/RedTeamPentesting/monsoon/pull/3/files#diff-3ad9aea5b78bcb9b0ca9164d4cd81a2c462dbc82d2a3e7b5477f66134c65e05cR52

Check List

Tests

  • Unit test
  • Integration test

Side effects

  • Possible performance regression

Related changes

  • Need to cherry-pick to the release branch

Release note

  • Retry dumping on failed chunks.

@lichunzhu
Copy link
Contributor Author

I have tested using lldb to kill several socket connections while dumping. Dumpling works fine even in this situation.

kennytm
kennytm reviewed Nov 5, 2020
View reviewed changes
Copy link
Collaborator

@kennytm kennytm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rest LGTM

v4/export/config.go Outdated Show resolved Hide resolved
v4/export/dump_test.go Outdated Show resolved Hide resolved
kennytm
kennytm reviewed Nov 5, 2020
View reviewed changes
v4/export/dump.go Outdated Show resolved Hide resolved
@lichunzhu lichunzhu mentioned this pull request Nov 6, 2020
Closed
@kennytm kennytm merged commit 69579ad into pingcap:master Nov 6, 2020
@kennytm kennytm added the status/LGT1 One reviewer approved (LGTM1) label Nov 6, 2020
@lichunzhu lichunzhu deleted the addRetryMechanism branch November 6, 2020 08:21
@lichunzhu lichunzhu mentioned this pull request Nov 6, 2020
Closed
tisonkun pushed a commit to tisonkun/dumpling that referenced this pull request Oct 20, 2021
@lichunzhu @kennytm
add retry mechanism (pingcap#182)
f3b3f24 
* add retry mechanism

* add retry for aws http

* add timeout for http client

* don't block when one error occured

* fix

* update go.mod from br

* address comment

* address comment again

* fix bug again

* use error group again

Co-authored-by: kennytm <kennytm@gmail.com>
tisonkun pushed a commit to tisonkun/dumpling that referenced this pull request Oct 20, 2021
@lichunzhu @kennytm
add retry mechanism (pingcap#182)
8a8d5b7 
* add retry mechanism

* add retry for aws http

* add timeout for http client

* don't block when one error occured

* fix

* update go.mod from br

* address comment

* address comment again

* fix bug again

* use error group again

Co-authored-by: kennytm <kennytm@gmail.com>
tisonkun pushed a commit to tisonkun/dumpling that referenced this pull request Oct 20, 2021
@lichunzhu @kennytm
add retry mechanism (pingcap#182)
a2a0658 
* add retry mechanism

* add retry for aws http

* add timeout for http client

* don't block when one error occured

* fix

* update go.mod from br

* address comment

* address comment again

* fix bug again

* use error group again

Co-authored-by: kennytm <kennytm@gmail.com>
tisonkun pushed a commit to tisonkun/dumpling that referenced this pull request Oct 20, 2021
@lichunzhu @kennytm
add retry mechanism (pingcap#182)
461eeda 
* add retry mechanism

* add retry for aws http

* add timeout for http client

* don't block when one error occured

* fix

* update go.mod from br

* address comment

* address comment again

* fix bug again

* use error group again

Co-authored-by: kennytm <kennytm@gmail.com>
tisonkun pushed a commit to tisonkun/tidb that referenced this pull request Oct 20, 2021
@lichunzhu @kennytm @tisonkun
add retry mechanism (pingcap/dumpling#182)
3d72067 
* add retry mechanism

* add retry for aws http

* add timeout for http client

* don't block when one error occured

* fix

* update go.mod from br

* address comment

* address comment again

* fix bug again

* use error group again

Co-authored-by: kennytm <kennytm@gmail.com>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@kennytm kennytm kennytm approved these changes

Assignees
No one assigned
Labels
status/LGT1 One reviewer approved (LGTM1)
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@lichunzhu @kennytm