Skip to content
This repository has been archived by the owner on Aug 21, 2023. It is now read-only.

Improve for_select_default_busy #169

Merged
merged 4 commits into from
Oct 21, 2020
Merged

Improve for_select_default_busy #169

merged 4 commits into from
Oct 21, 2020

Conversation

sylzd
Copy link
Contributor

@sylzd sylzd commented Oct 21, 2020

Issue Number: #126

What problem does this PR solve?

Related to #126.
Reduce the cost of selectgo in writeInsert method, which can improve 30%+ perfomance of dumping data.

What is changed and how it works?

selectgo runs only in the case of buffer Reseting.

Check List

Tests

  • Unit test
    Result:
    ? github.com/pingcap/dumpling/cmd/dumpling [no test files]
    ? github.com/pingcap/dumpling/v4/cli [no test files]
    ok github.com/pingcap/dumpling/v4/export 0.145s coverage: 51.6% of statements
    ? github.com/pingcap/dumpling/v4/log [no test files]

  • Integration test
    Result:
    [2020/10/21 13:53:13.812 +08:00] [INFO] [main.go:253] ["dump data successfully, dumpling will exit now"]
    Cleaning up test output dir: /tmp/dumpling_test_result/sql_res.views
    Passed integration tests.

  • Manual test (add detailed scripts or steps below)

    • Command:
    time ./dumpling -h 10.162.1.1 -u dump_test2 -p* -P 4000 --filetype sql -r 10000 --threads 16 -B sbtest -o dumpling_output
    • Before changing:

      • Costs:

                   real    0m21.262s
                  user    3m7.963s
                  sys     0m27.534s
        
    • Flamegraph:
      image

    • After changing

      • Costs:

        real    0m15.175s
        user    1m48.148s
        sys     0m21.989s
        
      • Flamegraph:
        image

Side effects

  • Possible performance regression

    Maybe little delay to handle ctx.Done()

  • Increased code complexity

    None

  • Breaking backward compatibility

Related changes

  • Need to cherry-pick to the release branch
  • Need to update the documentation

Release note

@CLAassistant
Copy link

CLAassistant commented Oct 21, 2020

CLA assistant check
All committers have signed the CLA.

Copy link
Contributor

@lichunzhu lichunzhu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great! Please also update this part for WriteInsertInCsv.
https://github.com/pingcap/dumpling/blob/master/v4/export/writer_util.go#L299

@sylzd
Copy link
Contributor Author

sylzd commented Oct 21, 2020

Great! Please also update this part for WriteInsertInCsv.
https://github.com/pingcap/dumpling/blob/master/v4/export/writer_util.go#L299

Done. unit test & integration test also passed.

@lichunzhu
Copy link
Contributor

Done. unit test & integration test also passed.

OK. Please sign the Contributor License Agreement.

@sylzd
Copy link
Contributor Author

sylzd commented Oct 21, 2020

Done. unit test & integration test also passed.

OK. Please sign the Contributor License Agreement.

signed.

Copy link
Contributor

@lichunzhu lichunzhu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lichunzhu lichunzhu merged commit 2701704 into pingcap:master Oct 21, 2020
@kennytm
Copy link
Collaborator

kennytm commented Oct 21, 2020

/reward 600

@ti-challenge-bot
Copy link

This PR already closed!

@Rustin170506
Copy link
Member

/reward 600

Already added manually. Please reward before merging.

tisonkun pushed a commit to tisonkun/dumpling that referenced this pull request Oct 20, 2021
* fix for_select_default_busy
tisonkun pushed a commit to tisonkun/dumpling that referenced this pull request Oct 20, 2021
* fix for_select_default_busy
tisonkun pushed a commit to tisonkun/dumpling that referenced this pull request Oct 20, 2021
* fix for_select_default_busy
tisonkun pushed a commit to tisonkun/dumpling that referenced this pull request Oct 20, 2021
* fix for_select_default_busy
tisonkun pushed a commit to tisonkun/tidb that referenced this pull request Oct 20, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants