Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

coprocessor: skip empty tidb nodes #36548

Closed
wants to merge 20 commits into from

Conversation

mornyx
Copy link
Contributor

@mornyx mornyx commented Jul 26, 2022

Issue Number: close #36546

What is changed and how it works?

As described in #36546, we should avoid trying to send coprocessor requests to apparently illegal TiDB addresses. In this PR, our solution is: When building cop tasks, check the node address to decide whether to skip it directly.

Release note

None

Signed-off-by: mornyx mornyx.z@gmail.com

Signed-off-by: mornyx <mornyx.z@gmail.com>
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Jul 26, 2022

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • crazycs520
  • xhebox

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jul 26, 2022
Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>
@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Jul 26, 2022
@zhongzc
Copy link
Contributor

zhongzc commented Jul 26, 2022

Can we ignore tidb info with empty IP returned by etcd?

func (is *InfoSyncer) getAllServerInfo(ctx context.Context) (map[string]*ServerInfo, error) {
allInfo := make(map[string]*ServerInfo)
if is.etcdCli == nil {
allInfo[is.info.ID] = getServerInfo(is.info.ID, is.info.ServerIDGetter)
return allInfo, nil
}
allInfo, err := getInfo(ctx, is.etcdCli, ServerInformationPath, keyOpDefaultRetryCnt, keyOpDefaultTimeout, clientv3.WithPrefix())
if err != nil {
return nil, err
}
return allInfo, nil
}

@xhebox
Copy link
Contributor

xhebox commented Jul 26, 2022

Can we ignore tidb info with empty IP returned by etcd?

func (is *InfoSyncer) getAllServerInfo(ctx context.Context) (map[string]*ServerInfo, error) {
allInfo := make(map[string]*ServerInfo)
if is.etcdCli == nil {
allInfo[is.info.ID] = getServerInfo(is.info.ID, is.info.ServerIDGetter)
return allInfo, nil
}
allInfo, err := getInfo(ctx, is.etcdCli, ServerInformationPath, keyOpDefaultRetryCnt, keyOpDefaultTimeout, clientv3.WithPrefix())
if err != nil {
return nil, err
}
return allInfo, nil
}

I think we can, but I am not sure if we should.

@mornyx
Copy link
Contributor Author

mornyx commented Jul 26, 2022

Can we ignore tidb info with empty IP returned by etcd?

func (is *InfoSyncer) getAllServerInfo(ctx context.Context) (map[string]*ServerInfo, error) {
allInfo := make(map[string]*ServerInfo)
if is.etcdCli == nil {
allInfo[is.info.ID] = getServerInfo(is.info.ID, is.info.ServerIDGetter)
return allInfo, nil
}
allInfo, err := getInfo(ctx, is.etcdCli, ServerInformationPath, keyOpDefaultRetryCnt, keyOpDefaultTimeout, clientv3.WithPrefix())
if err != nil {
return nil, err
}
return allInfo, nil
}

I can't guarantee whether other code has special dependencies on nodes, even BR nodes. So I just skipped where the problem occurred and didn't modify the entire data source..

@ti-chi-bot ti-chi-bot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jul 26, 2022
@ti-chi-bot ti-chi-bot added needs-cherry-pick-release-5.4 Should cherry pick this PR to release-5.4 branch. needs-cherry-pick-release-6.1 Should cherry pick this PR to release-6.1 branch. needs-cherry-pick-6.2 labels Jul 27, 2022
Signed-off-by: mornyx <mornyx.z@gmail.com>
@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jul 27, 2022
@crazycs520
Copy link
Contributor

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 0efecb2

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jul 27, 2022
@ti-chi-bot
Copy link
Member

@mornyx: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

Signed-off-by: mornyx <mornyx.z@gmail.com>
@ti-chi-bot ti-chi-bot removed the status/can-merge Indicates a PR has been approved by a committer. label Jul 27, 2022
@ti-chi-bot
Copy link
Member

Merge canceled because a new commit is pushed.

@mornyx
Copy link
Contributor Author

mornyx commented Jul 29, 2022

This PR avoids sending copr requests to TiDB nodes with empty IP addresses. Unfortunately, many tests depend on this behavior. When we create a mock store and use it to bootstrap a mock session, all tests will depend on this fact: there is only one node in the entire mocked cluster with an empty IP address and it can handle copr requests normally.

So the optimization of this PR is not easy to achieve, since issue #36546 has been resolved by pr #36558, I will close this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs-cherry-pick-release-5.4 Should cherry pick this PR to release-5.4 branch. needs-cherry-pick-release-6.1 Should cherry pick this PR to release-6.1 branch. release-note-none Denotes a PR that doesn't merit a release note. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Cluster in-mem table query is slow when there are empty tidb nodes in etcd
7 participants