Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

*: modify GetGlobalConfig() to provide a thread safe API for unit test #17954

Closed
wants to merge 2 commits into from

Conversation

tiancaiamao
Copy link
Contributor

What problem does this PR solve?

[2020-06-10T08:16:47.771Z] ==================
[2020-06-10T08:16:47.771Z] WARNING: DATA RACE
[2020-06-10T08:16:47.771Z] Write at 0x00c000232ed0 by goroutine 204:
[2020-06-10T08:16:47.771Z]   github.com/pingcap/tidb/planner/core_test.(*testIntegrationSerialSuite).TestNoneAccessPathsFoundByIsolationRead()
[2020-06-10T08:16:47.771Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/planner/core/integration_test.go:304 +0x614
[2020-06-10T08:16:47.772Z]   github.com/pingcap/tidb/planner/core_test.(*testIntegrationSerialSuite).TestNoneAccessPathsFoundByIsolationRead()
[2020-06-10T08:16:47.772Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/planner/core/integration_test.go:286 +0x166
[2020-06-10T08:16:47.772Z]   runtime.call32()
[2020-06-10T08:16:47.772Z]       /usr/local/go/src/runtime/asm_amd64.s:539 +0x3a
[2020-06-10T08:16:47.772Z]   reflect.Value.Call()
[2020-06-10T08:16:47.772Z]       /usr/local/go/src/reflect/value.go:321 +0xd3
[2020-06-10T08:16:47.772Z]   github.com/pingcap/check.(*suiteRunner).forkTest.func1()
[2020-06-10T08:16:47.772Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:850 +0x9aa
[2020-06-10T08:16:47.772Z]   github.com/pingcap/check.(*suiteRunner).forkCall.func1()
[2020-06-10T08:16:47.772Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:739 +0x113
[2020-06-10T08:16:47.772Z] 

[2020-06-10T08:16:47.772Z] Previous read at 0x00c000232ed0 by goroutine 313:
[2020-06-10T08:16:47.772Z]   github.com/pingcap/tidb/sessionctx/variable.NewSessionVars()
[2020-06-10T08:16:47.772Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/sessionctx/variable/session.go:788 +0x1090
[2020-06-10T08:16:47.772Z]   github.com/pingcap/tidb/util/mock.NewContext()
[2020-06-10T08:16:47.772Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/util/mock/context.go:272 +0x9f
[2020-06-10T08:16:47.772Z]   github.com/pingcap/tidb/store/mockstore/unistore/cophandler.newClosureExecutor()
[2020-06-10T08:16:47.772Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/store/mockstore/unistore/cophandler/closure_exec.go:118 +0x1eb
[2020-06-10T08:16:47.772Z]   github.com/pingcap/tidb/store/mockstore/unistore/cophandler.buildClosureExecutor()
[2020-06-10T08:16:47.772Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/store/mockstore/unistore/cophandler/closure_exec.go:57 +0x60
[2020-06-10T08:16:47.772Z]   github.com/pingcap/tidb/store/mockstore/unistore/cophandler.handleCopDAGRequest()
[2020-06-10T08:16:47.772Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/store/mockstore/unistore/cophandler/cop_handler.go:75 +0x1cb
[2020-06-10T08:16:47.772Z]   github.com/pingcap/tidb/store/mockstore/unistore/cophandler.HandleCopRequest()
[2020-06-10T08:16:47.772Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/store/mockstore/unistore/cophandler/cop_handler.go:47 +0x269
[2020-06-10T08:16:47.772Z]   github.com/ngaut/unistore/tikv.(*Server).Coprocessor()
[2020-06-10T08:16:47.772Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/github.com/ngaut/unistore@v0.0.0-20200604061006-d8e9dc0ad154/tikv/server.go:474 +0x3ff
[2020-06-10T08:16:47.772Z]   github.com/pingcap/tidb/store/mockstore/unistore.(*RPCClient).SendRequest()
[2020-06-10T08:16:47.772Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/store/mockstore/unistore/rpc.go:161 +0x2cde
[2020-06-10T08:16:47.772Z]   github.com/pingcap/tidb/store/tikv.reqCollapse.SendRequest()
[2020-06-10T08:16:47.772Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/store/tikv/client_collapse.go:49 +0x17d
[2020-06-10T08:16:47.772Z]   github.com/pingcap/tidb/store/tikv.(*reqCollapse).SendRequest()
[2020-06-10T08:16:47.772Z]       <autogenerated>:1 +0xc5
[2020-06-10T08:16:47.772Z]   github.com/pingcap/tidb/store/tikv.(*RegionRequestSender).sendReqToRegion()
[2020-06-10T08:16:47.772Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/store/tikv/region_request.go:253 +0x1d0
[2020-06-10T08:16:47.772Z]   github.com/pingcap/tidb/store/tikv.(*RegionRequestSender).SendReqCtx()
[2020-06-10T08:16:47.772Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/store/tikv/region_request.go:216 +0x52f
[2020-06-10T08:16:47.772Z]   github.com/pingcap/tidb/store/tikv.(*clientHelper).SendReqCtx()
[2020-06-10T08:16:47.772Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/store/tikv/coprocessor.go:844 +0x1fd
[2020-06-10T08:16:47.772Z]   github.com/pingcap/tidb/store/tikv.(*copIteratorWorker).handleTaskOnce()
[2020-06-10T08:16:47.772Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/store/tikv/coprocessor.go:751 +0x948
[2020-06-10T08:16:47.772Z]   github.com/pingcap/tidb/store/tikv.(*copIteratorWorker).handleTask()
[2020-06-10T08:16:47.772Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/store/tikv/coprocessor.go:686 +0x205
[2020-06-10T08:16:47.772Z]   github.com/pingcap/tidb/store/tikv.(*copIteratorWorker).run()
[2020-06-10T08:16:47.772Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/store/tikv/coprocessor.go:493 +0x141

Problem Summary:

There is only one instance of global config, providing that many test cases run concurrently,
when an unit test modify the global config, it is easy to DATA RACE.

We have been struggle with this problem, in this commit I modify the GetGlobalConfig API to provide a thread safe mechanism for each test to use its own config.

What is changed and how it works?

What's Changed:

  • Change GetGlobalConfig() to GetGlobalConfig(ctx)
  • Provide config.WithGlobalConfig() function

How it Works:

GetGlobalConfig will check the context first, if it find the overrideConfigForTest from the context, use that.

Each test can use MustExecWithCtx() to override the global config for testing.

Check List

Tests

  • Unit test

Side effects

  • Breaking backward compatibility

Modify exposed API break code compatibility

Release note

  • No release note

@tiancaiamao tiancaiamao added type/enhancement The issue or PR belongs to an enhancement. component/config labels Jun 11, 2020
@tiancaiamao tiancaiamao requested review from lysu and jackysp June 11, 2020 07:56
@github-actions github-actions bot added sig/sql-infra SIG: SQL Infra sig/execution SIG execution labels Jun 11, 2020
Copy link
Contributor

@lysu lysu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

tangenta
tangenta previously approved these changes Jun 11, 2020
Copy link
Contributor

@tangenta tangenta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@@ -139,13 +139,17 @@ func (tk *TestKit) GetConnectionID() {

// Exec executes a sql statement.
func (tk *TestKit) Exec(sql string, args ...interface{}) (sqlexec.RecordSet, error) {
return tk.ExecWithCtx(context.Background(), sql, args)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
return tk.ExecWithCtx(context.Background(), sql, args)
return tk.ExecWithCtx(context.Background(), sql, args...)

@tangenta tangenta dismissed their stale review June 11, 2020 09:03

Maybe we should update BR first.

@coocood
Copy link
Member

coocood commented Jun 12, 2020

This change has performance cost, we have to lookup for the context every time we need the global config.

This PR fix the race problem in another way #17964

@tangenta
Copy link
Contributor

I think this PR provides a test-local config mechanism, which allows a few tests in SerialSuite can be run in parallel. In terms of performance cost, perhaps we can use conditional compilation to avoid it in production.

@coocood
Copy link
Member

coocood commented Jun 12, 2020

@tangenta
global config is not the only factor that affects parallel tests, there are many other global variables and objects.

Making an isolated env to run test requires much more effort.

@tiancaiamao
Copy link
Contributor Author

#17964 tried another way to fix DATA RACE

@tiancaiamao tiancaiamao deleted the config-data-race branch June 15, 2020 05:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/config sig/execution SIG execution sig/sql-infra SIG: SQL Infra type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants