create table failed with Table 't' already exists on network partition

[D3Hunter]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

analyzed by @GMHDBJD : this case start 2 domain, both might be owner, and 2 workers are handling create table
https://do.pingcap.net/jenkins/blue/rest/organizations/jenkins/pipelines/pingcap/pipelines/tidb/pipelines/ghpr_unit_test/runs/1295/nodes/59/steps/65/log/?start=0

[2024/01/09 07:14:05.458 +00:00] [INFO] [ddl_worker.go:1156] ["run DDL job"] [worker="worker 1, tp general"] [category=ddl] [jobID=105] [conn=1] [category=ddl] [job="ID:105, Type:create table, State:queueing, SchemaState:none, SchemaID:2, TableID:104, RowCount:0, ArgLen:0, start time: 2024-01-09 07:14:05.433 +0000 UTC, Err:<nil>, ErrCount:0, SnapshotVersion:0, LocalMode: false"]
[2024/01/09 07:14:05.460 +00:00] [INFO] [ddl_worker.go:1156] ["run DDL job"] [worker="worker 3, tp general"] [category=ddl] [jobID=105] [conn=1] [category=ddl] [job="ID:105, Type:create table, State:queueing, SchemaState:none, SchemaID:2, TableID:104, RowCount:0, ArgLen:0, start time: 2024-01-09 07:14:05.433 +0000 UTC, Err:<nil>, ErrCount:0, SnapshotVersion:0, LocalMode: false"]

this err is exposed in test env, not sure how it behave in real network partition

[2024/01/09 07:14:05.429 +00:00] [INFO] [wait_group_wrapper.go:133] ["background process started"] [source=domain] [process=loadSigningCertLoop]
[2024/01/09 07:14:05.429 +00:00] [INFO] [task_manager.go:216] ["scale ttl worker"] [ttl-worker=job-manager] [ttl-worker=task-manager] [originalCount=0] [newCount=4]
[2024/01/09 07:14:05.430 +00:00] [INFO] [task_manager.go:216] ["scale ttl worker"] [ttl-worker=job-manager] [ttl-worker=task-manager] [originalCount=0] [newCount=4]
[2024/01/09 07:14:05.430 +00:00] [INFO] [wait_group_wrapper.go:133] ["background process started"] [source=domain] [process=distTaskFrameworkLoop]
[2024/01/09 07:14:05.431 +00:00] [INFO] [mockstore.go:251] ["domain DDL IDs"] [IDs="[1f9a5bb9-b3e7-4667-b680-308dbe196406,573b1f6d-5b41-4a24-a87c-b5fcd6b838fe]"]
[2024/01/09 07:14:05.431 +00:00] [INFO] [domain.go:1511] ["dist task executor manager started"]
[2024/01/09 07:14:05.432 +00:00] [INFO] [session.go:3884] ["CRUCIAL OPERATION"] [conn=1] [schemaVersion=51] [cur_db=test] [sql="drop table if exists t"] [user=]
[2024/01/09 07:14:05.432 +00:00] [INFO] [session.go:3884] ["CRUCIAL OPERATION"] [conn=1] [schemaVersion=51] [cur_db=test] [sql="create table t(a int not null, b int not null)"] [user=]
[2024/01/09 07:14:05.435 +00:00] [INFO] [ddl_worker.go:259] ["add DDL jobs"] [category=ddl] ["batch count"=1] [jobs="ID:105, Type:create table, State:queueing, SchemaState:none, SchemaID:2, TableID:104, RowCount:0, ArgLen:2, start time: 2024-01-09 07:14:05.433 +0000 UTC, Err:<nil>, ErrCount:0, SnapshotVersion:0, LocalMode: false; "] [table=true]
[2024/01/09 07:14:05.435 +00:00] [INFO] [ddl.go:1188] ["start DDL job"] [category=ddl] [job="ID:105, Type:create table, State:queueing, SchemaState:none, SchemaID:2, TableID:104, RowCount:0, ArgLen:2, start time: 2024-01-09 07:14:05.433 +0000 UTC, Err:<nil>, ErrCount:0, SnapshotVersion:0, LocalMode: false"] [query="create table t(a int not null, b int not null)"]
[2024/01/09 07:14:05.457 +00:00] [INFO] [job_table.go:335] ["get global state and global state change"] [category=ddl] [oldState=false] [currState=false]
[2024/01/09 07:14:05.457 +00:00] [INFO] [job_table.go:350] ["the owner sets owner operator value"] [category=ddl] [ownerOp=none]
[2024/01/09 07:14:05.458 +00:00] [INFO] [ddl_worker.go:1156] ["run DDL job"] [worker="worker 1, tp general"] [category=ddl] [jobID=105] [conn=1] [category=ddl] [job="ID:105, Type:create table, State:queueing, SchemaState:none, SchemaID:2, TableID:104, RowCount:0, ArgLen:0, start time: 2024-01-09 07:14:05.433 +0000 UTC, Err:<nil>, ErrCount:0, SnapshotVersion:0, LocalMode: false"]
[2024/01/09 07:14:05.460 +00:00] [INFO] [ddl_worker.go:1156] ["run DDL job"] [worker="worker 3, tp general"] [category=ddl] [jobID=105] [conn=1] [category=ddl] [job="ID:105, Type:create table, State:queueing, SchemaState:none, SchemaID:2, TableID:104, RowCount:0, ArgLen:0, start time: 2024-01-09 07:14:05.433 +0000 UTC, Err:<nil>, ErrCount:0, SnapshotVersion:0, LocalMode: false"]
[2024/01/09 07:14:05.461 +00:00] [INFO] [domain.go:272] ["diff load InfoSchema success"] [currentSchemaVersion=51] [neededSchemaVersion=52] ["start time"=180.369µs] [gotSchemaVersion=52] [phyTblIDs="[104]"] [actionTypes="[3]"] [diffTypes="[\"create table\"]"]
[2024/01/09 07:14:05.461 +00:00] [INFO] [ddl_worker.go:1115] ["DDL job is cancelled normally"] [worker="worker 3, tp general"] [category=ddl] [jobID=105] [conn=1] [error="[schema:1050]Table 't' already exists"] [errorVerbose="[schema:1050]Table 't' already exists\ngithub.com/pingcap/errors.AddStack\n\texternal/com_github_pingcap_errors/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\texternal/com_github_pingcap_errors/normalize.go:164\ngithub.com/pingcap/tidb/pkg/ddl.checkTableNotExistsFromStore\n\tpkg/ddl/table.go:1577\ngithub.com/pingcap/tidb/pkg/ddl.checkTableNotExists\n\tpkg/ddl/table.go:1500\ngithub.com/pingcap/tidb/pkg/ddl.createTable\n\tpkg/ddl/table.go:65\ngithub.com/pingcap/tidb/pkg/ddl.onCreateTable\n\tpkg/ddl/table.go:176\ngithub.com/pingcap/tidb/pkg/ddl.(*worker).runDDLJob\n\tpkg/ddl/ddl_worker.go:1203\ngithub.com/pingcap/tidb/pkg/ddl.(*worker).HandleDDLJobTable\n\tpkg/ddl/ddl_worker.go:932\ngithub.com/pingcap/tidb/pkg/ddl.(*ddl).delivery2worker.func1\n\tpkg/ddl/job_table.go:462\ngithub.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).Run.func1\n\tpkg/util/wait_group_wrapper.go:157\nruntime.goexit\n\tsrc/runtime/asm_amd64.s:1650"]
[2024/01/09 07:14:05.461 +00:00] [INFO] [ddl_worker.go:738] ["finish DDL job"] [worker="worker 3, tp general"] [category=ddl] [jobID=105] [conn=1] [job="ID:105, Type:create table, State:cancelled, SchemaState:none, SchemaID:2, TableID:104, RowCount:0, ArgLen:2, start time: 2024-01-09 07:14:05.433 +0000 UTC, Err:[schema:1050]Table 't' already exists, ErrCount:1, SnapshotVersion:0, LocalMode: false"]
[2024/01/09 07:14:05.462 +00:00] [INFO] [domain.go:873] ["mdl gets lock, update to owner"] [jobID=105] [version=52]
[2024/01/09 07:14:05.462 +00:00] [INFO] [ddl_worker.go:1390] ["wait latest schema version changed(get the metadata lock if tidb_enable_metadata_lock is true)"] [category=ddl] [ver=52] ["take time"=2.026676ms] [job="ID:105, Type:create table, State:done, SchemaState:public, SchemaID:2, TableID:104, RowCount:0, ArgLen:2, start time: 2024-01-09 07:14:05.433 +0000 UTC, Err:<nil>, ErrCount:0, SnapshotVersion:0, LocalMode: false"]
[2024/01/09 07:14:05.466 +00:00] [INFO] [ddl.go:1298] ["DDL job is failed"] [category=ddl] [jobID=105]
[2024/01/09 07:14:05.466 +00:00] [INFO] [tidb.go:286] ["rollbackTxn called due to ddl/autocommit failure"]
[2024/01/09 07:14:05.466 +00:00] [WARN] [session.go:2251] ["run statement failed"] [schemaVersion=51] [error="[schema:1050]Table 't' already exists"] [errorVerbose="[schema:1050]Table 't' already exists\ngithub.com/pingcap/errors.AddStack\n\texternal/com_github_pingcap_errors/errors.go:174\ngithub.com/pingcap/errors.Trace\n\texternal/com_github_pingcap_errors/juju_adaptor.go:15\ngithub.com/pingcap/tidb/pkg/ddl.(*ddl).DoDDLJob\n\tpkg/ddl/ddl.go:1299\ngithub.com/pingcap/tidb/pkg/ddl.(*ddl).CreateTableWithInfo\n\tpkg/ddl/ddl_api.go:2748\ngithub.com/pingcap/tidb/pkg/ddl.(*ddl).CreateTable\n\tpkg/ddl/ddl_api.go:2587\ngithub.com/pingcap/tidb/pkg/executor.(*DDLExec).executeCreateTable\n\tpkg/executor/ddl.go:267\ngithub.com/pingcap/tidb/pkg/executor.(*DDLExec).Next\n\tpkg/executor/ddl.go:159\ngithub.com/pingcap/tidb/pkg/executor/internal/exec.Next\n\tpkg/executor/internal/exec/executor.go:314\ngithub.com/pingcap/tidb/pkg/executor.(*ExecStmt).next\n\tpkg/executor/adapter.go:1252\ngithub.com/pingcap/tidb/pkg/executor.(*ExecStmt).handleNoDelayExecutor\n\tpkg/executor/adapter.go:997\ngithub.com/pingcap/tidb/pkg/executor.(*ExecStmt).handleNoDelay\n\tpkg/executor/adapter.go:823\ngithub.com/pingcap/tidb/pkg/executor.(*ExecStmt).Exec\n\tpkg/executor/adapter.go:598\ngithub.com/pingcap/tidb/pkg/session.runStmt\n\tpkg/session/session.go:2380\ngithub.com/pingcap/tidb/pkg/session.(*session).ExecuteStmt\n\tpkg/session/session.go:2239\ngithub.com/pingcap/tidb/pkg/testkit.(*TestKit).ExecWithContext\n\tpkg/testkit/testkit.go:364\ngithub.com/pingcap/tidb/pkg/testkit.(*TestKit).MustExecWithContext\n\tpkg/testkit/testkit.go:136\ngithub.com/pingcap/tidb/pkg/testkit.(*TestKit).MustExec\n\tpkg/testkit/testkit.go:131\npkg/disttask/framework/framework_test_test.TestFrameworkWithQuery\n\tpkg/disttask/framework/framework_test.go:131\ntesting.tRunner\n\tGOROOT/src/testing/testing.go:1595\nruntime.goexit\n\tsrc/runtime/asm_amd64.s:1650"] [session="{\n  \"currDBName\": \"test\",\n  \"id\": 1,\n  \"status\": 2,\n  \"strictMode\": true,\n  \"user\": null\n}"]
    testkit.go:138: 
        	Error Trace:	pkg/testkit/testkit.go:138
        	            				pkg/testkit/testkit.go:131
        	            				pkg/disttask/framework/framework_test.go:131
        	Error:      	Received unexpected error:
        	            	[schema:1050]Table 't' already exists
        	            	github.com/pingcap/errors.AddStack
        	            		external/com_github_pingcap_errors/errors.go:174
        	            	github.com/pingcap/errors.Trace
        	            		external/com_github_pingcap_errors/juju_adaptor.go:15
        	            	github.com/pingcap/tidb/pkg/ddl.(*ddl).DoDDLJob
        	            		pkg/ddl/ddl.go:1299
        	            	github.com/pingcap/tidb/pkg/ddl.(*ddl).CreateTableWithInfo
        	            		pkg/ddl/ddl_api.go:2748
        	            	github.com/pingcap/tidb/pkg/ddl.(*ddl).CreateTable
        	            		pkg/ddl/ddl_api.go:2587
        	            	github.com/pingcap/tidb/pkg/executor.(*DDLExec).executeCreateTable
        	            		pkg/executor/ddl.go:267
        	            	github.com/pingcap/tidb/pkg/executor.(*DDLExec).Next
        	            		pkg/executor/ddl.go:159
        	            	github.com/pingcap/tidb/pkg/executor/internal/exec.Next
        	            		pkg/executor/internal/exec/executor.go:314
        	            	github.com/pingcap/tidb/pkg/executor.(*ExecStmt).next
        	            		pkg/executor/adapter.go:1252
        	            	github.com/pingcap/tidb/pkg/executor.(*ExecStmt).handleNoDelayExecutor
        	            		pkg/executor/adapter.go:997
        	            	github.com/pingcap/tidb/pkg/executor.(*ExecStmt).handleNoDelay
        	            		pkg/executor/adapter.go:823
        	            	github.com/pingcap/tidb/pkg/executor.(*ExecStmt).Exec
        	            		pkg/executor/adapter.go:598
        	            	github.com/pingcap/tidb/pkg/session.runStmt
        	            		pkg/session/session.go:2380
        	            	github.com/pingcap/tidb/pkg/session.(*session).ExecuteStmt
        	            		pkg/session/session.go:2239
        	            	github.com/pingcap/tidb/pkg/testkit.(*TestKit).ExecWithContext
        	            		pkg/testkit/testkit.go:364
        	            	github.com/pingcap/tidb/pkg/testkit.(*TestKit).MustExecWithContext
        	            		pkg/testkit/testkit.go:136
        	            	github.com/pingcap/tidb/pkg/testkit.(*TestKit).MustExec
        	            		pkg/testkit/testkit.go:131
        	            	pkg/disttask/framework/framework_test_test.TestFrameworkWithQuery
        	            		pkg/disttask/framework/framework_test.go:131
        	            	testing.tRunner
        	            		GOROOT/src/testing/testing.go:1595
        	            	runtime.goexit
        	            		src/runtime/asm_amd64.s:1650

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiDB version? (Required)

master

[ywqzzy]

#50055 before create table speed up, this case occurs. Don't know the root cause.

[tangenta]

Close by #50251