add index for partition table find next partition ID failed

[aytrack]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. after use ctrl-c cancel reorganize partition as describe in CTRL-C reorganize partition report error canceling DDL job #42447
2. add index for the table

alter table t1 add index idx_kc (k, c);

2. What did you expect to see? (Required)

add index success

3. What did you see instead (Required)

1. add failed

TiDB root@127.0.0.1:db1> alter table t1 add index idx_kc (k, c);
You're about to run a destructive command.
Do you want to proceed? (y/n): y
Your call!
(1105, 'DDL job rollback, error msg: partition id not found 3898')

3. log

[2023/03/21 19:02:41.377 +08:00] [INFO] [ddl.go:1068] ["[ddl] start DDL job"] [job="ID:4391, Type:add index, State:queueing, SchemaState:none, SchemaID:324, TableID:3897, RowCount:0, ArgLen:6, start time: 2023-03-21 19:02:41.321 +0800 CST, Err:<nil>, ErrCount:0, SnapshotVersion:0, UniqueWarnings:0"] [query="alter table t1 add index idx_kc (k, c)"]
[2023/03/21 19:02:41.377 +08:00] [INFO] [ddl_worker.go:234] ["[ddl] add DDL jobs"] ["batch count"=1] [jobs="ID:4391, Type:add index, State:queueing, SchemaState:none, SchemaID:324, TableID:3897, RowCount:0, ArgLen:6, start time: 2023-03-21 19:02:41.321 +0800 CST, Err:<nil>, ErrCount:0, SnapshotVersion:0, UniqueWarnings:0; "] [table=true]
[2023/03/21 19:02:41.397 +08:00] [INFO] [ddl_worker.go:960] ["[ddl] run DDL job"] [worker="worker 5, tp add index"] [job="ID:4391, Type:add index, State:queueing, SchemaState:none, SchemaID:324, TableID:3897, RowCount:0, ArgLen:0, start time: 2023-03-21 19:02:41.321 +0800 CST, Err:<nil>, ErrCount:0, SnapshotVersion:0, UniqueWarnings:0"]
[2023/03/21 19:02:41.400 +08:00] [INFO] [index.go:613] ["[ddl] run add index job"] [job="ID:4391, Type:add index, State:running, SchemaState:none, SchemaID:324, TableID:3897, RowCount:0, ArgLen:6, start time: 2023-03-21 19:02:41.321 +0800 CST, Err:<nil>, ErrCount:0, SnapshotVersion:0, UniqueWarnings:0"] [indexInfo="{\"id\":2,\"idx_name\":{\"O\":\"idx_kc\",\"L\":\"idx_kc\"},\"tbl_name\":{\"O\":\"\",\"L\":\"\"},\"idx_cols\":[{\"name\":{\"O\":\"k\",\"L\":\"k\"},\"offset\":1,\"length\":-1},{\"name\":{\"O\":\"c\",\"L\":\"c\"},\"offset\":2,\"length\":-1}],\"state\":0,\"backfill_state\":0,\"comment\":\"\",\"index_type\":1,\"is_unique\":false,\"is_primary\":false,\"is_invisible\":false,\"is_global\":false,\"mv_index\":false}"]
[2023/03/21 19:02:41.419 +08:00] [INFO] [domain.go:224] ["diff load InfoSchema success"] [currentSchemaVersion=294] [neededSchemaVersion=295] ["start time"=3.02062ms] [gotSchemaVersion=295] [phyTblIDs="[3897,3898,3899,3900,3901,3902,3903]"] [actionTypes="[7,7,7,7,7,7,7]"]
[2023/03/21 19:02:41.422 +08:00] [INFO] [domain.go:795] ["mdl gets lock, update to owner"] [jobID=4391] [version=295]
.....


[2023/03/21 19:02:46.644 +08:00] [INFO] [index_cop.go:79] ["[ddl-ingest] finish a cop-request task"] [id=4]
[2023/03/21 19:02:46.644 +08:00] [INFO] [index_cop.go:84] ["[ddl-ingest] task is found in results"] [id=4] [task="taskID: 4, physicalTableID: 3898, range: [748000000000000f3a5f72800000000016e392, 748000000000000f3a5f7280000000001e847f], jobID: 4391"]
[2023/03/21 19:02:46.644 +08:00] [INFO] [backfilling.go:376] ["[ddl] backfill worker finish task"] [worker="backfill-worker 0, tp ingest index"] [task="taskID: 4, physicalTableID: 3898, range: [748000000000000f3a5f72800000000016e392, 748000000000000f3a5f7280000000001e847f], jobID: 4391"] ["added count"=771310] ["scan count"=771310] ["next key"=748000000000000f3a5f7280000000001e847f] ["take time"=3.80479982s]
[2023/03/21 19:02:46.653 +08:00] [INFO] [backfilling.go:617] ["[ddl] backfill workers successfully processed batch"] [element=ID:2,TypeKey:_idx_] ["total added count"=1999999] ["start key"=748000000000000f3a5f728000000000000001] ["next key"=748000000000000f3a5f7280000000001e847f] ["batch added count"=1999999] ["take time"=4.492926233s] []
[2023/03/21 19:02:46.653 +08:00] [INFO] [index_cop.go:217] ["[ddl-ingest] close cop-request sender pool"] ["results not handled"=0]
[2023/03/21 19:02:46.654 +08:00] [INFO] [backfilling.go:460] ["[ddl] backfill worker exit"] [worker="backfill-worker 0, tp ingest index"]
[2023/03/21 19:02:46.654 +08:00] [INFO] [backfilling.go:460] ["[ddl] backfill worker exit"] [worker="backfill-worker 2, tp ingest index"]
[2023/03/21 19:02:46.654 +08:00] [INFO] [backfilling.go:460] ["[ddl] backfill worker exit"] [worker="backfill-worker 1, tp ingest index"]
[2023/03/21 19:02:46.654 +08:00] [INFO] [backfilling.go:460] ["[ddl] backfill worker exit"] [worker="backfill-worker 3, tp ingest index"]
[2023/03/21 19:02:46.654 +08:00] [ERROR] [index.go:1972] ["[ddl] find next partition ID failed"] [table="{\"Columns\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":8,\"Flag\":515,\"Flen\":20,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2,\"GeneratedExpr\":null,\"DefaultExpr\":null},{\"id\":2,\"name\":{\"O\":\"k\",\"L\":\"k\"},\"offset\":1,\"origin_default\":null,\"origin_default_bit\":null,\"default\":\"0\",\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":9,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2,\"GeneratedExpr\":null,\"DefaultExpr\":null},{\"id\":3,\"name\":{\"O\":\"c\",\"L\":\"c\"},\"offset\":2,\"origin_default\":null,\"origin_default_bit\":null,\"default\":\"\",\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":254,\"Flag\":1,\"Flen\":120,\"Decimal\":0,\"Charset\":\"utf8mb4\",\"Collate\":\"utf8mb4_bin\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2,\"GeneratedExpr\":null,\"DefaultExpr\":null},{\"id\":4,\"name\":{\"O\":\"pad\",\"L\":\"pad\"},\"offset\":3,\"origin_default\":null,\"origin_default_bit\":null,\"default\":\"\",\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":254,\"Flag\":1,\"Flen\":60,\"Decimal\":0,\"Charset\":\"utf8mb4\",\"Collate\":\"utf8mb4_bin\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2,\"GeneratedExpr\":null,\"DefaultExpr\":null}],\"PublicColumns\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":8,\"Flag\":515,\"Flen\":20,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2,\"GeneratedExpr\":null,\"DefaultExpr\":null},{\"id\":2,\"name\":{\"O\":\"k\",\"L\":\"k\"},\"offset\":1,\"origin_default\":null,\"origin_default_bit\":null,\"default\":\"0\",\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":9,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2,\"GeneratedExpr\":null,\"DefaultExpr\":null},{\"id\":3,\"name\":{\"O\":\"c\",\"L\":\"c\"},\"offset\":2,\"origin_default\":null,\"origin_default_bit\":null,\"default\":\"\",\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":254,\"Flag\":1,\"Flen\":120,\"Decimal\":0,\"Charset\":\"utf8mb4\",\"Collate\":\"utf8mb4_bin\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2,\"GeneratedExpr\":null,\"DefaultExpr\":null},{\"id\":4,\"name\":{\"O\":\"pad\",\"L\":\"pad\"},\"offset\":3,\"origin_default\":null,\"origin_default_bit\":null,\"default\":\"\",\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":254,\"Flag\":1,\"Flen\":60,\"Decimal\":0,\"Charset\":\"utf8mb4\",\"Collate\":\"utf8mb4_bin\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2,\"GeneratedExpr\":null,\"DefaultExpr\":null}],\"VisibleColumns\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":8,\"Flag\":515,\"Flen\":20,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2,\"GeneratedExpr\":null,\"DefaultExpr\":null},{\"id\":2,\"name\":{\"O\":\"k\",\"L\":\"k\"},\"offset\":1,\"origin_default\":null,\"origin_default_bit\":null,\"default\":\"0\",\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":9,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2,\"GeneratedExpr\":null,\"DefaultExpr\":null},{\"id\":3,\"name\":{\"O\":\"c\",\"L\":\"c\"},\"offset\":2,\"origin_default\":null,\"origin_default_bit\":null,\"default\":\"\",\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":254,\"Flag\":1,\"Flen\":120,\"Decimal\":0,\"Charset\":\"utf8mb4\",\"Collate\":\"utf8mb4_bin\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2,\"GeneratedExpr\":null,\"DefaultExpr\":null},{\"id\":4,\"name\":{\"O\":\"pad\",\"L\":\"pad\"},\"offset\":3,\"origin_default\":null,\"origin_default_bit\":null,\"default\":\"\",\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":254,\"Flag\":1,\"Flen\":60,\"Decimal\":0,\"Charset\":\"utf8mb4\",\"Collate\":\"utf8mb4_bin\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2,\"GeneratedExpr\":null,\"DefaultExpr\":null}],\"HiddenColumns\":[],\"WritableColumns\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":8,\"Flag\":515,\"Flen\":20,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2,\"GeneratedExpr\":null,\"DefaultExpr\":null},{\"id\":2,\"name\":{\"O\":\"k\",\"L\":\"k\"},\"offset\":1,\"origin_default\":null,\"origin_default_bit\":null,\"default\":\"0\",\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":9,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2,\"GeneratedExpr\":null,\"DefaultExpr\":null},{\"id\":3,\"name\":{\"O\":\"c\",\"L\":\"c\"},\"offset\":2,\"origin_default\":null,\"origin_default_bit\":null,\"default\":\"\",\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":254,\"Flag\":1,\"Flen\":120,\"Decimal\":0,\"Charset\":\"utf8mb4\",\"Collate\":\"utf8mb4_bin\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2,\"GeneratedExpr\":null,\"DefaultExpr\":null},{\"id\":4,\"name\":{\"O\":\"pad\",\"L\":\"pad\"},\"offset\":3,\"origin_default\":null,\"origin_default_bit\":null,\"default\":\"\",\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":254,\"Flag\":1,\"Flen\":60,\"Decimal\":0,\"Charset\":\"utf8mb4\",\"Collate\":\"utf8mb4_bin\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2,\"GeneratedExpr\":null,\"DefaultExpr\":null}],\"FullHiddenColsAndVisibleColumns\":[{\"id\":1,\"name\":{\"O\":\"id\",\"L\":\"id\"},\"offset\":0,\"origin_default\":null,\"origin_default_bit\":null,\"default\":null,\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":8,\"Flag\":515,\"Flen\":20,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2,\"GeneratedExpr\":null,\"DefaultExpr\":null},{\"id\":2,\"name\":{\"O\":\"k\",\"L\":\"k\"},\"offset\":1,\"origin_default\":null,\"origin_default_bit\":null,\"default\":\"0\",\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":3,\"Flag\":9,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2,\"GeneratedExpr\":null,\"DefaultExpr\":null},{\"id\":3,\"name\":{\"O\":\"c\",\"L\":\"c\"},\"offset\":2,\"origin_default\":null,\"origin_default_bit\":null,\"default\":\"\",\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":254,\"Flag\":1,\"Flen\":120,\"Decimal\":0,\"Charset\":\"utf8mb4\",\"Collate\":\"utf8mb4_bin\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2,\"GeneratedExpr\":null,\"DefaultExpr\":null},{\"id\":4,\"name\":{\"O\":\"pad\",\"L\":\"pad\"},\"offset\":3,\"origin_default\":null,\"origin_default_bit\":null,\"default\":\"\",\"default_bit\":null,\"default_is_expr\":false,\"generated_expr_string\":\"\",\"generated_stored\":false,\"dependences\":null,\"type\":{\"Tp\":254,\"Flag\":1,\"Flen\":60,\"Decimal\":0,\"Charset\":\"utf8mb4\",\"Collate\":\"utf8mb4_bin\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false},\"state\":5,\"comment\":\"\",\"hidden\":false,\"change_state_info\":null,\"version\":2,\"GeneratedExpr\":null,\"DefaultExpr\":null}]}"] [error="partition id not found 3898"]
[2023/03/21 19:02:46.655 +08:00] [WARN] [reorg.go:234] ["[ddl] run reorg job done"] ["handled rows"=1999999] [error="partition id not found 3898"]
[2023/03/21 19:02:46.655 +08:00] [INFO] [backend.go:469] ["engine close start"] [engineTag=t1:2] [engineUUID=a0f82a3c-5421-5636-a09c-8e48474d5839]

4. What is your TiDB version? (Required)

+-----------------------------------------------------------+
| tidb_version()                                            |
+-----------------------------------------------------------+
| Release Version: v7.0.0                                   |
| Edition: Community                                        |
| Git Commit Hash: fc166ffae8576ac3efb2f5abac848d5edbaf2657 |
| Git Branch: heads/refs/tags/v7.0.0                        |
| UTC Build Time: 2023-03-20 07:08:59                       |
| GoVersion: go1.20.2                                       |
| Race Enabled: false                                       |
| TiKV Min Version: 6.2.0-alpha                             |
| Check Table Before Drop: false                            |
| Store: tikv                                               |
+-----------------------------------------------------------+