ddl: fix foreign key and check constraint verify records under async commit

[jiyfhust]

What problem does this PR solve?

Issue Number: close #48297

Problem Summary:

What is changed and how it works?

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No need to test

  • I checked and no code files have been changed.

Code review instead of tests.

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign benjamin2037 for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• pkg/ddl/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jiyfhust]

/ok-to-test

[tiprow]

Hi @jiyfhust. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 80.5818%. Comparing base (44f9430) to head (8dd0362).
Report is 2645 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #48524        +/-   ##
================================================
+ Coverage   71.3984%   80.5818%   +9.1833%     
================================================
  Files          1403       2676      +1273     
  Lines        406971     766530    +359559     
================================================
+ Hits         290571     617684    +327113     
- Misses        96445     124837     +28392     
- Partials      19955      24009      +4054     

Flag	Coverage Δ
integration	53.9315% <78.5714%> (?)
unit	79.4941% <100.0000%> (+8.0956%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	73.6130% <ø> (+19.6255%)	⬆️
parser	∅ <ø> (∅)
br	70.9575% <ø> (+17.8663%)	⬆️

[jiyfhust]

/retest

[jiyfhust]

/cc @CbcWestwolf @mjonss @Benjamin2037
Hi，brothers, this pr is easy to review, PTAL.

[mjonss]

Is it possible to add a test case showing the effect of this PR?

[jiyfhust]

Is it possible to add a test case showing the effect of this PR?

I do not have good idea to reproduce the steps to show the effects. But as async commit logic, we should wait before all txns use the old schema to finish then we can verify records, just as you known that exchange partition handle it the same way.

[jiyfhust]

em... Who could help for this pr?

[jiyfhust]

em... Who could help for this pr?

[jiyfhust]

For example, modify column null to not null, also use delayAsyncCommit.

[jiyfhust]

/cc @tangenta @zimulala PTAL

[ti-chi-bot]

@jiyfhust: GitHub didn't allow me to request PR reviews from the following users: PTAL.

Note that only pingcap members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @tangenta @zimulala PTAL

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jiyfhust]

The case:

1. set tidb_enable_metadata_lock off
2. set tidb_enable_async_commit on
3. execute a add check foreign key ddl
4. when ddl enter model.StateNone, a dml execute with the schemaVersion and commit use async commit.
5. while the commit is slow for some reason, and the ddl enter model.StateWriteOnly
6. the dml finish the prepare use a maxTx which is greate than the checkForeignKeyConstrain readTs
7. then the dml will be a potential bug for the check.

Is there some thing wrong above?
It is not eazy to reproduce the case, but logically it can occurrs.

right? not right?

[tiprow]

@jiyfhust: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
tiprow_fast_test	8dd0362	link	true	/test tiprow_fast_test

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[jiyfhust]

Related #23363.

[jiyfhust]

/cc @sticnarf @lance6716 @lcwangchao @cfzjywxk

[ti-chi-bot]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ti-chi-bot]

@jiyfhust: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-lightning-integration-test	8dd0362	link	true	/test pull-lightning-integration-test

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[YangKeao]

Let's merge this PR! Would you mind adding a test for it?

Ref https://github.com/pingcap/tidb/pull/56349/files, as I've tested, the test and failpoint in that PR should work.

It's also preferred to add more similar tests related to other situations (like check constraints, modify column...).

[jiyfhust]

Let's merge this PR! Would you mind adding a test for it?

I'm pleased for it，we can also work together to make this pr more complete.

[mjonss]

For tests with different DDL States, I wrote tests for partitioning DDL, that have multiple domains, so one can even tests new transactions started in previous schema version, see here. @YangKeao, @jiyfhust That may work as inspiration for writing tests for this PR as well?

[jiyfhust]

Let's merge this PR! Would you mind adding a test for it?

Ref https://github.com/pingcap/tidb/pull/56349/files, as I've tested, the test and failpoint in that PR should work.

It's also preferred to add more similar tests related to other situations (like check constraints, modify column...).

My poor virtual host with only 4G memory has diffcuity to reproduce the case, expecially to run txn tests you added.
I have checked the code again, except for the "foreign key", we can't verify there really need delayForAsyncCommit(they add a schema version to handle it, but not sure can handle all cases) , I think you can fix the 'foreign key' case as you pull request does, and we now can left the others aside.

For others, maybe case like this(this is only for note):

normal 2pc and mdl off:
1. prepare
2. get commit_ts
3. if commit_ts < curruent_schema_version 
        return error

however:
async 2pc:
1. tidb node1 prepare with schema v1, not return from tikv yet, it may table a long time but before `SafeWindow` expire
2. tidb ddl owner which is node2 run ddl and schema version enter v3
3. node1 prepare success alther it is not normal now(v1->v3, the schema version is too old)
4. node1 don't need check commit_ts with schema version becasuse it is async commit
5. finally transaction committed.