PXB-3241 : Assertion failure: os0file.cc:3416:!exists while taking ba…

[satya-bodapati]

…ckups with lock-ddl=REDUCED

Problem:

Backups taken with lock-ddl=reduced, prepare failed to complete.

Analysis:

When handling .ren files, the destination file name already exists and this causes assertion failure. See the below backup log

102: 2024-02-28T12:15:49.061631-00:00 1 [Note] [MY-011825] [Xtrabackup] DDL tracking : LSN: 73749548 create table ID: 788 Name: test/#sql-1fc79d_13#p#p3.ibd

423: 2024-02-28T12:15:50.121767-00:00 1 [Note] [MY-011825] [Xtrabackup] DDL tracking : LSN: 74312497 rename table ID: 425 From: test/tt_28_p#p#p3.ibd To: test/#sql2-1fc79d-13#p#p3.ibd

870: 2024-02-28T12:15:50.609015-00:00 2 [Note] [MY-011825] [Xtrabackup] Copying ./test/#sql-1fc79d_13#p#p3.ibd to /home/mohit.joshi/dbbackup_28_02_2024/full/test/#sql-1fc79d_13#p#p3.ibd

1337 2024-02-28T12:15:51.183699-00:00 1 [Note] [MY-011825] [Xtrabackup] DDL tracking : LSN: 74967007 rename table ID: 788 From: test/#sql-1fc79d_13#p#p3.ibd To: test/tt_28_p#p#p3.ibd

1491: 2024-02-28T12:15:51.398615-00:00 2 [Note] [MY-011825] [Xtrabackup] Copying ./test/tt_28_p#p#p3.ibd to /home/mohit.joshi/dbbackup_28_02_2024/full/test/tt_28_p#p#p3.ibd

2115:  2024-02-28T12:15:52.209645-00:00 1 [Note] [MY-011825] [Xtrabackup] DDL tracking : LSN: 75267178 delete table ID: 425 Name: test/#sql2-1fc79d-13#p#p3.ibd

Whats going on here?

Lets say we have partition p3 with space id 425. This is being altered. So partition algorithm does this:

1. create new copy of space_id: 788 (#sql1).
2. rename the existing table 425 to some temp taname (#sql2)
3. we copied the new copy space_id 788 (#sql1) to backup.
4. we also copied the space_id 425 with original name (p3).
5. Later we saw a rename file for the copied tablespace 788. (788.ren created with destination name as tt_28_p#p#p3.ibd

The rename file for space_id 425 is skipped, because we knowthat it is dropped. So only a .del file. Final state of backup is:

===

788 in backup with name #sql1
425 in backup with name p3
788.ren file-> 788 From: test/#sql-1fc79d_13#p#p3.ibd To: test/tt_28_p#p#p3.ibd 425.drop file

===

Now prepare starts to process .ren files
it tries to rename 788 from #sql1 to p3. but p3 already exists..

Fix:

we skip rename and other operations if we know that tablespace is going to be dropped. In the above example, we skipped 425.ren file.

So while preparing, we should handle the .del files first. Then we are applying all the consolidated operations in a way. Then .ren can be processed

[satya-bodapati]

./run.sh -s lockless -f -j 8
Using /home/satya/WORK/pxb-aibek/bld/storage/innobase/xtrabackup/test/var as test root
Detecting server version...
Running against MySQL 8.2.0-debug (InnoDB 8.2.0)
Using 'xtrabackup' as xtrabackup binary
Using 8 parallel workers
Running suite(s): lockless

==============================================================================
TEST                                   WORKER    RESULT     TIME(s) or COMMENT
------------------------------------------------------------------------------
encryption_alter_MK                      w8	[passed]    32
create_user                              w6	[passed]    35
alter_table_add_index                    w2	[passed]    68
alter_table_drop_index                   w4	[passed]    68
ddl_between_discovery_and_file_open      w7	[passed]    72
basic_operation                          w5	[passed]    75
encryption_force                         w8	[passed]    43
encryption_general_tablespace_error      w6	[passed]    58
lock_less_same_table                     w8	[passed]    40
incremental_rename                       w7	[passed]    48
incremental                              w5	[passed]    49
alter_table_drop_column                  w3	[passed]    132
optimize_table                           w6	[passed]    41
alter_table_add_column                   w1	[passed]    147
partial_backup                           w7	[passed]    36
encryption_single_tablespace             w4	[passed]    92
encryption_general_tablespace            w2	[passed]    94
xbcloud                                  w2	[skipped]   Requires XBCLOUD_CREDENTIALS
rename_table_different_database          w5	[passed]    44
rename_table_within_checkpoint_age       w6	[passed]    37
rename_table_multiple_times              w3	[passed]    41
parallel_copy                            w8	[passed]    59
truncate                                 w7	[passed]    41
rename_then_drop_table                   w1	[passed]    70
undo                                     w4	[passed]    167
==============================================================================
Spent 1591 of 342 seconds executing testcases

SUMMARY: 25 run, 24 successful, 1 skipped, 0 failed

Skipped tests:  xbcloud

See results/ for detailed output