forked from opencurve/curve
-
Notifications
You must be signed in to change notification settings - Fork 2
/
curve_choas.txt
143 lines (128 loc) · 4.06 KB
/
curve_choas.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
*** Settings ***
Library Resources/keywords/base_operate.py
Library Resources/test/curve_base_test.py
Library Resources/swig/swig_operate.py
Library Collections
#Library RequestsLibrary
Library Process
#Library SSHLibrary
Library Resources/keywords/mythread.py
Library Resources/keywords/fault_inject.py
Library Resources/keywords/deploy.py
Library Resources/keywords/test_curve_stability.py
Library Resources/config/config.py
Suite Setup init failover cluster
Suite Teardown clean failover env
*** Variables ***
${fio_vdsize} 10
${vdbench_vdsize} 10
${vm_iops_limit} 8000
${chunk_num} 0
${WaitTimeBetweenIterations} 300
${WaitTimeBetweenFaults} 30
${ChoasLevel} 3
${MaxClusterOkTimeout} 1800
@{choas_level1} Create List
@{choas_level2} Create List
@{choas_level3} Create List
*** Test Cases ***
test one volume perf
[Tags] P1 base first release longtime perf
remove vm key
attach new vol 200 200
stop rwio
perf test
#启动大压力情况下的混沌测试：分等级进行随机故障注入。每次注入完成后恢复集群所有业务，目前设置100次的全流程注入
inject cluster chaos test
[Tags] P2 chaos longtime
init rw cloud disk
# add data disk
get level cases
${num} evaluate int(10)
init create curve vm ${num}
:FOR ${i} IN RANGE 10
log "启动第"${i}"轮故障"
${choas1} evaluate random.choice($choas_level1) random
log "开始启动一级故障"
do thrasher ${choas1}
sleep 30
${choas2} evaluate random.choice($choas_level2) random
log "开始启动二级故障"
do thrasher ${choas2}
sleep 30
${choas3} evaluate random.choice($choas_level3) random
log "开始启动三级故障"
do thrasher ${choas3}
sleep 30
clean env
start all services
wait cluster healthy
check nbd iops
${active_num} get all curvevm active num ${num}
${active_num} Convert To Integer ${active_num}
should be equal ${active_num} ${num}
check data consistency
check io error
sleep 300
END
[Teardown] clean curve data
*** Keywords ***
get level cases
${level} evaluate str("level1")
${choas} get level list ${level}
Set Global Variable ${choas_level1} ${choas}
${level} evaluate str("level2")
${choas} get level list ${level}
Set Global Variable ${choas_level2} ${choas}
${level} evaluate str("level3")
${choas} get level list ${level}
Set Global Variable ${choas_level3} ${choas}
start all services
test start etcd
sleep 10
test start mds
start retired and down chunkservers
test start snap
init failover cluster
clean nbd
destroy test env
${begin_num} get all chunk num
${begin_num} Convert To Integer ${begin_num}
Set Global Variable ${chunk_num} ${begin_num}
init abnormal test env
check chunkserver online
init abnormal test env
clean env
# destroy etcd
# destroy mds
# destroy snapshotclone server
# stop nebd
# drop all chunkserver dat
use ansible deploy
# install deb
# add config
# start abnormal test services
# create pool
# start nebd
restart cinder server
wait cinder server up
init rw cloud disk
map nbd
stop rwio
# write full disk ${fio_vdsize}
run rwio
clean kernel log
clean failover env
stop rwio
check io error
# stop scheduler
check copies consistency
clean nbd
delete nbd
sleep 10
clear RecycleBin
recycle chunk
# sleep 60
# ${final_num} get all chunk num
# ${final_num} Convert To Integer ${final_num}
# should be equal ${final_num} ${chunk_num}