-
Notifications
You must be signed in to change notification settings - Fork 0
/
atom.xml
1363 lines (1193 loc) · 257 KB
/
atom.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title>Have A Nice Day</title>
<link href="/atom.xml" rel="self"/>
<link href="https://caitianxiong.com/"/>
<updated>2020-02-02T01:33:49.179Z</updated>
<id>https://caitianxiong.com/</id>
<author>
<name>Leon</name>
</author>
<generator uri="http://hexo.io/">Hexo</generator>
<entry>
<title>K8s对接GlusterFS</title>
<link href="https://caitianxiong.com/2019/08/01/K8s%E5%AF%B9%E6%8E%A5GlusterFS/"/>
<id>https://caitianxiong.com/2019/08/01/K8s对接GlusterFS/</id>
<published>2019-07-31T16:00:00.000Z</published>
<updated>2020-02-02T01:33:49.179Z</updated>
<content type="html"><![CDATA[<p>Kubernetes存储功能模块(GlusterFS)部署文档</p>
<h2 id="1-部署架构"><a href="#1-部署架构" class="headerlink" title="1. 部署架构"></a>1. 部署架构</h2><p>Kubernetes —–> Heketi ——> GlusterFS集群<br>Heketi是GlusterFS集群的一个代理，提供REST API方式管理volume的功能<br>本文仅作一个记录，感谢我的同事@sixgod。<br><a id="more"></a></p>
<h2 id="2-环境准备（对接esb存储可忽略此步骤2-只需要提供集群ip及裸盘）"><a href="#2-环境准备（对接esb存储可忽略此步骤2-只需要提供集群ip及裸盘）" class="headerlink" title="2. 环境准备（对接esb存储可忽略此步骤2,只需要提供集群ip及裸盘）"></a>2. 环境准备（对接esb存储可忽略此步骤2,只需要提供集群ip及裸盘）</h2><h3 id="2-1-GlusterFS安装"><a href="#2-1-GlusterFS安装" class="headerlink" title="2.1 GlusterFS安装"></a>2.1 GlusterFS安装</h3><p><strong>每个存储节点</strong>需要提前安装GlusterFS客户端和服务。本次安装以3存储节点为例（glustefs版本为3.8.4）。<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line">yum install -y centos-release-gluster</div><div class="line">yum install -y glusterfs glusterfs-server glusterfs-fuse glusterfs-rdma</div><div class="line">systemctl start glusterd.service</div><div class="line">systemctl <span class="built_in">enable</span> glusterd.service</div></pre></td></tr></table></figure></p>
<h3 id="2-2-裸盘准备"><a href="#2-2-裸盘准备" class="headerlink" title="2.2 裸盘准备"></a>2.2 裸盘准备</h3><p>每个存储节点需要至少准备一块裸盘用来做存储设备。登录每个节点，通过<code>lsblk</code>命令检查是否存在可用裸盘。<br>本次安装均以<code>/dev/sdb</code>为例。<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># lsblk</span></div><div class="line">NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT</div><div class="line">fd0 2:0 1 4K 0 disk</div><div class="line">sda 8:0 0 100G 0 disk</div><div class="line">└─sda1 8:1 0 100G 0 part</div><div class="line"> └─centos-root 253:0 0 100G 0 lvm /</div><div class="line">sdb 8:16 0 100G 0 disk</div></pre></td></tr></table></figure></p>
<h2 id="3-Heketi部署"><a href="#3-Heketi部署" class="headerlink" title="3. Heketi部署"></a>3. Heketi部署</h2><h3 id="3-1-准备heketi-json配置文件"><a href="#3-1-准备heketi-json配置文件" class="headerlink" title="3.1 准备heketi.json配置文件"></a>3.1 准备heketi.json配置文件</h3><p>通过configmap准备heketi.json文件</p>
<h3 id="3-1-1-通过文件heketi-json创建configmap"><a href="#3-1-1-通过文件heketi-json创建configmap" class="headerlink" title="3.1.1 通过文件heketi.json创建configmap"></a>3.1.1 通过文件heketi.json创建configmap</h3><pre><code>kubectl create cm heketi --from-file=/root/heketi/heketi.json --namespace=kube-system
</code></pre><p>下面是一个heketi.json的配置文件示例</p>
<figure class="highlight json"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div><div class="line">61</div><div class="line">62</div></pre></td><td class="code"><pre><div class="line">{</div><div class="line"> <span class="attr">"_port_comment"</span>: <span class="string">"Heketi Server Port Number"</span>,</div><div class="line"> <span class="attr">"port"</span>: <span class="string">"8080"</span>,</div><div class="line"></div><div class="line"> <span class="attr">"_use_auth"</span>: <span class="string">"Enable JWT authorization. Please enable for deployment"</span>,</div><div class="line"> <span class="attr">"use_auth"</span>: <span class="literal">false</span>,</div><div class="line"></div><div class="line"> <span class="attr">"_jwt"</span>: <span class="string">"Private keys for access"</span>,</div><div class="line"> <span class="attr">"jwt"</span>: {</div><div class="line"> <span class="attr">"_admin"</span>: <span class="string">"Admin has access to all APIs"</span>,</div><div class="line"> <span class="attr">"admin"</span>: {</div><div class="line"> <span class="attr">"key"</span>: <span class="string">"My Secret"</span></div><div class="line"> },</div><div class="line"> <span class="attr">"_user"</span>: <span class="string">"User only has access to /volumes endpoint"</span>,</div><div class="line"> <span class="attr">"user"</span>: {</div><div class="line"> <span class="attr">"key"</span>: <span class="string">"My Secret"</span></div><div class="line"> }</div><div class="line"> },</div><div class="line"></div><div class="line"> <span class="attr">"_glusterfs_comment"</span>: <span class="string">"GlusterFS Configuration"</span>,</div><div class="line"> <span class="attr">"glusterfs"</span>: {</div><div class="line"> <span class="attr">"_executor_comment"</span>: [</div><div class="line"> <span class="string">"Execute plugin. Possible choices: mock, ssh"</span>,</div><div class="line"> <span class="string">"mock: This setting is used for testing and development."</span>,</div><div class="line"> <span class="string">" It will not send commands to any node."</span>,</div><div class="line"> <span class="string">"ssh: This setting will notify Heketi to ssh to the nodes."</span>,</div><div class="line"> <span class="string">" It will need the values in sshexec to be configured."</span>,</div><div class="line"> <span class="string">"kubernetes: Communicate with GlusterFS containers over"</span>,</div><div class="line"> <span class="string">" Kubernetes exec api."</span></div><div class="line"> ],</div><div class="line"> <span class="attr">"executor"</span>: <span class="string">"ssh"</span>,</div><div class="line"></div><div class="line"> <span class="attr">"_sshexec_comment"</span>: <span class="string">"SSH username and private key file information"</span>,</div><div class="line"> <span class="attr">"sshexec"</span>: {</div><div class="line"> <span class="attr">"keyfile"</span>: <span class="string">"/etc/heketi/ssh-key"</span>,</div><div class="line"> <span class="attr">"user"</span>: <span class="string">"root"</span>,</div><div class="line"> <span class="attr">"port"</span>: <span class="string">"22"</span>,</div><div class="line"> <span class="attr">"fstab"</span>: <span class="string">"/etc/fstab"</span></div><div class="line"> },</div><div class="line"></div><div class="line"> <span class="attr">"_kubeexec_comment"</span>: <span class="string">"Kubernetes configuration"</span>,</div><div class="line"> <span class="attr">"kubeexec"</span>: {</div><div class="line"> <span class="attr">"host"</span> :<span class="string">"https://kubernetes.host:8443"</span>,</div><div class="line"> <span class="attr">"cert"</span> : <span class="string">"/path/to/crt.file"</span>,</div><div class="line"> <span class="attr">"insecure"</span>: <span class="literal">false</span>,</div><div class="line"> <span class="attr">"user"</span>: <span class="string">"kubernetes username"</span>,</div><div class="line"> <span class="attr">"password"</span>: <span class="string">"password for kubernetes user"</span>,</div><div class="line"> <span class="attr">"namespace"</span>: <span class="string">"OpenShift project or Kubernetes namespace"</span>,</div><div class="line"> <span class="attr">"fstab"</span>: <span class="string">"Optional: Specify fstab file on node. Default is /etc/fstab"</span></div><div class="line"> },</div><div class="line"></div><div class="line"> <span class="attr">"_db_comment"</span>: <span class="string">"Database file name"</span>,</div><div class="line"> <span class="attr">"db"</span>: <span class="string">"/var/lib/heketi/heketi.db"</span>,</div><div class="line"></div><div class="line"> <span class="attr">"_loglevel_comment"</span>: [</div><div class="line"> <span class="string">"Set log level. Choices are:"</span>,</div><div class="line"> <span class="string">" none, critical, error, warning, info, debug"</span>,</div><div class="line"> <span class="string">"Default is warning"</span></div><div class="line"> ],</div><div class="line"> <span class="attr">"loglevel"</span> : <span class="string">"debug"</span></div><div class="line"> }</div><div class="line">}</div></pre></td></tr></table></figure>
<p>关键参数：</p>
<p>a. <code>"executor"</code>，设置heketi与存储节点交互方式，默认<code>ssh</code>即可。</p>
<p>b. <code>"_sshexec_commen"."keyfile"</code>，启用<code>ssh</code>交互方式时<strong>私钥</strong>的路径。</p>
<h3 id="3-2-准备sshkey"><a href="#3-2-准备sshkey" class="headerlink" title="3.2 准备sshkey"></a>3.2 准备sshkey</h3><p>heketi需要通过sshkey来进行存储节点的管理操作。为避免密码输入，需要准备一对公私钥。</p>
<h4 id="3-2-1-密钥生成"><a href="#3-2-1-密钥生成" class="headerlink" title="3.2.1 密钥生成"></a>3.2.1 密钥生成</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">mkdir /etc/heketi</div></pre></td></tr></table></figure>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">ssh-keygen -t rsa -q -f /etc/heketi/heketi_key -N ""</div></pre></td></tr></table></figure>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">chmod 600 /etc/heketi/heketi_key.pub</div></pre></td></tr></table></figure>
<h4 id="3-2-2-ssh公钥传递，这里只以一个节点为例"><a href="#3-2-2-ssh公钥传递，这里只以一个节点为例" class="headerlink" title="3.2.2 ssh公钥传递，这里只以一个节点为例"></a>3.2.2 ssh公钥传递，这里只以一个节点为例</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">ssh-copy-id -i /etc/heketi/heketi_key.pub root@192.168.75.175</div></pre></td></tr></table></figure>
<p>==注：（需要刚公钥拷贝到所有glusterfs节点）==</p>
<h4 id="3-2-3-验证是否能通过ssh密钥正常连接到glusterfs节点"><a href="#3-2-3-验证是否能通过ssh密钥正常连接到glusterfs节点" class="headerlink" title="3.2.3 验证是否能通过ssh密钥正常连接到glusterfs节点"></a>3.2.3 验证是否能通过ssh密钥正常连接到glusterfs节点</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">ssh -i /etc/heketi/heketi_key root@192.168.75.175</div></pre></td></tr></table></figure>
<h4 id="3-2-4-从sshkey私钥文件创建secret"><a href="#3-2-4-从sshkey私钥文件创建secret" class="headerlink" title="3.2.4 从sshkey私钥文件创建secret"></a>3.2.4 从sshkey私钥文件创建secret</h4><figure class="highlight plain"><figcaption><span>create secret generic heketi --from-file=ssh-key=/etc/heketi/heketi_key -n kube-system```</span></figcaption><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div><div class="line">61</div><div class="line">62</div><div class="line">63</div><div class="line">64</div><div class="line">65</div><div class="line">66</div><div class="line">67</div><div class="line">68</div><div class="line">69</div><div class="line">70</div><div class="line">71</div><div class="line">72</div><div class="line">73</div><div class="line">74</div><div class="line">75</div><div class="line">76</div><div class="line">77</div><div class="line">78</div><div class="line">79</div><div class="line">80</div><div class="line">81</div><div class="line">82</div><div class="line">83</div><div class="line">84</div><div class="line">85</div><div class="line">86</div><div class="line">87</div><div class="line">88</div><div class="line">89</div><div class="line">90</div></pre></td><td class="code"><pre><div class="line"></div><div class="line">### 3.3 部署heketi(建议采用方式2)</div><div class="line"></div><div class="line">#### 3.3.1 部署heketi（方式1）</div><div class="line">命令如下：</div><div class="line"></div><div class="line"> kubectl create -f heketi-deploy.yaml</div><div class="line"></div><div class="line">heketi-deploy.yaml文件如下：</div><div class="line"></div><div class="line"></div><div class="line">``` yaml</div><div class="line">---</div><div class="line">kind: Service</div><div class="line">apiVersion: v1</div><div class="line">metadata:</div><div class="line"> name: heketi</div><div class="line"> namespace: kube-system</div><div class="line"> labels:</div><div class="line"> glusterfs: heketi-service</div><div class="line"> heketi: service</div><div class="line"> annotations:</div><div class="line"> description: Exposes Heketi Service</div><div class="line">spec:</div><div class="line"> selector:</div><div class="line"> glusterfs: heketi-pod</div><div class="line"> ports:</div><div class="line"> - name: heketi</div><div class="line"> port: 8080</div><div class="line"> targetPort: 8080</div><div class="line">---</div><div class="line">kind: Deployment</div><div class="line">apiVersion: extensions/v1beta1</div><div class="line">metadata:</div><div class="line"> name: heketi</div><div class="line"> namespace: kube-system</div><div class="line"> labels:</div><div class="line"> glusterfs: heketi-deployment</div><div class="line"> heketi: deployment</div><div class="line"> annotations:</div><div class="line"> description: Defines how to deploy Heketi</div><div class="line">spec:</div><div class="line"> replicas: 1</div><div class="line"> template:</div><div class="line"> metadata:</div><div class="line"> name: heketi</div><div class="line"> labels:</div><div class="line"> glusterfs: heketi-pod</div><div class="line"> heketi: pod</div><div class="line"> spec:</div><div class="line"> containers:</div><div class="line"> - image: registry.paas/library/heketi:latest</div><div class="line"> imagePullPolicy: IfNotPresent</div><div class="line"> name: heketi</div><div class="line"> ports:</div><div class="line"> - containerPort: 8080</div><div class="line"> volumeMounts:</div><div class="line"> - name: db</div><div class="line"> mountPath: "/var/lib/heketi"</div><div class="line"> - name: config</div><div class="line"> mountPath: /etc/heketi/heketi.json</div><div class="line"> subPath: heketi.json</div><div class="line"> - name: ssh-key</div><div class="line"> mountPath: /etc/heketi/ssh-key</div><div class="line"> subPath: ssh-key</div><div class="line"> readinessProbe:</div><div class="line"> timeoutSeconds: 3</div><div class="line"> initialDelaySeconds: 3</div><div class="line"> httpGet:</div><div class="line"> path: "/hello"</div><div class="line"> port: 8080</div><div class="line"> livenessProbe:</div><div class="line"> timeoutSeconds: 3</div><div class="line"> initialDelaySeconds: 30</div><div class="line"> httpGet:</div><div class="line"> path: "/hello"</div><div class="line"> port: 8080</div><div class="line"> nodeSelector:</div><div class="line"> kubernetes.io/hostname: gdpaasn5</div><div class="line"> volumes:</div><div class="line"> - name: db</div><div class="line"> hostPath:</div><div class="line"> type: DirectoryOrCreate</div><div class="line"> path: /var/lib/heketi</div><div class="line"> - name: config</div><div class="line"> configMap:</div><div class="line"> name: heketi</div><div class="line"> - name: ssh-key</div><div class="line"> secret:</div><div class="line"> secretName: heketi</div></pre></td></tr></table></figure>
<p>==kubernetes.io/hostname: gdpaasn5要修改成具体调度节点==.</p>
<p>获取cluster ip<br><code>kubectl get svc -n kube-system | grep heketi</code>，执行<code>curl cluster-ip:8080/hello</code>看是否正常返回<code>Hello from Heketi</code></p>
<h4 id="3-3-2-设置topoloy"><a href="#3-3-2-设置topoloy" class="headerlink" title="3.3.2 设置topoloy"></a>3.3.2 设置topoloy</h4><p>第一步：进入到heketi的pod，执行下面命令<code>heketi-cli topology load -j topology.json</code>, topoloy.json文件如下：</p>
<figure class="highlight json"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div></pre></td><td class="code"><pre><div class="line">{</div><div class="line"> <span class="attr">"clusters"</span>: [</div><div class="line"> {</div><div class="line"> <span class="attr">"nodes"</span>: [</div><div class="line"> {</div><div class="line"> <span class="attr">"node"</span>: {</div><div class="line"> <span class="attr">"hostnames"</span>: {</div><div class="line"> <span class="attr">"manage"</span>: [</div><div class="line"> <span class="string">"192.168.1.19"</span></div><div class="line"> ],</div><div class="line"> <span class="attr">"storage"</span>: [</div><div class="line"> <span class="string">"192.168.1.19"</span></div><div class="line"> ]</div><div class="line"> },</div><div class="line"> <span class="attr">"zone"</span>: <span class="number">1</span></div><div class="line"> },</div><div class="line"> <span class="attr">"devices"</span>: [</div><div class="line"> <span class="string">"/dev/sdb"</span></div><div class="line"> ]</div><div class="line"> },</div><div class="line"> {</div><div class="line"> <span class="attr">"node"</span>: {</div><div class="line"> <span class="attr">"hostnames"</span>: {</div><div class="line"> <span class="attr">"manage"</span>: [</div><div class="line"> <span class="string">"192.168.1.20"</span></div><div class="line"> ],</div><div class="line"> <span class="attr">"storage"</span>: [</div><div class="line"> <span class="string">"192.168.1.20"</span></div><div class="line"> ]</div><div class="line"> },</div><div class="line"> <span class="attr">"zone"</span>: <span class="number">2</span></div><div class="line"> },</div><div class="line"> <span class="attr">"devices"</span>: [</div><div class="line"> <span class="string">"/dev/sdb"</span></div><div class="line"> ]</div><div class="line"> },</div><div class="line"> {</div><div class="line"> <span class="attr">"node"</span>: {</div><div class="line"> <span class="attr">"hostnames"</span>: {</div><div class="line"> <span class="attr">"manage"</span>: [</div><div class="line"> <span class="string">"192.168.1.21"</span></div><div class="line"> ],</div><div class="line"> <span class="attr">"storage"</span>: [</div><div class="line"> <span class="string">"192.168.1.21"</span></div><div class="line"> ]</div><div class="line"> },</div><div class="line"> <span class="attr">"zone"</span>: <span class="number">3</span></div><div class="line"> },</div><div class="line"> <span class="attr">"devices"</span>: [</div><div class="line"> <span class="string">"/dev/sdb"</span></div><div class="line"> ]</div><div class="line"> }</div><div class="line"> ]</div><div class="line"> }</div><div class="line"> ]</div><div class="line">}</div></pre></td></tr></table></figure>
<p>其中IP地址为glusterfs集群地址（注：glusterfs集群盘必须是裸盘），操作方法如下所示：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div></pre></td><td class="code"><pre><div class="line">[root@heketi-69c758c9cc-7v97t heketi]# heketi-cli topology load --json=topology.json</div><div class="line">Creating cluster ... ID: fe3b97728a0d0267a247c28c37645288</div><div class="line"> Allowing file volumes on cluster.</div><div class="line"> Allowing block volumes on cluster.</div><div class="line"> Creating node 10.142.113.145 ... ID: 2f32bdac26a285e0b9a5438d83f0796b</div><div class="line"> Adding device /dev/sdb ... OK</div><div class="line"> Creating node 10.142.113.144 ... ID: 07cc108c81f3e3669070685951e67391</div><div class="line"> Adding device /dev/sdb ... OK</div></pre></td></tr></table></figure></p>
<p>通过查看拓扑看是否成功<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">heketi-cli topology info 查看拓扑结构</div><div class="line">heketi-cli db dump --json 查看heketi.db数据</div></pre></td></tr></table></figure></p>
<p>注意：</p>
<p>这一步在创建集群的时候可能出现有node节点连接不成功的情况，可使用<code>heketi-cli node add xxx</code>等命令进行单节点或者单设备添加工作。</p>
<h4 id="3-3-3-heketi数据高可用-方式2"><a href="#3-3-3-heketi数据高可用-方式2" class="headerlink" title="3.3.3 heketi数据高可用 (方式2)"></a>3.3.3 heketi数据高可用 (方式2)</h4><p>之前的heketi部署将db的数据保存在host的目录上，使用node selector绑定到指定节点。<br>可以将db数据也存到gluster集群上。</p>
<ul>
<li>步骤1： 事先在gluster集群创建好一个volume，此volume不能创建在要加载的裸盘上。 操作如下：<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">gluster volume create heketi-xj replica 2 192.168.1.20:/mnt/xj 192.168.1.21:/mnt/xj force</div></pre></td></tr></table></figure>
</li>
</ul>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">gluster volume start heketi-xj</div></pre></td></tr></table></figure>
<p>==注： k8s集群的slave节点要装glusterfs-client==</p>
<ul>
<li>步骤2: 创建endpoint和service，<code>kubectl create -f endpoint-gluster.yaml</code>， endpoint-gluster.yaml文件如下：</li>
</ul>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div></pre></td><td class="code"><pre><div class="line">---</div><div class="line">apiVersion: v1</div><div class="line">kind: Endpoints</div><div class="line">metadata:</div><div class="line"> name: glusterfs-cluster</div><div class="line">subsets:</div><div class="line"> - addresses:</div><div class="line"> - ip: 192.168.1.20</div><div class="line"> ports:</div><div class="line"> - port: 1</div><div class="line"> protocol: TCP</div><div class="line"> - addresses:</div><div class="line"> - ip: 192.168.1.21</div><div class="line"> ports:</div><div class="line"> - port: 1</div><div class="line"> protocol: TCP</div><div class="line"> - addresses:</div><div class="line"> - ip: 192.168.1.19</div><div class="line"> ports:</div><div class="line"> - port: 1</div><div class="line"> protocol: TCP</div><div class="line">---</div><div class="line"></div><div class="line">kind: Service</div><div class="line">apiVersion: v1</div><div class="line">metadata:</div><div class="line"> name: glusterfs-cluster</div><div class="line">spec:</div><div class="line"> ports:</div><div class="line"> - port: 1</div></pre></td></tr></table></figure>
<ul>
<li>步骤3：deployment yaml文件更改如下：</li>
</ul>
<figure class="highlight yaml"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div><div class="line">61</div><div class="line">62</div><div class="line">63</div><div class="line">64</div><div class="line">65</div><div class="line">66</div><div class="line">67</div><div class="line">68</div><div class="line">69</div><div class="line">70</div><div class="line">71</div><div class="line">72</div><div class="line">73</div><div class="line">74</div><div class="line">75</div><div class="line">76</div></pre></td><td class="code"><pre><div class="line"><span class="meta">---</span></div><div class="line"><span class="attr">kind:</span> Service</div><div class="line"><span class="attr">apiVersion:</span> v1</div><div class="line"><span class="attr">metadata:</span></div><div class="line"><span class="attr"> name:</span> heketi</div><div class="line"><span class="attr"> labels:</span></div><div class="line"><span class="attr"> glusterfs:</span> heketi-service</div><div class="line"><span class="attr"> heketi:</span> service</div><div class="line"><span class="attr"> annotations:</span></div><div class="line"><span class="attr"> description:</span> Exposes Heketi Service</div><div class="line"><span class="attr">spec:</span></div><div class="line"><span class="attr"> selector:</span></div><div class="line"><span class="attr"> glusterfs:</span> heketi-pod</div><div class="line"><span class="attr"> ports:</span></div><div class="line"><span class="attr"> - name:</span> heketi</div><div class="line"><span class="attr"> port:</span> <span class="number">8080</span></div><div class="line"><span class="attr"> targetPort:</span> <span class="number">8080</span></div><div class="line"><span class="meta">---</span></div><div class="line"><span class="attr">kind:</span> Deployment</div><div class="line"><span class="attr">apiVersion:</span> extensions/v1beta1</div><div class="line"><span class="attr">metadata:</span></div><div class="line"><span class="attr"> name:</span> heketi</div><div class="line"><span class="attr"> labels:</span></div><div class="line"><span class="attr"> glusterfs:</span> heketi-deployment</div><div class="line"><span class="attr"> heketi:</span> deployment</div><div class="line"><span class="attr"> annotations:</span></div><div class="line"><span class="attr"> description:</span> Defines how to deploy Heketi</div><div class="line"><span class="attr">spec:</span></div><div class="line"><span class="attr"> replicas:</span> <span class="number">1</span></div><div class="line"><span class="attr"> template:</span></div><div class="line"><span class="attr"> metadata:</span></div><div class="line"><span class="attr"> name:</span> heketi</div><div class="line"><span class="attr"> labels:</span></div><div class="line"><span class="attr"> glusterfs:</span> heketi-pod</div><div class="line"><span class="attr"> heketi:</span> pod</div><div class="line"><span class="attr"> spec:</span></div><div class="line"><span class="attr"> containers:</span></div><div class="line"><span class="attr"> - image:</span> registry.paas/library/heketi:latest</div><div class="line"><span class="attr"> imagePullPolicy:</span> IfNotPresent</div><div class="line"><span class="attr"> name:</span> heketi</div><div class="line"><span class="attr"> ports:</span></div><div class="line"><span class="attr"> - containerPort:</span> <span class="number">8080</span></div><div class="line"><span class="attr"> volumeMounts:</span></div><div class="line"><span class="attr"> - name:</span> db</div><div class="line"><span class="attr"> mountPath:</span> <span class="string">"/var/lib/heketi"</span></div><div class="line"><span class="attr"> - name:</span> config</div><div class="line"><span class="attr"> mountPath:</span> /etc/heketi/heketi.json</div><div class="line"><span class="attr"> subPath:</span> heketi.json</div><div class="line"><span class="attr"> - name:</span> ssh-key</div><div class="line"><span class="attr"> mountPath:</span> /etc/heketi/ssh-key</div><div class="line"><span class="attr"> subPath:</span> ssh-key</div><div class="line"><span class="attr"> readinessProbe:</span></div><div class="line"><span class="attr"> timeoutSeconds:</span> <span class="number">3</span></div><div class="line"><span class="attr"> initialDelaySeconds:</span> <span class="number">3</span></div><div class="line"><span class="attr"> httpGet:</span></div><div class="line"><span class="attr"> path:</span> <span class="string">"/hello"</span></div><div class="line"><span class="attr"> port:</span> <span class="number">8080</span></div><div class="line"><span class="attr"> livenessProbe:</span></div><div class="line"><span class="attr"> timeoutSeconds:</span> <span class="number">3</span></div><div class="line"><span class="attr"> initialDelaySeconds:</span> <span class="number">30</span></div><div class="line"><span class="attr"> httpGet:</span></div><div class="line"><span class="attr"> path:</span> <span class="string">"/hello"</span></div><div class="line"><span class="attr"> port:</span> <span class="number">8080</span></div><div class="line"><span class="attr"> nodeSelector:</span></div><div class="line"> kubernetes.io/hostname: gdpaasn5</div><div class="line"><span class="attr"> volumes:</span></div><div class="line"><span class="attr"> - name:</span> db</div><div class="line"><span class="attr"> glusterfs:</span></div><div class="line"><span class="attr"> endpoints:</span> glusterfs-cluster</div><div class="line"><span class="attr"> path:</span> heketi-xj //事先创建好的gluster volume</div><div class="line"><span class="attr"> - name:</span> config</div><div class="line"><span class="attr"> configMap:</span></div><div class="line"><span class="attr"> name:</span> heketi</div><div class="line"><span class="attr"> - name:</span> ssh-key</div><div class="line"><span class="attr"> secret:</span></div><div class="line"><span class="attr"> secretName:</span> heketi</div></pre></td></tr></table></figure>
<ul>
<li>步骤4 ==同3.3.2一样，设置topoloy==</li>
</ul>
<h3 id="5-升级注意"><a href="#5-升级注意" class="headerlink" title="5. 升级注意"></a>5. 升级注意</h3><p>==如果升级已经存在heketi应用，需要保存好heketi.db数据。数据存储在heketi的pod /var/lib/heketi目录下==</p>
]]></content>
<summary type="html">
<p>Kubernetes存储功能模块(GlusterFS)部署文档</p>
<h2 id="1-部署架构"><a href="#1-部署架构" class="headerlink" title="1. 部署架构"></a>1. 部署架构</h2><p>Kubernetes —–&gt; Heketi ——&gt; GlusterFS集群<br>Heketi是GlusterFS集群的一个代理，提供REST API方式管理volume的功能<br>本文仅作一个记录，感谢我的同事@sixgod。<br>
</summary>
<category term="Kubernetes" scheme="https://caitianxiong.com/tags/Kubernetes/"/>
<category term="K8s" scheme="https://caitianxiong.com/tags/K8s/"/>
<category term="GlussterFS" scheme="https://caitianxiong.com/tags/GlussterFS/"/>
</entry>
<entry>
<title>Ingress参数优化</title>
<link href="https://caitianxiong.com/2018/05/04/Ingress%E5%8F%82%E6%95%B0%E4%BC%98%E5%8C%96/"/>
<id>https://caitianxiong.com/2018/05/04/Ingress参数优化/</id>
<published>2018-05-03T16:00:00.000Z</published>
<updated>2020-02-02T01:38:32.767Z</updated>
<content type="html"><![CDATA[<p>结合项目上生产使用经验，研发给出的参数调优建议。本篇仅作记录，及做一个分享，版权归我的好兄弟@董琪。<br><a id="more"></a></p>
<h1 id="内核优化"><a href="#内核优化" class="headerlink" title="内核优化"></a>内核优化</h1><h2 id="kernel-pid-max-600000"><a href="#kernel-pid-max-600000" class="headerlink" title="kernel.pid_max = 600000"></a>kernel.pid_max = 600000</h2><p>之前江苏环境 nginx 产生大量 close-wait 就是由于该参数设置过小，具体参考附件【BCLinux系统出现大量CLOSE_WAIT连接问题分析报告】</p>
<h2 id="fs-aio-max-nr-1065535"><a href="#fs-aio-max-nr-1065535" class="headerlink" title="fs.aio-max-nr = 1065535"></a>fs.aio-max-nr = 1065535</h2><p>北京环境 nginx 报错 io_setup() failed (11: resource temporarily unavailable)<br>由于 nginx 使用 linux 原生的异步 io，所以当 <code>/proc/sys/fs/aio-max-nr</code> （默认 65535） 过小时，出现以上错误。<code>/proc/sys/fs/aio-nr</code> 用于显示当前 <code>aio</code> 请求数。<br>解决办法：设置 <code>aio-max-nr</code> 为 1048576</p>
<h2 id="net-ipv4-tcp-tw-recycle-1"><a href="#net-ipv4-tcp-tw-recycle-1" class="headerlink" title="net.ipv4.tcp_tw_recycle=1"></a>net.ipv4.tcp_tw_recycle=1</h2><p>表示开启TCP连接中TIME-WAIT sockets的快速回收，默认为0，表示关闭。</p>
<h2 id="net-ipv4-tcp-tw-reuse-1"><a href="#net-ipv4-tcp-tw-reuse-1" class="headerlink" title="net.ipv4.tcp_tw_reuse=1"></a>net.ipv4.tcp_tw_reuse=1</h2><p>表示开启重用。允许将TIME-WAIT sockets重新用于新的TCP连接，默认为0，表示关闭；</p>
<h2 id="net-ipv4-tcp-timestamps-1"><a href="#net-ipv4-tcp-timestamps-1" class="headerlink" title="net.ipv4.tcp_timestamps=1"></a>net.ipv4.tcp_timestamps=1</h2><p>TCP时间戳（会在TCP包头增加12个字节），以一种比重发超时更精确的方法（参考RFC 1323）来启用对RTT 的计算，为实现更好的性能应该启用这个选项。<br>更重要的是，启用TCP时间戳可以防止回绕，因为TCP头的序列号字段定义为32字节，在高速网络环境里，TCP序列号容易产生回绕，既接收者最新收到分片的序列号比之前到达的序列号要小，导致不正确的丢包行为。<br>在1Gbps网络带宽环境中，可能会在17秒内会发生TCP序列号回绕，而在10Gbps网络带宽环境中，发生回绕的时间甚至缩短到1.7秒，所以在高速网络环境里，必须要开启TCP时间戳选项。<br>首先确认TCP时间戳是否启用：<br><code>sysctl net.ipv4.tcp_timestamps</code><br><code>net.ipv4.tcp_timestamps = 0</code><br>如果没有启用该项，建议开启：<br><code>sysctl -w net.ipv4.tcp_timestamps=1</code></p>
<h2 id="net-ipv4-tcp-fin-timeout-20"><a href="#net-ipv4-tcp-fin-timeout-20" class="headerlink" title="net.ipv4.tcp_fin_timeout=20"></a>net.ipv4.tcp_fin_timeout=20</h2><p>TCP FIN Timeout定义了对于本端断开的socket连接，TCP保持在FIN-WAIT-2状态的时间（秒），而对方可能会断开连接或一直不结束连接或不可预料的进程死亡。其默认值为60有点偏高，建议调整该值至20或者30，以便TCP连接尽快关闭并释放资源：<br><code>sysctl -w net.ipv4.tcp_fin_timeout=20</code></p>
<h2 id="net-core-somaxconn-65535"><a href="#net-core-somaxconn-65535" class="headerlink" title="net.core.somaxconn=65535"></a>net.core.somaxconn=65535</h2><p>全连接队列长度，对应 listen 函数中 backlog （监听队列的长度）的大小，和 <code>tcp_max_syn_backlog</code> 定义的半连接队列相对应</p>
<h2 id="net-ipv4-tcp-max-syn-backlog-65535"><a href="#net-ipv4-tcp-max-syn-backlog-65535" class="headerlink" title="net.ipv4.tcp_max_syn_backlog=65535"></a>net.ipv4.tcp_max_syn_backlog=65535</h2><p>记录的那些尚未收到客户端确认信息的连接请求的最大值。对于超过128M内存的系统而言，缺省值是1024，低于128M小内存的系统则是128。<br>SYN Flood攻击利用TCP协议散布握手的缺陷，伪造虚假源IP地址发送大量TCP-SYN半打开连接到目标系统，最终导致目标系统Socket队列资源耗尽而无法接受新的连接。为了应付这种攻击，现代Unix系统中普遍采用多连接队列处理的方式来缓冲(而不是解决)这种攻击，是用一个基本队列处理正常的完全连接应用(Connect()和Accept() )，是用另一个队列单独存放半打开连接。<br>这种双队列处理方式和其他一些系统内核措施(例如Syn-Cookies/Caches)联合应用时，能够比较有效的缓解小规模的SYN Flood攻击(事实证明<1000p s)加大syn队列长度可以容纳更多等待连接的网络连接数，一般遭受syn="" flood攻击的网站，都存在大量syn_recv状态，所以调大tcp_max_syn_backlog值能增加抵抗syn攻击的能力。="" 如果服务器端向客户端发送syn＋ack后，客户端不返回ack，则服务器保持半连接(syn_recv)状态：="" <figure="" class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">netstat -np | grep SYN_RECV</div><div class="line">tcp 0 0 0.0.0.0:9999 127.0.0.0.1:5334 SYN_RECV -</div></pre></td></tr></table></1000p></p>
<p>若队列中的连接均处于半连接状态，服务器将不能处理正常的请求，syn泛洪攻击(syn flood)就是利用这个特点完成DoS(拒绝服务攻击)。<br>当连接数超过队列长度backlog时，超出的连接也保持为半连接状态，直到数量达到内核参数tcp_max_syn_backlog值，超出该值的连接请求将被丢弃：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">linux # sysctl -a | grep tcp_max_syn</div><div class="line">net.ipv4.tcp_max_syn_backlog = 1024</div></pre></td></tr></table></figure></p>
<h2 id="net-ipv4-tcp-syncookies-0"><a href="#net-ipv4-tcp-syncookies-0" class="headerlink" title="net.ipv4.tcp_syncookies = 0"></a>net.ipv4.tcp_syncookies = 0</h2><p>apache ab压力测试报错（apr_socket_recv: Connection reset by peer (104)）<br>查看应用服务器和数据库均未报错，连接被重置，apr_socket_recv这个是操作系统内核的一个参数，在高并发的情况下，内核会认为系统受到了SYN flood攻击，会发送cookies（possible SYN flooding on port 80. Sending cookies），这样会减慢影响请求的速度，所以在应用服务武器上设置下这个参数为0禁用系统保护就可以进行大并发测试了：</p>
<h2 id="net-core-netdev-max-backlog-300000"><a href="#net-core-netdev-max-backlog-300000" class="headerlink" title="net.core.netdev_max_backlog=300000"></a>net.core.netdev_max_backlog=300000</h2><p>在内核协议栈接收处理数据包之前，数据默认是存放在内核队列里的。<code>netdev_max_backlog</code> 定义了在当接口收到数据包的速率快于内核处理速率时，允许发送到队列数据包的最大数目，对于高负载的网络其默认值显然是不够的。<br>首先查看netdev_max_backlog的默认值：<br><code>sysctl net.core.netdev_max_backlog</code><br><code>net.core. netdev_max_backlog = 1000</code><br>对于10Gbps带宽的网络环境，建议调整其值如下：<br><code>sysctl -w net.core.netdev_max_backlog=300000</code></p>
<h2 id="net-ipv4-tcp-keepalive-time-10"><a href="#net-ipv4-tcp-keepalive-time-10" class="headerlink" title="net.ipv4.tcp_keepalive_time=10"></a>net.ipv4.tcp_keepalive_time=10</h2><p>TCP发送keepalive探测消息的间隔时间（秒），用于确认TCP连接是否有效。</p>
<h2 id="net-ipv4-tcp-keepalive-probes-2"><a href="#net-ipv4-tcp-keepalive-probes-2" class="headerlink" title="net.ipv4.tcp_keepalive_probes=2"></a>net.ipv4.tcp_keepalive_probes=2</h2><p>在认定TCP连接失效之前，最多发送多少个keepalive探测消息。</p>
<h2 id="net-ipv4-tcp-keepalive-intvl-2"><a href="#net-ipv4-tcp-keepalive-intvl-2" class="headerlink" title="net.ipv4.tcp_keepalive_intvl=2"></a>net.ipv4.tcp_keepalive_intvl=2</h2><p>探测消息未获得响应时，重发该消息的间隔时间（秒）。</p>
<h2 id="net-ipv4-tcp-mem-94500000-915000000-927000000"><a href="#net-ipv4-tcp-mem-94500000-915000000-927000000" class="headerlink" title="net.ipv4.tcp_mem=94500000 915000000 927000000"></a>net.ipv4.tcp_mem=94500000 915000000 927000000</h2><p>除了用于缓冲收发数据包的 tcp_wmem 和 tcp_rmem，对于每个socket，内核还要分配一些数据结构用于保持连接状态，内核对tcp层可使用的内存大小进行了限制，所以有了 tcp_mem：<br>tcp_mem 有 3 个 INTEGER 变量：low, pressure, high</p>
<ul>
<li>low：当 TCP 使用了低于该值的内存页面数时，TCP 没有内存压力，TCP 不会考虑释放内存。(理<br>想情况下，这个值应与指定给 tcp_wmem 的第 2 个值相匹配。这第 2 个值表明，最大页面大小乘以最大<br>并发请求数除以页大小 (131072*300/4096)</li>
<li>pressure：当 TCP 使用了超过该值的内存页面数量时，TCP 试图稳定其内存使用，进入 pressure<br>模式，当内存消耗低于 low 值时则退出 pressure 状态。(理想情况下这个值应该是 TCP 可以使用的总缓<br>冲区大小的最大值(204800*300/4096)</li>
<li>high：允许所有 TCP Sockets 用于排队缓冲数据报的页面量。如果超过这个值，TCP 连接将被拒绝，<br>这就是为什么不要令其过于保守(512000*300/4096)的原因了。在这种情况下，提供的价值很大，它能<br>处理很多连接，是所预期的 2.5 倍；或者使现有连接能够传输 2.5 倍的数据。<br>一般情况下这些值是在系统启动时根据系统内存数量计算得到的。<blockquote>
<p>以上值以页为单位，分别对应最小值、压力值和最大值，并在系统启动、tcp栈初始化时根据内存总量设定。通过proc提供的接口，我们可以查到tcp已用的内存页数：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line">>linux # cat /proc/net/sockstat</div><div class="line">>sockets : used 91</div><div class="line">>TCP : inuse 8 orphan 0 tw 11 alloc 13 mem 2</div><div class="line">></div></pre></td></tr></table></figure>
</blockquote>
</li>
</ul>
<h2 id="net-ipv4-tcp-rmem-4096-87380-134217728"><a href="#net-ipv4-tcp-rmem-4096-87380-134217728" class="headerlink" title="net.ipv4.tcp_rmem=4096 87380 134217728"></a>net.ipv4.tcp_rmem=4096 87380 134217728</h2><p>内核为tcp socket预留的用于读取数据包的缓冲区大小<br>TCP 的性能取决于几个方面的因素。两个最重要的因素是链接带宽（link bandwidth）（报文在网络上传输的速率）和 往返时间（round-trip time） 或 RTT（发送报文与接收到另一端的响应之间的延时）。这两个值确定了称为 Bandwidth Delay Product（BDP）的内容。<br>BDP 给出了一种简单的方法来计算理论上最优的 TCP socket 缓冲区大小（其中保存了排队等待传输和等待应用程序接收的数据）。如果缓冲区太小，那么 TCP 窗口就不能完全打开，这会对性能造成限制。如果缓冲区太大，那么宝贵的内存资源就会造成浪费。如果您设置的缓冲区大小正好合适，那么就可以完全利用可用的带宽，计算公式如下：<br>BDP = link_bandwidth <em> RTT<br>RTT的值可以通过ping获取到一个平均值，例如，在10Gbps宽带网络里，如果RTT值为100ms，那么BDP就是：<br>10Gps </em> 100ms = (10<em>2^30)bits </em> 0.1s / 8 = 134217728 Bytes.<br>TCP套接字缓存参数主要包括：<code>net.ipv4.tcp_rmem</code>，<code>net.ipv4.tcp_wmem</code><br>这些参数值主要包括3个字段，分别代表最小、默认、最大的TCP接收和发送缓存大小。在TCP缓存成为性能瓶颈的时候，建议调整该参数的最大发送缓存大小至BDP的值，例如10Gbps业务环境的配置参考如下：<br><code>sysctl -w net.ipv4.tcp_rmem="40960 873800 1342177280"</code><br><code>sysctl -w net.ipv4.tcp_wmem="40960 655360 1342177280"</code></p>
<h2 id="net-ipv4-tcp-wmem-4096-87380-134217728"><a href="#net-ipv4-tcp-wmem-4096-87380-134217728" class="headerlink" title="net.ipv4.tcp_wmem=4096 87380 134217728"></a>net.ipv4.tcp_wmem=4096 87380 134217728</h2><p>如上</p>
<h2 id="net-core-wmem-default-68388608"><a href="#net-core-wmem-default-68388608" class="headerlink" title="net.core.wmem_default = 68388608"></a>net.core.wmem_default = 68388608</h2><p>该参数指定了发送套接字缓冲区大小的缺省值(以字节为单位)</p>
<h2 id="net-core-rmem-default-68388608"><a href="#net-core-rmem-default-68388608" class="headerlink" title="net.core.rmem_default = 68388608"></a>net.core.rmem_default = 68388608</h2><p>该参数指定了接收套接字缓冲区大小的缺省值(以字节为单位)</p>
<h2 id="net-core-rmem-max-1116777216"><a href="#net-core-rmem-max-1116777216" class="headerlink" title="net.core.rmem_max = 1116777216"></a>net.core.rmem_max = 1116777216</h2><p>该参数指定了接收套接字缓冲区大小的最大值(以字节为单位)</p>
<h2 id="net-core-wmem-max-1116777216"><a href="#net-core-wmem-max-1116777216" class="headerlink" title="net.core.wmem_max = 1116777216"></a>net.core.wmem_max = 1116777216</h2><p>该参数指定了发送套接字缓冲区大小的最大值(以字节为单位)</p>
<h2 id="net-ipv4-tcp-sack-0"><a href="#net-ipv4-tcp-sack-0" class="headerlink" title="net.ipv4.tcp_sack = 0"></a>net.ipv4.tcp_sack = 0</h2><p>TCP SACK启用有选择的应答（1表示启用），通过有选择地应答乱序接收到的报文来提供网络利用率，让发送者只发送丢失的报文段，但是有研究表明如果启用该项会增加对CPU的占用，降低TCP连接的整体效率。除非在高延迟或者高丢包率的场景下需要启用该项，而对于高负载网络业务环境下建议关闭SACK，调整命令如下：<br><code>sysctl -w net.ipv4.tcp_sack=0</code><br><code>net.ipv4.tcp_sack = 0</code></p>
<h2 id="net-ipv4-tcp-max-tw-buckets-65535"><a href="#net-ipv4-tcp-max-tw-buckets-65535" class="headerlink" title="net.ipv4.tcp_max_tw_buckets=65535"></a>net.ipv4.tcp_max_tw_buckets=65535</h2><p>表示系统同时保持TIME_WAIT套接字的最大数量，如果超过这个数字，TIME_WAIT套接字将立刻被清除并打印警告信息。默认为180000，改为5000。</p>
<h2 id="net-ipv4-tcp-max-orphans-32768"><a href="#net-ipv4-tcp-max-orphans-32768" class="headerlink" title="net.ipv4.tcp_max_orphans=32768"></a>net.ipv4.tcp_max_orphans=32768</h2><p>系统所能处理不属于任何进程的TCP sockets最大数量。假如超过这个数量﹐那么不属于任何进程的连接会被立即reset，并同时显示警告信息。之所以要设定这个限制﹐纯粹为了抵御那些简单的DoS攻击﹐千万不要依赖这个或是人为的降低这个限制</p>
<h2 id="net-ipv4-tcp-synack-retries-2"><a href="#net-ipv4-tcp-synack-retries-2" class="headerlink" title="net.ipv4.tcp_synack_retries = 2"></a>net.ipv4.tcp_synack_retries = 2</h2><p>对于远端的连接请求 SYN，内核会发送 SYN＋ACK 数据报，以确认收到上一个 SYN 连接请求包。这是所谓的三次握手(threeway handshake)机制的第二个步骤。这里决定内核在放弃连接之前所送出的SYN+ACK 数目。不应该大于 255，默认值是 5，对应于 180 秒左右时间。(可以根据 tcp_syn_retries 来决定这个值)</p>
<h2 id="net-ipv4-tcp-syn-retries-2"><a href="#net-ipv4-tcp-syn-retries-2" class="headerlink" title="net.ipv4.tcp_syn_retries = 2"></a>net.ipv4.tcp_syn_retries = 2</h2><p>对于一个新建连接，内核要发送多少个 SYN 连接请求才决定放弃。不应该大于 255，默认值是 5，对应于 180 秒左右时间。(对于大负载而物理通信良好的网络而言,这个值偏高,可修改为 2.这个值仅仅是针对对外的连接,对进来的连接,是由 tcp_retries1 决定的)</p>
<h2 id="net-ipv4-ip-forward-1"><a href="#net-ipv4-ip-forward-1" class="headerlink" title="net.ipv4.ip_forward=1"></a>net.ipv4.ip_forward=1</h2><p>ip地址分公有地址和私有地址，public address是由INIC(internet network information center)负责，这些ip地址分配给注册并向INIC提出申请的组织机构。通过它访问internet.private address是属于非注册地址，专门为组织内部使用，private ip address是不可能直接用来跟WAN通信的，要么利用帧来通信（FRE帧中继，HDLC,PPP）,要么需要路由的NAT功能把私有地址转换为一个公有ip!<br>选择一台电脑（有两个网卡或者用单网卡然后用软件虚拟多一个网卡）充当网关，一个网卡(eth0)连接外网ISP，另一网卡(eth1)连接内网(即局域网)。局域网内的ip地址都是私用地址，只能在内部使用，在公网上是不可见的，所以局域网电脑要上网必须修改ip，这就是网关的工作。<br>工作原理：<br>内网主机向公网发送数据包时，由于目的主机跟源主机不在同一网段，所以数据包暂时发往内网默认网关处理，而本网段的主机对此数据包不做任何回应。由于源主机ip是私有的，禁止在公网使用，所以必须将数据包的源发送地址修改成公网上的可用ip，这就是网关收到数据包之后首先要做的工作–ip转换。然后网关再把数据包发往目的主机。目的主机收到数据包之后，只认为这是网关发送的请求，并不知道内网主机的存在，也没必要知道，目的主机处理完请求，把回应信息发还给网关。网关收到后，将目的主机发还的数据包的目的ip地址修改为发出请求的内网主机的ip地址，并将其发给内网主机。这就是网关的第二个工作–数据包的路由转发。内网的主机只要查看数据包的目的ip与发送请求的源主机ip地址相同，就会回应，这就完成了一次请求。<br>出于安全考虑，Linux系统默认是禁止数据包转发的。所谓转发即当主机拥有多于一块的网卡时，其中一块收到数据包，根据数据包的目的ip地址将包发往本机另一网卡，该网卡根据路由表继续发送数据包。这通常就是路由器所要实现的功能。<br>配置Linux系统的ip转发功能，首先保证硬件连通，然后打开系统的转发功能<br>less /proc/sys/net/ipv4/ip_forward，该文件内容为0，表示禁止数据包转发，1表示允许，将其修改为1。可使用命令echo “1” > /proc/sys/net/ipv4/ip_forward 修改文件内容，重启网络服务或主机后效果不再。若要其自动执行，可将命令echo “1” > /proc/sys/net/ipv4/ip_forward 写入脚本/etc/rc.d/rc.local 或者 在/etc/sysconfig/network脚本中添加 FORWARD_IPV4=”YES”<br><a href="http://www.cnblogs.com/jackhub/p/3575879.html" target="_blank" rel="external">http://www.cnblogs.com/jackhub/p/3575879.html</a></p>
<h2 id="net-ipv6-conf-all-disable-ipv6-1"><a href="#net-ipv6-conf-all-disable-ipv6-1" class="headerlink" title="net.ipv6.conf.all.disable_ipv6=1"></a>net.ipv6.conf.all.disable_ipv6=1</h2><h2 id="net-ipv6-conf-default-disable-ipv6-1"><a href="#net-ipv6-conf-default-disable-ipv6-1" class="headerlink" title="net.ipv6.conf.default.disable_ipv6=1"></a>net.ipv6.conf.default.disable_ipv6=1</h2><h2 id="net-ipv4-ip-forward-1-1"><a href="#net-ipv4-ip-forward-1-1" class="headerlink" title="net.ipv4.ip_forward=1"></a>net.ipv4.ip_forward=1</h2><h2 id="net-ipv4-ip-local-port-range-1024-65000"><a href="#net-ipv4-ip-local-port-range-1024-65000" class="headerlink" title="net.ipv4.ip_local_port_range = 1024 65000"></a>net.ipv4.ip_local_port_range = 1024 65000</h2><h1 id="nginx-部分配置优化"><a href="#nginx-部分配置优化" class="headerlink" title="nginx 部分配置优化"></a>nginx 部分配置优化</h1><ul>
<li>worker_processes：子进程数优化</li>
<li>worker_rlimit_nofile：进程打开的最大文件描述符优化</li>
<li>multi_accept：优化 worker 进程可以一次接收监听队列里所有请求</li>
<li>worker_connections：优化 worker 进程可以处理的最大请求数</li>
<li>use epoll：使用 epoll 事件驱动模型</li>
<li>sendfile：开启高效文件传输模式</li>
<li>aio：使用异步 io 优化</li>
<li>tcp_nopush：长连接时生效</li>
<li>tcp_nodelay：配合 sendfile 使用</li>
<li>reset_timedout_connection：关闭不响应的客户端连接</li>
<li>keepalive_timeout：优化 keepalive 长连接</li>
<li>keepalive_requests：优化每个长连接上可以服务的最大请求数量</li>
<li>client_body_buffer_size 系列参数的优化，优化客户端请求的缓存及body</li>
<li>gzip：压缩数据，提高传输速率</li>
<li>upstream 中的 keepalive：优化 nginx 与上游服务器间的空闲连接</li>
<li>proxy_connect_timeout 系列：优化 nginx 与上游服务器间的超时</li>
<li>proxy_buffers 系列：上游服务器返回数据给 nginx 时，nginx 缓存数据后一起发送，提高效率。</li>
</ul>
<h1 id="nginx-排除总结"><a href="#nginx-排除总结" class="headerlink" title="nginx 排除总结"></a>nginx 排除总结</h1><h2 id="io-setup-failed-11-resource-temporarily-unavailable"><a href="#io-setup-failed-11-resource-temporarily-unavailable" class="headerlink" title="io_setup() failed (11: resource temporarily unavailable)"></a>io_setup() failed (11: resource temporarily unavailable)</h2><p>北京环境 nginx 报错 io_setup() failed (11: resource temporarily unavailable)<br>由于 nginx 使用 linux 原生的异步 io，所以当 <code>/proc/sys/fs/aio-max-nr</code> （默认 65535） 过小时，出现以上错误。<code>/proc/sys/fs/aio-nr</code> 用于显示当前 <code>aio</code> 请求数。<br>解决办法：设置 <code>aio-max-nr</code> 为 1048576</p>
<h2 id="nginx-返回-500-错误"><a href="#nginx-返回-500-错误" class="headerlink" title="nginx 返回 500 错误"></a>nginx 返回 500 错误</h2><p>这是因为 post 请求体中的数据太多了，可以通过设置 <code>client_body_buffer_size</code>，<code>client_max_body_size</code> 等参数搞定。</p>
<blockquote>
<p>同时针对post请求还需要注意参数(client_body_in_single_buffer)的配置，如果不打开这个选项当请求串大于client_body_buffer_size大小时，需要手动去读取存储在磁盘的请求，这里需要注意的是存入磁盘的请求是一个完整的请求并不是大于client_body_buffer_size的部分</p>
</blockquote>
<p>解决办法如下：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line">client_header_buffer_size 4K;</div><div class="line">large_client_header_buffers 4 8k;</div><div class="line">client_body_buffer_size 60k;</div><div class="line">client_max_body_size 2m;</div><div class="line">client_body_in_single_buffer on;</div></pre></td></tr></table></figure></p>
<h2 id="nginx-css-等静态文件找不到"><a href="#nginx-css-等静态文件找不到" class="headerlink" title="nginx css 等静态文件找不到"></a>nginx css 等静态文件找不到</h2><p>解决办法：<br>nginx 中配置 <code>proxy_set_header Host $http_host;</code> ，<code>$host</code> 和 <code>$http_host</code> 的区别是后者带端口</p>
<p>HTTP header 中的 X_Forward_For 表示该条 http 请求是由谁发起的。如果反向代理服务器不重写该请求头的话，那么后端真实 web 服务器在处理时会认为所有的请求都来自反向代理服务器。如果后端 web 服务器有防攻击策略的话，那么反向代理服务器对应的 ip 地址就会被封掉。因此，在配置用作反向代理的 nginx 时，一般会增加以下两条配置修改 http 的请求头：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">proxy_set_header Host $http_host;</div><div class="line">proxy_set_header X-Forward-For $remote_addr;</div></pre></td></tr></table></figure></p>
<p>这里，<code>$http_host</code> 和 <code>$remote_addr</code>都是 nginx 的导出变量，可以在配置文件中直接使用。如果 Host 没有出现在 HTTP header 中，则 <code>$http_host</code> 的值为空，而 <code>$host</code> 和 <code>$http_host</code> 同样表示请求头中的 Host 字段，但若 Host 字段不存在，则以实际处理的虚拟主机 server 的 server_name 替代。因此一般而言，会用 <code>$host</code> 代替 <code>$http_host</code> 变量（如下），从而避免 http 请求中丢失 Host 头部的情况下 Host 不被重写的失误。</p>
<p>X-real-ip 和 X-Forwarded-For 的区别</p>
<h2 id="nginx-江苏环境返回真实-IP"><a href="#nginx-江苏环境返回真实-IP" class="headerlink" title="nginx 江苏环境返回真实 IP"></a>nginx 江苏环境返回真实 IP</h2><p>在一级负载均衡的 nginx location 中添加 <code>proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;</code></p>
<p>此时应用可以从 <code>X-Forwarded-For</code> Header 中获取真实 IP</p>
<h2 id="swagger-跨域问题"><a href="#swagger-跨域问题" class="headerlink" title="swagger 跨域问题"></a>swagger 跨域问题</h2><p>用 nignx 做反向代理，在 <code>locaion</code> 中添加 <code>add_header Access-Control-Allow-Origin *;</code> 即可。</p>
<blockquote>
<p>注意这里使用了 <code>add_header</code> 表示在 response 返回中添加 header</p>
</blockquote>
<p>参考文献：</p>
<p><a href="http://www.yunweipai.com/archives/9381.html" target="_blank" rel="external">http://www.yunweipai.com/archives/9381.html</a></p>
<p><a href="https://developer.mozilla.org/zh-CN/docs/Web/HTTP/Access_control_CORS" target="_blank" rel="external">https://developer.mozilla.org/zh-CN/docs/Web/HTTP/Access_control_CORS</a></p>
<h2 id="nginx-静态缓存问题"><a href="#nginx-静态缓存问题" class="headerlink" title="nginx 静态缓存问题"></a>nginx 静态缓存问题</h2><figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div></pre></td><td class="code"><pre><div class="line">http {</div><div class="line">proxy_cache_path /tmp/nginx levels=1:2 keys_zone=my_zone:10m inactive=60m;</div><div class="line">proxy_cache_valid 200 304 302 24h;</div><div class="line"># proxy_cache_key "$scheme$request_method$host$request_uri";</div><div class="line"></div><div class="line">server {</div><div class="line">server_name _ ;</div><div class="line"></div><div class="line">listen 30000;</div><div class="line"></div><div class="line">location /index {</div><div class="line">proxy_cache my_zone;</div><div class="line">add_header X-Proxy-Cache $upstream_cache_status;</div><div class="line"></div><div class="line">proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;</div><div class="line">proxy_pass http://10.142.21.251:36000;</div><div class="line">}</div><div class="line">}</div><div class="line">}</div></pre></td></tr></table></figure>
<p>先说下 proxy_cache_path 中的参数：</p>
<ul>
<li><p>/tmp/nginx 是缓存存放的路径</p>
</li>
<li><p>levels=1:2 是用来指定缓存文件夹的级数，该例中表示建立2级目录，例如资源路径进行hash处理，得到e0bd86606797639426a92306b1b98ad9，再hash值最后1位(9)拿出建一个目录，然后再把9前面的2位(ad)拿来建一个目录, 那么缓存文件的路径就是/tmp/nginx/9/d/e0bd86606797639426a92306b1b98ad9。</p>
</li>
<li><p>keys_zone=cache_zone:10m 其中 cache_zone 表示共享池的名称，随便起个名字，10m表示10兆，这个共享池的大小。在共享内存中设置一块存储区域来存放缓存的key和metadata（类似使用次数），这样nginx可以快速判断一个request是否命中或者未命中缓存，1m可以存储8000个key，10m可以存储80000个key；</p>
</li>
<li><p>inactive=1d 表示指定时间内缓存的数据如果没有被请求则会被删除。</p>
</li>
<li><p>max_size=100m 最大cache空间，如果不指定，会使用掉所有disk space，当达到配额后，会删除最少使用的cache文件；</p>
</li>
<li><p>proxy_cache cache_zone 用来指定用哪个 keys_zone 名字，因为 proxy_cache_path 可以使用多次，指定多个目录。</p>
</li>
<li><p>proxy_cache_key \$scheme\$host$request_uri; 用来指定生成hash的url地址的格式。他会根据这个key映射成一个hash值，然后存入到本地文件。</p>
</li>
<li><p>proxy_cache_valid any 10s; 表示对所有的状态码设置缓存的有效时间为 10s。</p>
</li>
</ul>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">proxy_cache_valid 200 302 10m;</div><div class="line">proxy_cache_valid 301 1h;</div><div class="line">proxy_cache_valid any 1m; #所有的状态都缓存1小时</div></pre></td></tr></table></figure>
<h2 id="close-wait-问题"><a href="#close-wait-问题" class="headerlink" title="close-wait 问题"></a>close-wait 问题</h2><p>江苏环境产生大量 close-wait ，由于 pid 数量超过内核限制，导致 nginx 工作进程无法创建线程。解决办法：调整内核 pid_max 参数</p>
]]></content>
<summary type="html">
<p>结合项目上生产使用经验，研发给出的参数调优建议。本篇仅作记录，及做一个分享，版权归我的好兄弟@董琪。<br>
</summary>
<category term="Kubernetes" scheme="https://caitianxiong.com/tags/Kubernetes/"/>
<category term="Ingress" scheme="https://caitianxiong.com/tags/Ingress/"/>
<category term="Nginx" scheme="https://caitianxiong.com/tags/Nginx/"/>
</entry>
<entry>
<title>部署K8s集群</title>
<link href="https://caitianxiong.com/2018/01/05/%E9%83%A8%E7%BD%B2K8s%E9%9B%86%E7%BE%A4/"/>
<id>https://caitianxiong.com/2018/01/05/部署K8s集群/</id>
<published>2018-01-04T16:00:00.000Z</published>
<updated>2020-02-02T01:42:26.822Z</updated>
<content type="html"><![CDATA[<h3 id="0-参考"><a href="#0-参考" class="headerlink" title="0. 参考"></a>0. 参考</h3><ul>
<li><a href="https://steemit.com/kubernetes/@cloudman6/k8s-cluster-kubernetes-5" target="_blank" rel="external">cloudman 博客</a></li>
<li><a href="https://kubernetes.io/docs/setup/independent/install-kubeadm/" target="_blank" rel="external">K8s官方文档</a><a id="more"></a>
</li>
</ul>
<h3 id="一、部署步骤"><a href="#一、部署步骤" class="headerlink" title="一、部署步骤"></a>一、部署步骤</h3><h4 id="1-所有节点装docker-io"><a href="#1-所有节点装docker-io" class="headerlink" title="1. 所有节点装docker.io"></a>1. 所有节点装docker.io</h4><table>
<thead>
<tr>
<th>hostname</th>
<th>内网IP</th>
<th>外网IP</th>
</tr>
</thead>
<tbody>
<tr>
<td>k8smaster</td>
<td>192.168.56.105</td>
<td>172.20.22.106</td>
</tr>
<tr>
<td>k8snode1</td>
<td>192.168.56.106</td>
<td>172.20.22.121</td>
</tr>
<tr>
<td>k8snode2</td>
<td>192.168.56.107</td>
<td>172.20.22.154</td>
</tr>
</tbody>
</table>
<h4 id="2-所有节点装kubectl-kubeadm-kubelet"><a href="#2-所有节点装kubectl-kubeadm-kubelet" class="headerlink" title="2. 所有节点装kubectl / kubeadm / kubelet"></a>2. 所有节点装kubectl / kubeadm / kubelet</h4><p>在所有节点上安装 kubelet、kubeadm 和 kubectl。<br>kubelet 运行在 Cluster 所有节点上，负责启动 Pod 和容器。<br>kubeadm 用于初始化 Cluster。<br>kubectl 是 Kubernetes 命令行工具。通过 kubectl 可以部署和管理应用，查看各种资源，创建、删除和更新各种组件。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line">apt-get update && apt-get install -y apt-transport-https</div><div class="line">curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -</div><div class="line">cat <<EOF >/etc/apt/sources.list.d/kubernetes.list</div><div class="line">deb http://apt.kubernetes.io/ kubernetes-xenial main</div><div class="line">EOF</div><div class="line">apt-get update</div><div class="line">apt-get install -y kubelet kubeadm kubectl\</div></pre></td></tr></table></figure></p>
<h4 id="3-手动下载Master组件docker-镜像"><a href="#3-手动下载Master组件docker-镜像" class="headerlink" title="3. 手动下载Master组件docker 镜像"></a>3. 手动下载Master组件docker 镜像</h4><p>gcr.io/google_containers/etcd-amd64:3.1.10<br>gcr.io/google_containers/kube-apiserver-amd64:v1.9.1<br>gcr.io/google_containers/kube-controller-manager-amd64:v1.9.1<br>gcr.io/google_containers/kube-scheduler-amd64:v1.9.1</p>
<h4 id="4-初始化集群Master节点"><a href="#4-初始化集群Master节点" class="headerlink" title="4. 初始化集群Master节点"></a>4. 初始化集群Master节点</h4><p><code>kubelet init --XXXX</code></p>
<h3 id="二、踩坑记录"><a href="#二、踩坑记录" class="headerlink" title="二、踩坑记录"></a>二、踩坑记录</h3><ul>
<li>virtualbox上虚机网络配置<br>加一个内部网卡，自己给3台虚机配置同一网段的IP、GW，重启network service报错不管，三台能通</li>
<li>配命令行翻墙代理<br>shadowsocks + polipo，配置文件记得改一致了</li>
<li>关闭Ubuntu swap分区<br><code>swapoff -a</code></li>
<li>初始化集群时，boot up control plane超时<br><img src="/images/K8s报错1.png" alt="K8s报错1"><br>提前拉取好dashboard的docker镜像，用 <code>docker load < XXX</code> 装好</li>
<li>初始化Master节点时，无法给Master节点添加标签<br><img src="/images/K8s报错2.png" alt="K8s报错2"><br>Ubuntu系统日志syslog中的报错：<br><img src="/images/K8s报错3.png" alt="K8s报错3"><br>改完hostname后（主机名不要带-），清理环境不彻底，导致还留有ubuntu-1的信息，要用<code>kubeadm reset</code>!!!</li>
<li>切换网络后无法上外网的问题，参考 <a href="https://blog.jysoftware.com/2013/04/ubuntu设置缺省路由/" target="_blank" rel="external">https://blog.jysoftware.com/2013/04/ubuntu设置缺省路由/</a><br><img src="/images/K8s报错4.png" alt="K8s报错4"></li>
<li><code>kubeadm init</code> 完成后，查看kubectl Server版本报错【服务已经起了，非root用户可以正常查看】<br><img src="/images/K8s报错5.png" alt="K8s报错5"></li>
<li>node节点加入集群后，状态NotReady<br>node节点上的proxy和flannel pod状态异常，已将flannel和proxy的docker镜像传到相应节点并load了。<br><img src="/images/K8s报错6.png" alt="K8s报错6"></li>
</ul>
]]></content>
<summary type="html">
<h3 id="0-参考"><a href="#0-参考" class="headerlink" title="0. 参考"></a>0. 参考</h3><ul>
<li><a href="https://steemit.com/kubernetes/@cloudman6/k8s-cluster-kubernetes-5">cloudman 博客</a></li>
<li><a href="https://kubernetes.io/docs/setup/independent/install-kubeadm/">K8s官方文档</a>
</summary>
<category term="Kubernetes" scheme="https://caitianxiong.com/tags/Kubernetes/"/>
<category term="K8s" scheme="https://caitianxiong.com/tags/K8s/"/>
</entry>
<entry>
<title>【翻译】Kubernetes生产模式以及反模式</title>
<link href="https://caitianxiong.com/2017/08/18/Kubernetes%E7%94%9F%E4%BA%A7%E6%A8%A1%E5%BC%8F%E4%BB%A5%E5%8F%8A%E5%8F%8D%E6%A8%A1%E5%BC%8F/"/>
<id>https://caitianxiong.com/2017/08/18/Kubernetes生产模式以及反模式/</id>
<published>2017-08-17T16:00:00.000Z</published>
<updated>2020-02-02T02:08:01.304Z</updated>
<content type="html"><![CDATA[<h1 id="Kubernetes生产模式-。。。及反模式"><a href="#Kubernetes生产模式-。。。及反模式" class="headerlink" title="Kubernetes生产模式 。。。及反模式"></a>Kubernetes生产模式 。。。及反模式</h1><p>本文是一篇英文博客的翻译，<a href="https://gravitational.com/blog/kubernetes-production-patterns/" target="_blank" rel="external">原文地址</a>。<br>以下开始正文：</p>
<p>我们将探索一些有用的技术来提高Kubernetes部署的弹性和高可用性，同时看看一些和Docker和Kubernetes协作的常见错误，以避免犯错。<br><a id="more"></a></p>
<h2 id="安装"><a href="#安装" class="headerlink" title="安装"></a>安装</h2><p>首先，跟着<a href="README.md#installation">安装引导</a></p>
<h3 id="反模式：混淆构建环境和运行时环境"><a href="#反模式：混淆构建环境和运行时环境" class="headerlink" title="反模式：混淆构建环境和运行时环境"></a>反模式：混淆构建环境和运行时环境</h3><p>来看下这个dockerfile<br><figure class="highlight dockerfile"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">FROM</span> ubuntu:<span class="number">14.04</span></div><div class="line"></div><div class="line"><span class="keyword">RUN</span> <span class="bash">apt-get update</span></div><div class="line"><span class="keyword">RUN</span> <span class="bash">apt-get install gcc</span></div><div class="line"><span class="keyword">RUN</span> <span class="bash">gcc hello.c -o /hello</span></div></pre></td></tr></table></figure></p>
<p>它将编译并运行一个简单的 helloword 程序：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line">$ <span class="built_in">cd</span> prod/build</div><div class="line">$ docker build -t prod .</div><div class="line">$ docker run prod</div><div class="line">Hello World</div></pre></td></tr></table></figure>
<p>生成的Dockerfile有好几个问题：</p>
<p><strong>大小</strong></p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">$ docker images | grep prod</div><div class="line">prod latest b2c197180350 14 minutes ago 293.7 MB</div></pre></td></tr></table></figure>
<p>用将近300M的镜像来装载几kb的C程序！我们把C编译器和好多不必要的工具——运行这个程序不需要的——都打包进来了。</p>
<p>这也导致了第二个问题：</p>
<p><strong>安全</strong></p>
<p>我们将整个构建工具链分散化/打散。另外，我们装载了镜像的源码：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div></pre></td><td class="code"><pre><div class="line">$ docker run --entrypoint=cat prod /build/hello.c</div><div class="line"><span class="comment">#include<stdio.h></span></div><div class="line"></div><div class="line">int <span class="function"><span class="title">main</span></span>()</div><div class="line">{</div><div class="line"> <span class="built_in">printf</span>(<span class="string">"Hello World\n"</span>);</div><div class="line"> <span class="built_in">return</span> 0;</div><div class="line">}</div></pre></td></tr></table></figure>
<p><strong>分开编译环境和运行时环境</strong></p>
<p>我们将用”buildbox”模式在构建环境里构建一个镜像，然后用一个小得多的运行时环境来运行程序。</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">$ <span class="built_in">cd</span> prod/build-fix</div><div class="line">$ docker build <span class="_">-f</span> build.dockerfile -t buildbox .</div></pre></td></tr></table></figure>
<p><strong>注意：</strong> 我们用了新的<code>-f</code>参数来指定使用的dockerfile。</p>
<p>现在我们有了个包含构建环境的<code>buildbox</code>镜像，可以用它来编译C程序了：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">$ docker run -v $(<span class="built_in">pwd</span>):/build buildbox gcc /build/hello.c -o /build/hello</div></pre></td></tr></table></figure>
<p>这次没有用<code>docker build</code>，挂载了源码然后直接运行编译器。</p>
<p><strong>注意：</strong>Docker很快将原生支持这个模式，通过集成<a href="https://github.com/docker/docker/pull/32063" target="_blank" rel="external">build stages</a>到构建进程的方式。<br><strong>更新：</strong> <a href="https://docs.docker.com/engine/userguide/eng-image/multistage-build/" target="_blank" rel="external">多阶段编译在社区版已可用</a>.</p>
<p>现在可以用一个简单得多（也小得多）的 dockerfile 来运行镜像：</p>
<figure class="highlight dockerfile"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">FROM</span> quay.io/gravitational/debian-tall:<span class="number">0.0</span>.<span class="number">1</span></div><div class="line"></div><div class="line"><span class="keyword">ADD</span> <span class="bash">hello /hello</span></div><div class="line"><span class="keyword">ENTRYPOINT</span> <span class="bash">[<span class="string">"/hello"</span>]</span></div></pre></td></tr></table></figure>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div></pre></td><td class="code"><pre><div class="line">$ docker build <span class="_">-f</span> run.dockerfile -t prod:v2 .</div><div class="line">$ docker run prod:v2</div><div class="line">Hello World</div><div class="line">$ docker images | grep prod</div><div class="line">prod v2 ef93cea87a7c 17 seconds ago 11.05 MB</div><div class="line">prod latest b2c197180350 45 minutes ago 293.7 MB</div></pre></td></tr></table></figure>
<p><strong>注意：</strong> 请注意你应该在运行时镜像里提供必须的“共享库”，或者“静态构建”二进制文件来让其包含所有必须的库文件。</p>
<h3 id="反模式：僵尸和孤儿"><a href="#反模式：僵尸和孤儿" class="headerlink" title="反模式：僵尸和孤儿"></a>反模式：僵尸和孤儿</h3><p><strong>注意：</strong> 这个示例只能运行在Linux上</p>
<p><strong>孤儿</strong></p>
<p>很容易留下孤儿进程在后台运行。以我们前面例子中构建的镜像为例：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">docker run busybox sleep 10000</div></pre></td></tr></table></figure>
<p>现在，再打开一个终端来定位那个进程</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">ps uax | grep sleep</div><div class="line">sasha 14171 0.0 0.0 139736 17744 pts/18 Sl+ 13:25 0:00 docker run busybox sleep 10000</div><div class="line">root 14221 0.1 0.0 1188 4 ? Ss 13:25 0:00 sleep 10000</div></pre></td></tr></table></figure>
<p>你可以看到有两个进程运行在容器中：<code>docker run</code>和<code>sleep 1000</code>。<br>给<code>docker run</code>进程发送kill信号（就像CI/CD会对长时间运行的进程做的那样）：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line"><span class="built_in">kill</span> 14171</div></pre></td></tr></table></figure>
<p><code>docker run</code> process has not exited, and <code>sleep</code> process is running!<br><code>docker run</code>进程没有退出，<code>sleep</code>进程还在运行！</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">ps uax | grep sleep</div><div class="line">root 14221 0.0 0.0 1188 4 ? Ss 13:25 0:00 sleep 10000</div></pre></td></tr></table></figure>
<p>Yelp工程师对此有个很好的<a href="https://github.com/Yelp/dumb-init" target="_blank" rel="external">解释</a>： </p>
<blockquote>
<p>Linux内核对PID为1的进程应用特殊的信号处理。<br>在普通的Linux操作系统上，发送一个信号给进程时，内核会首先检查进程给信号注册的常规处理程序是否可用，都不可用的话就回退到默认行为（比如，用SIGTREM杀掉进程）。</p>
<p>然而，如果收到信号的进程的PID是1，内核就会区别对待；如果进程没有为信号注册处理程序，内核就不会回退到默认行为，什么都不会发生。换句话说，如果进程没有显式处理这些信号，发送SIGTERM将不会有任何效果。</p>
</blockquote>
<p>为了解决这个（和其他）问题，需要一个简单的init系统，指定合适的信号处理程序。所幸<code>Yelp</code>的工程师构建了这个简单而轻量的init系统，<code>dumb-init</code>。</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">docker run quay.io/gravitational/debian-tall /usr/bin/dumb-init /bin/sh -c <span class="string">"sleep 10000"</span></div></pre></td></tr></table></figure>
<p>现在你可以简单地用SIGTERM来停止<code>docker run</code>进程，它会适当地处理终止。</p>
<h3 id="反模式：Pods的直接使用"><a href="#反模式：Pods的直接使用" class="headerlink" title="反模式：Pods的直接使用"></a>反模式：Pods的直接使用</h3><p><a href="https://kubernetes.io/docs/user-guide/pods/#what-is-a-pod" target="_blank" rel="external">Kubernetes Pod</a> 是非持久的构建块。</p>
<p>不要在生产中直接使用Pod，它们无法重新调度、保存数据或者保证持久性。</p>
<p>然而，你可以使用副本数为1的<code>Deloyment</code>，它可以保证pod被重新调度，并且在eviction和节点故障时存活。</p>
<h3 id="反模式：使用后台进程"><a href="#反模式：使用后台进程" class="headerlink" title="反模式：使用后台进程"></a>反模式：使用后台进程</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line">$ <span class="built_in">cd</span> prod/background</div><div class="line">$ docker build -t $(minikube ip):5000/background:0.0.1 .</div><div class="line">$ docker push $(minikube ip):5000/background:0.0.1</div><div class="line">$ kubectl create <span class="_">-f</span> crash.yaml</div><div class="line">$ kubectl get pods</div><div class="line">NAME READY STATUS RESTARTS AGE</div><div class="line">crash 1/1 Running 0 5s</div></pre></td></tr></table></figure>
<p>容器在运行了，让我们来检查下服务器是否在运行：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div></pre></td><td class="code"><pre><div class="line">$ kubectl <span class="built_in">exec</span> -ti crash /bin/bash</div><div class="line">root@crash:/<span class="comment"># </span></div><div class="line">root@crash:/<span class="comment"># </span></div><div class="line">root@crash:/<span class="comment"># ps uax</span></div><div class="line">USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND</div><div class="line">root 1 0.0 0.0 21748 1596 ? Ss 00:17 0:00 /bin/bash /start.sh</div><div class="line">root 6 0.0 0.0 5916 612 ? S 00:17 0:00 sleep 100000</div><div class="line">root 7 0.0 0.0 21924 2044 ? Ss 00:18 0:00 /bin/bash</div><div class="line">root 11 0.0 0.0 19180 1296 ? R+ 00:18 0:00 ps uax</div><div class="line">root@crash:/<span class="comment">#</span></div></pre></td></tr></table></figure>
<p><strong>使用探针</strong></p>
<p>我们犯了个错误导致HTTP服务器没有运行，但由于父进程在运行着没有报错的指示。</p>
<p>第一个明显的修复措施是使用一个合适的init系统来监控web服务的状态。但是，来把这当做使用survive eviction的一个机会吧：</p>
<figure class="highlight yaml"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div></pre></td><td class="code"><pre><div class="line"><span class="attr">apiVersion:</span> v1</div><div class="line"><span class="attr">kind:</span> Pod</div><div class="line"><span class="attr">metadata:</span></div><div class="line"><span class="attr"> name:</span> fix</div><div class="line"><span class="attr"> namespace:</span> default</div><div class="line"><span class="attr">spec:</span></div><div class="line"><span class="attr"> containers:</span></div><div class="line"><span class="attr"> - command:</span> [<span class="string">'/start.sh'</span>]</div><div class="line"><span class="attr"> image:</span> localhost:<span class="number">5000</span>/background:<span class="number">0.0</span><span class="number">.1</span></div><div class="line"><span class="attr"> name:</span> server</div><div class="line"><span class="attr"> imagePullPolicy:</span> Always</div><div class="line"><span class="attr"> livenessProbe:</span></div><div class="line"><span class="attr"> httpGet:</span></div><div class="line"><span class="attr"> path:</span> /</div><div class="line"><span class="attr"> port:</span> <span class="number">5000</span></div><div class="line"><span class="attr"> timeoutSeconds:</span> <span class="number">1</span></div></pre></td></tr></table></figure>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">$ kubectl create <span class="_">-f</span> fix.yaml</div></pre></td></tr></table></figure>
<p>survive eviction将会失败，容器将会重启。</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line">$ kubectl get pods</div><div class="line">NAME READY STATUS RESTARTS AGE</div><div class="line">crash 1/1 Running 0 11m</div><div class="line">fix 1/1 Running 1 1m</div></pre></td></tr></table></figure>
<h3 id="生产模式：日志"><a href="#生产模式：日志" class="headerlink" title="生产模式：日志"></a>生产模式：日志</h3><p>配置日志到stdout:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">$ kubectl create <span class="_">-f</span> logs/logs.yaml</div><div class="line">$ kubectl logs logs</div><div class="line">hello, world!</div></pre></td></tr></table></figure></p>
<p>Kubernetes 和 Docker 有个插件系统，来确保发送到stdout和stderr的日志可以被收集、转发或rotated。</p>
<p><strong>注意：</strong> 这是<a href="https://12factor.net/logs" target="_blank" rel="external">十二要素应用</a>之一，Kubernetes原生支持。</p>
<h3 id="生产模式："><a href="#生产模式：" class="headerlink" title="生产模式："></a>生产模式：</h3><p> 每次你写东西到容器的文件系统，它会激活<a href="https://docs.docker.com/engine/userguide/storagedriver/imagesandcontainers/#container-and-layers" target="_blank" rel="external">写时复制策略</a>。</p>
<p>这时存储驱动（devicemapper，overlayfs或其他）会创建一个存储层。在使用写时复制时，存储层会给存储驱动施加很多负载，尤其是使用Devicemapper或者BTRFS时。</p>
<p>确保你的容器只将数据写入到volumes。临时文件可以使用<code>tmpfs</code>（因为tmpfs将所有数据存储在内存中）。</p>
<figure class="highlight yaml"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div></pre></td><td class="code"><pre><div class="line"><span class="attr">apiVersion:</span> v1</div><div class="line"><span class="attr">kind:</span> Pod</div><div class="line"><span class="attr">metadata:</span></div><div class="line"><span class="attr"> name:</span> test-pd</div><div class="line"><span class="attr">spec:</span></div><div class="line"><span class="attr"> containers:</span></div><div class="line"><span class="attr"> - image:</span> busybox</div><div class="line"><span class="attr"> name:</span> test-container</div><div class="line"><span class="attr"> volumeMounts:</span></div><div class="line"><span class="attr"> - mountPath:</span> /tmp</div><div class="line"><span class="attr"> name:</span> tempdir</div><div class="line"><span class="attr"> volumes:</span></div><div class="line"><span class="attr"> - name:</span> tempdir</div><div class="line"><span class="attr"> emptyDir:</span> {}</div></pre></td></tr></table></figure>
<h3 id="反模式：使用latest标签"><a href="#反模式：使用latest标签" class="headerlink" title="反模式：使用latest标签"></a>反模式：使用<code>latest</code>标签</h3><p>不要在生产中使用<code>latest</code>标签，因为它含义模糊，没有说明应用的真正版本是多少。</p>
<p>处于开发目的使用<code>latest</code>是可以的，只要确保你设置了<code>imagePullPolicy</code>为<code>lways</code>，来保证创建pod时Kubernetes总是拉取最新版本：</p>
<figure class="highlight yaml"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div></pre></td><td class="code"><pre><div class="line"><span class="attr">apiVersion:</span> v1</div><div class="line"><span class="attr">kind:</span> Pod</div><div class="line"><span class="attr">metadata:</span></div><div class="line"><span class="attr"> name:</span> always</div><div class="line"><span class="attr"> namespace:</span> default</div><div class="line"><span class="attr">spec:</span></div><div class="line"><span class="attr"> containers:</span></div><div class="line"><span class="attr"> - command:</span> [<span class="string">'/bin/sh'</span>, <span class="string">'-c'</span>, <span class="string">"echo hello, world!"</span>]</div><div class="line"><span class="attr"> image:</span> busybox:latest</div><div class="line"><span class="attr"> name:</span> server</div><div class="line"><span class="attr"> imagePullPolicy:</span> Always</div></pre></td></tr></table></figure>
<h3 id="生产模式：Pod就绪"><a href="#生产模式：Pod就绪" class="headerlink" title="生产模式：Pod就绪"></a>生产模式：Pod就绪</h3><p>假设你的容器需要一些时间来启动，我们来写个脚本模拟：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div></pre></td><td class="code"><pre><div class="line"><span class="meta">#!/bin/bash</span></div><div class="line"></div><div class="line"><span class="built_in">echo</span> <span class="string">"Starting up"</span></div><div class="line">sleep 30</div><div class="line"><span class="built_in">echo</span> <span class="string">"Started up successfully"</span></div><div class="line">python -m http.serve 5000</div></pre></td></tr></table></figure>
<p>推送镜像并使用服务和deployment：</p>
<figure class="highlight yaml"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line">$ cd prod/delay</div><div class="line">$ docker build -t $(minikube ip):<span class="number">5000</span>/delay:<span class="number">0.0</span><span class="number">.1</span> .</div><div class="line">$ docker push $(minikube ip):<span class="number">5000</span>/delay:<span class="number">0.0</span><span class="number">.1</span></div><div class="line">$ kubectl create -f service.yaml</div><div class="line">$ kubectl create -f deployment.yaml</div></pre></td></tr></table></figure>
<p>进入容器内部使用curl命令，确保service和deployment都生效了：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line">kubectl run -i -t --rm cli --image=tutum/curl --restart=Never</div><div class="line">curl http://delay:5000</div><div class="line"><!DOCTYPE html></div><div class="line">...</div></pre></td></tr></table></figure>
<p>在你尝试获取服务的前30秒，你会发现一个<code>connection refused error</code>。</p>
<p>更新deployment来模拟部署：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">$ docker build -t $(minikube ip):5000/delay:0.0.2 .</div><div class="line">$ docker push $(minikube ip):5000/delay:0.0.2</div><div class="line">$ kubectl replace <span class="_">-f</span> deployment-update.yaml</div></pre></td></tr></table></figure>
<p>在另一个窗口，看看是否还有服务不可用时间：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">curl http://delay:5000</div><div class="line">curl: (7) Failed to connect to delay port 5000: Connection refused</div></pre></td></tr></table></figure>
<p>尽管在滚动更新的策略中设置了<code>maxUnavailable: 0</code>，还是出现了生产中断。这是因为Kubernetes不知道服务存在启动延迟和就绪时间。</p>
<p>来用readiness probe修复之：</p>
<figure class="highlight yaml"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div></pre></td><td class="code"><pre><div class="line"><span class="attr">readinessProbe:</span></div><div class="line"><span class="attr"> httpGet:</span></div><div class="line"><span class="attr"> path:</span> /</div><div class="line"><span class="attr"> port:</span> <span class="number">5000</span></div><div class="line"><span class="attr"> timeoutSeconds:</span> <span class="number">1</span></div><div class="line"><span class="attr"> periodSeconds:</span> <span class="number">5</span></div></pre></td></tr></table></figure>
<p>Readiness probe表征pod容器的就绪状态，K8s在部署deployment时会将此纳入考虑：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">$ kubectl replace <span class="_">-f</span> deployment-fix.yaml</div></pre></td></tr></table></figure>
<p>这下就不会有故障时间了。</p>
<h3 id="反模式：迅速解绑失效服务"><a href="#反模式：迅速解绑失效服务" class="headerlink" title="反模式：迅速解绑失效服务"></a>反模式：迅速解绑失效服务</h3><p>K8s提供新的工具——<a href="https://kubernetes.io/docs/concepts/jobs/run-to-completion-finite-workloads/" target="_blank" rel="external">jobs</a>——来调度容器执行一次性任务。</p>
<p>然而，有个问题：</p>
<figure class="highlight yaml"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div></pre></td><td class="code"><pre><div class="line"><span class="attr">apiVersion:</span> batch/v1</div><div class="line"><span class="attr">kind:</span> Job</div><div class="line"><span class="attr">metadata:</span></div><div class="line"><span class="attr"> name:</span> bad</div><div class="line"><span class="attr">spec:</span></div><div class="line"><span class="attr"> template:</span></div><div class="line"><span class="attr"> metadata:</span></div><div class="line"><span class="attr"> name:</span> bad</div><div class="line"><span class="attr"> spec:</span></div><div class="line"><span class="attr"> restartPolicy:</span> Never</div><div class="line"><span class="attr"> containers:</span></div><div class="line"><span class="attr"> - name:</span> box</div><div class="line"><span class="attr"> image:</span> busybox</div><div class="line"><span class="attr"> command:</span> [<span class="string">"/bin/sh"</span>, <span class="string">"-c"</span>, <span class="string">"exit 1"</span>]</div></pre></td></tr></table></figure>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">$ <span class="built_in">cd</span> prod/<span class="built_in">jobs</span></div><div class="line">$ kubectl create <span class="_">-f</span> job.yaml</div></pre></td></tr></table></figure>
<p>你将看到不停地重试任务创建数百个容器：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div></pre></td><td class="code"><pre><div class="line">$ kubectl describe <span class="built_in">jobs</span> </div><div class="line">Name: bad</div><div class="line">Namespace: default</div><div class="line">Image(s): busybox</div><div class="line">Selector: controller-uid=18a6678e-11d1-11e7-8169-525400c83acf</div><div class="line">Parallelism: 1</div><div class="line">Completions: 1</div><div class="line">Start Time: Sat, 25 Mar 2017 20:05:41 -0700</div><div class="line">Labels: controller-uid=18a6678e-11d1-11e7-8169-525400c83acf</div><div class="line"> job-name=bad</div><div class="line">Pods Statuses: 1 Running / 0 Succeeded / 24 Failed</div><div class="line">No volumes.</div><div class="line">Events:</div><div class="line"> FirstSeen LastSeen Count From SubObjectPath Type Reason Message</div><div class="line"> --------- -------- ----- ---- ------------- -------- ------ -------</div><div class="line"> 1m 1m 1 {job-controller } Normal SuccessfulCreate Created pod: bad-fws8g</div><div class="line"> 1m 1m 1 {job-controller } Normal SuccessfulCreate Created pod: bad-321pk</div><div class="line"> 1m 1m 1 {job-controller } Normal SuccessfulCreate Created pod: bad-2pxq1</div><div class="line"> 1m 1m 1 {job-controller } Normal SuccessfulCreate Created pod: bad-kl2tj</div><div class="line"> 1m 1m 1 {job-controller } Normal SuccessfulCreate Created pod: bad-wfw8q</div><div class="line"> 1m 1m 1 {job-controller } Normal SuccessfulCreate Created pod: bad-lz0hq</div><div class="line"> 1m 1m 1 {job-controller } Normal SuccessfulCreate Created pod: bad-0dck0</div><div class="line"> 1m 1m 1 {job-controller } Normal SuccessfulCreate Created pod: bad-0lm8k</div><div class="line"> 1m 1m 1 {job-controller } Normal SuccessfulCreate Created pod: bad-q6ctf</div><div class="line"> 1m 1s 16 {job-controller } Normal SuccessfulCreate (events with common reason combined)</div></pre></td></tr></table></figure>
<p>这可能不是你期待的结果。随着时间流逝，节点和docker上的负载将会非常巨大，尤其是如果job快速失败的话。</p>
<p>先来清理一直失败的任务：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">$ kubectl delete <span class="built_in">jobs</span>/bad</div></pre></td></tr></table></figure>
<p>再来用<code>activeDeadlineSeconds</code>限制重试次数：</p>
<figure class="highlight yaml"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div></pre></td><td class="code"><pre><div class="line"><span class="attr">apiVersion:</span> batch/v1</div><div class="line"><span class="attr">kind:</span> Job</div><div class="line"><span class="attr">metadata:</span></div><div class="line"><span class="attr"> name:</span> bound</div><div class="line"><span class="attr">spec:</span></div><div class="line"><span class="attr"> activeDeadlineSeconds:</span> <span class="number">10</span></div><div class="line"><span class="attr"> template:</span></div><div class="line"><span class="attr"> metadata:</span></div><div class="line"><span class="attr"> name:</span> bound</div><div class="line"><span class="attr"> spec:</span></div><div class="line"><span class="attr"> restartPolicy:</span> Never</div><div class="line"><span class="attr"> containers:</span></div><div class="line"><span class="attr"> - name:</span> box</div><div class="line"><span class="attr"> image:</span> busybox</div><div class="line"><span class="attr"> command:</span> [<span class="string">"/bin/sh"</span>, <span class="string">"-c"</span>, <span class="string">"exit 1"</span>]</div></pre></td></tr></table></figure>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">$ kubectl create <span class="_">-f</span> bound.yaml</div></pre></td></tr></table></figure>
<p>现在你会发现，10秒后任务就失败了：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">11s 11s 1 {job-controller } Normal DeadlineExceeded Job was active longer than specified deadline</div></pre></td></tr></table></figure>
<p><strong>注意：</strong> 有时一直重试是有意义的。这种情况下就需要设置一个适当的pod重启策略，来保护你的集群免于意外的DDOS攻击。</p>
<h3 id="生产模式：断路开关"><a href="#生产模式：断路开关" class="headerlink" title="生产模式：断路开关"></a>生产模式：断路开关</h3><p>这个例子中，假设我们的web应用是一个给email使用的服务器。前端需要发送两个请求给后端，来渲染页面：</p>
<ul>
<li>与天气服务器通信来获取当前的天气</li>
<li>从数据库获取当前的邮件</li>
</ul>
<p>如果天气服务挂了，用户仍然可以收到邮件，所以天气服务是附加的，而当前的邮件服务是关键的。</p>
<p>这是前端，用python写的天气和邮件服务：</p>
<p><strong>天气</strong></p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">from</span> flask <span class="keyword">import</span> Flask</div><div class="line">app = Flask(__name__)</div><div class="line"></div><div class="line"><span class="meta">@app.route("/")</span></div><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">hello</span><span class="params">()</span>:</span></div><div class="line"> <span class="keyword">return</span> <span class="string">'''Pleasanton, CA</span></div><div class="line">Saturday 8:00 PM</div><div class="line">Partly Cloudy</div><div class="line">12 C</div><div class="line">Precipitation: 9%</div><div class="line">Humidity: 74%</div><div class="line">Wind: 14 km/h</div><div class="line">'''</div><div class="line"></div><div class="line"><span class="keyword">if</span> __name__ == <span class="string">"__main__"</span>:</div><div class="line"> app.run(host=<span class="string">'0.0.0.0'</span>)</div></pre></td></tr></table></figure>
<p><strong>邮件</strong></p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">from</span> flask <span class="keyword">import</span> Flask,jsonify</div><div class="line">app = Flask(__name__)</div><div class="line"></div><div class="line"><span class="meta">@app.route("/")</span></div><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">hello</span><span class="params">()</span>:</span></div><div class="line"> <span class="keyword">return</span> jsonify([</div><div class="line"> {<span class="string">"from"</span>: <span class="string">"<bob@example.com>"</span>, <span class="string">"subject"</span>: <span class="string">"lunch at noon tomorrow"</span>},</div><div class="line"> {<span class="string">"from"</span>: <span class="string">"<alice@example.com>"</span>, <span class="string">"subject"</span>: <span class="string">"compiler docs"</span>}])</div><div class="line"></div><div class="line"><span class="keyword">if</span> __name__ == <span class="string">"__main__"</span>:</div><div class="line"> app.run(host=<span class="string">'0.0.0.0'</span>)</div></pre></td></tr></table></figure>
<p><strong>前端</strong></p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">from</span> flask <span class="keyword">import</span> Flask</div><div class="line"><span class="keyword">import</span> requests</div><div class="line"><span class="keyword">from</span> datetime <span class="keyword">import</span> datetime</div><div class="line">app = Flask(__name__)</div><div class="line"></div><div class="line"><span class="meta">@app.route("/")</span></div><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">hello</span><span class="params">()</span>:</span></div><div class="line"> weather = <span class="string">"weather unavailable"</span></div><div class="line"> <span class="keyword">try</span>:</div><div class="line"> <span class="keyword">print</span> <span class="string">"requesting weather..."</span></div><div class="line"> start = datetime.now()</div><div class="line"> r = requests.get(<span class="string">'http://weather'</span>)</div><div class="line"> <span class="keyword">print</span> <span class="string">"got weather in %s ..."</span> % (datetime.now() - start)</div><div class="line"> <span class="keyword">if</span> r.status_code == requests.codes.ok:</div><div class="line"> weather = r.text</div><div class="line"> <span class="keyword">except</span>:</div><div class="line"> <span class="keyword">print</span> <span class="string">"weather unavailable"</span></div><div class="line"></div><div class="line"> <span class="keyword">print</span> <span class="string">"requesting mail..."</span></div><div class="line"> r = requests.get(<span class="string">'http://mail'</span>)</div><div class="line"> mail = r.json()</div><div class="line"> <span class="keyword">print</span> <span class="string">"got mail in %s ..."</span> % (datetime.now() - start)</div><div class="line"></div><div class="line"> out = []</div><div class="line"> <span class="keyword">for</span> letter <span class="keyword">in</span> mail:</div><div class="line"> out.append(<span class="string">"<li>From: %s Subject: %s</li>"</span> % (letter[<span class="string">'from'</span>], letter[<span class="string">'subject'</span>]))</div><div class="line"> </div><div class="line"></div><div class="line"> <span class="keyword">return</span> <span class="string">'''<html></span></div><div class="line"><body></div><div class="line"> <h3>Weather</h3></div><div class="line"> <p>%s</p></div><div class="line"> <h3>Email</h3></div><div class="line"> <p></div><div class="line"> <ul></div><div class="line"> %s</div><div class="line"> </ul></div><div class="line"> </p></div><div class="line"></body></div><div class="line">''' % (weather, <span class="string">'<br/>'</span>.join(out))</div><div class="line"></div><div class="line"><span class="keyword">if</span> __name__ == <span class="string">"__main__"</span>:</div><div class="line"> app.run(host=<span class="string">'0.0.0.0'</span>)</div></pre></td></tr></table></figure>
<p>来开始部署和服务：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div></pre></td><td class="code"><pre><div class="line">$ <span class="built_in">cd</span> prod/cbreaker</div><div class="line">$ docker build -t $(minikube ip):5000/mail:0.0.1 .</div><div class="line">$ docker push $(minikube ip):5000/mail:0.0.1</div><div class="line">$ kubectl apply <span class="_">-f</span> service.yaml</div><div class="line">deployment <span class="string">"frontend"</span> configured</div><div class="line">deployment <span class="string">"weather"</span> configured</div><div class="line">deployment <span class="string">"mail"</span> configured</div><div class="line">service <span class="string">"frontend"</span> configured</div><div class="line">service <span class="string">"mail"</span> configured</div><div class="line">service <span class="string">"weather"</span> configured</div></pre></td></tr></table></figure>
<p>检查是否一切正常：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div></pre></td><td class="code"><pre><div class="line">$ kubectl run -i -t --rm cli --image=tutum/curl --restart=Never</div><div class="line">$ curl http://frontend</div><div class="line"><html></div><div class="line"><body></div><div class="line"> <h3>Weather</h3></div><div class="line"> <p>Pleasanton, CA</div><div class="line">Saturday 8:00 PM</div><div class="line">Partly Cloudy</div><div class="line">12 C</div><div class="line">Precipitation: 9%</div><div class="line">Humidity: 74%</div><div class="line">Wind: 14 km/h</div><div class="line"></p></div><div class="line"> <h3>Email</h3></div><div class="line"> <p></div><div class="line"> <ul></div><div class="line"> <li>From: <bob@example.com> Subject: lunch at noon tomorrow</li><br/><li>From: <alice@example.com> Subject: compiler docs</li></div><div class="line"> </ul></div><div class="line"> </p></div><div class="line"></body></div></pre></td></tr></table></figure>
<p>来看看崩溃的天气服务：</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">from</span> flask <span class="keyword">import</span> Flask</div><div class="line">app = Flask(__name__)</div><div class="line"></div><div class="line"><span class="meta">@app.route("/")</span></div><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">hello</span><span class="params">()</span>:</span></div><div class="line"> <span class="keyword">raise</span> Exception(<span class="string">"I am out of service"</span>)</div><div class="line"></div><div class="line"><span class="keyword">if</span> __name__ == <span class="string">"__main__"</span>:</div><div class="line"> app.run(host=<span class="string">'0.0.0.0'</span>)</div></pre></td></tr></table></figure>
<p>构建并重新部署：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line">$ docker build -t $(minikube ip):5000/weather-crash:0.0.1 <span class="_">-f</span> weather-crash.dockerfile .</div><div class="line">$ docker push $(minikube ip):5000/weather-crash:0.0.1</div><div class="line">$ kubectl apply <span class="_">-f</span> weather-crash.yaml </div><div class="line">deployment <span class="string">"weather"</span> configured</div></pre></td></tr></table></figure>
<p>确保它崩溃了：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div></pre></td><td class="code"><pre><div class="line">$ kubectl run -i -t --rm cli --image=tutum/curl --restart=Never</div><div class="line">$ curl http://weather</div><div class="line"><!DOCTYPE HTML PUBLIC <span class="string">"-//W3C//DTD HTML 3.2 Final//EN"</span>></div><div class="line"><title>500 Internal Server Error</title></div><div class="line"><h1>Internal Server Error</h1></div><div class="line"><p>The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error <span class="keyword">in</span> the application.</p></div></pre></td></tr></table></figure>
<p>不过前端应该是正常的：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div></pre></td><td class="code"><pre><div class="line">$ kubectl run -i -t --rm cli --image=tutum/curl --restart=Never</div><div class="line">curl http://frontend</div><div class="line"><html></div><div class="line"><body></div><div class="line"> <h3>Weather</h3></div><div class="line"> <p>weather unavailable</p></div><div class="line"> <h3>Email</h3></div><div class="line"> <p></div><div class="line"> <ul></div><div class="line"> <li>From: <bob@example.com> Subject: lunch at noon tomorrow</li><br/><li>From: <alice@example.com> Subject: compiler docs</li></div><div class="line"> </ul></div><div class="line"> </p></div><div class="line"></body></div><div class="line">root@cli:/<span class="comment"># curl http://frontend </span></div><div class="line"><html></div><div class="line"><body></div><div class="line"> <h3>Weather</h3></div><div class="line"> <p>weather unavailable</p></div><div class="line"> <h3>Email</h3></div><div class="line"> <p></div><div class="line"> <ul></div><div class="line"> <li>From: <bob@example.com> Subject: lunch at noon tomorrow</li><br/><li>From: <alice@example.com> Subject: compiler docs</li></div><div class="line"> </ul></div><div class="line"> </p></div><div class="line"></body></div></pre></td></tr></table></figure>
<p>一切都在意料之中。尽管还有个问题，我们刚刚看到服务很快就挂了，来看看如果天气服务很慢的话会怎么样。这在生产中更为常见，例如因为网络或者数据库过载。</p>
<p>我们引入一个人为的延迟来模拟这个故障：</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">from</span> flask <span class="keyword">import</span> Flask</div><div class="line"><span class="keyword">import</span> time</div><div class="line"></div><div class="line">app = Flask(__name__)</div><div class="line"></div><div class="line"><span class="meta">@app.route("/")</span></div><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">hello</span><span class="params">()</span>:</span></div><div class="line"> time.sleep(<span class="number">30</span>)</div><div class="line"> <span class="keyword">raise</span> Exception(<span class="string">"System overloaded"</span>)</div><div class="line"></div><div class="line"><span class="keyword">if</span> __name__ == <span class="string">"__main__"</span>:</div><div class="line"> app.run(host=<span class="string">'0.0.0.0'</span>)</div></pre></td></tr></table></figure>
<p>构建并重新部署：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line">$ docker build -t $(minikube ip):5000/weather-crash-slow:0.0.1 <span class="_">-f</span> weather-crash-slow.dockerfile .</div><div class="line">$ docker push $(minikube ip):5000/weather-crash-slow:0.0.1</div><div class="line">$ kubectl apply <span class="_">-f</span> weather-crash-slow.yaml </div><div class="line">deployment <span class="string">"weather"</span> configured</div></pre></td></tr></table></figure>
<p>正如所料，天气服务超时了：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line">curl http://weather </div><div class="line"><!DOCTYPE HTML PUBLIC <span class="string">"-//W3C//DTD HTML 3.2 Final//EN"</span>></div><div class="line"><title>500 Internal Server Error</title></div><div class="line"><h1>Internal Server Error</h1></div><div class="line"><p>The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error <span class="keyword">in</span> the application.</p></div></pre></td></tr></table></figure>
<p>问题是，每次到前端的请求也要花10秒</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">curl http://frontend</div></pre></td></tr></table></figure>
<p>这是个更常见的故障——服务不可用让用户沮丧。我们将用<a href="http://vulcand.github.io/proxy.html#circuit-breakers" target="_blank" rel="external">circuit breaker</a>引入一个特别的代理来修复这个问题。</p>
<p><img src="/images/CircuitStandby.png" alt="备用"> </p>
<p>Circuit breaker是一个特殊的中间件，用于在服务降级时提供故障切换操作。用来防止连环故障（一个故障会引起另一个故障）很有效。Circuit breaker监控请求数据并检查特殊错误状况的状态。</p>
<p><img src="/images/CircuitTripped.png" alt="触发后"></p>
<p>这是用Python写的简单的circuit breaker：</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">from</span> flask <span class="keyword">import</span> Flask</div><div class="line"><span class="keyword">import</span> requests</div><div class="line"><span class="keyword">from</span> datetime <span class="keyword">import</span> datetime, timedelta</div><div class="line"><span class="keyword">from</span> threading <span class="keyword">import</span> Lock</div><div class="line"><span class="keyword">import</span> logging, sys</div><div class="line"></div><div class="line"></div><div class="line">app = Flask(__name__)</div><div class="line"></div><div class="line">circuit_tripped_until = datetime.now()</div><div class="line">mutex = Lock()</div><div class="line"></div><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">trip</span><span class="params">()</span>:</span></div><div class="line"> <span class="keyword">global</span> circuit_tripped_until</div><div class="line"> mutex.acquire()</div><div class="line"> <span class="keyword">try</span>:</div><div class="line"> circuit_tripped_until = datetime.now() + timedelta(<span class="number">0</span>,<span class="number">30</span>)</div><div class="line"> app.logger.info(<span class="string">"circuit tripped until %s"</span> %(circuit_tripped_until))</div><div class="line"> <span class="keyword">finally</span>:</div><div class="line"> mutex.release()</div><div class="line"></div><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">is_tripped</span><span class="params">()</span>:</span></div><div class="line"> <span class="keyword">global</span> circuit_tripped_until </div><div class="line"> mutex.acquire()</div><div class="line"> <span class="keyword">try</span>:</div><div class="line"> <span class="keyword">return</span> datetime.now() < circuit_tripped_until</div><div class="line"> <span class="keyword">finally</span>:</div><div class="line"> mutex.release()</div><div class="line"> </div><div class="line"></div><div class="line"><span class="meta">@app.route("/")</span></div><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">hello</span><span class="params">()</span>:</span></div><div class="line"> weather = <span class="string">"weather unavailable"</span></div><div class="line"> <span class="keyword">try</span>:</div><div class="line"> <span class="keyword">if</span> is_tripped():</div><div class="line"> <span class="keyword">return</span> <span class="string">"circuit breaker: service unavailable (tripped)"</span></div><div class="line"></div><div class="line"> r = requests.get(<span class="string">'http://localhost:5000'</span>, timeout=<span class="number">1</span>)</div><div class="line"> app.logger.info(<span class="string">"requesting weather..."</span>)</div><div class="line"> start = datetime.now()</div><div class="line"> app.logger.info(<span class="string">"got weather in %s ..."</span> % (datetime.now() - start))</div><div class="line"> <span class="keyword">if</span> r.status_code == requests.codes.ok:</div><div class="line"> <span class="keyword">return</span> r.text</div><div class="line"> <span class="keyword">else</span>:</div><div class="line"> trip()</div><div class="line"> <span class="keyword">return</span> <span class="string">"circuit breaker: service unavailable (tripping 1)"</span></div><div class="line"> <span class="keyword">except</span>:</div><div class="line"> app.logger.info(<span class="string">"exception: %s"</span>, sys.exc_info()[<span class="number">0</span>])</div><div class="line"> trip()</div><div class="line"> <span class="keyword">return</span> <span class="string">"circuit breaker: service unavailable (tripping 2)"</span></div><div class="line"></div><div class="line"><span class="keyword">if</span> __name__ == <span class="string">"__main__"</span>:</div><div class="line"> app.logger.addHandler(logging.StreamHandler(sys.stdout))</div><div class="line"> app.logger.setLevel(logging.DEBUG)</div><div class="line"> app.run(host=<span class="string">'0.0.0.0'</span>, port=<span class="number">6000</span>)</div></pre></td></tr></table></figure>
<p>构建并重新部署circuit breaker：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div></pre></td><td class="code"><pre><div class="line">$ docker build -t $(minikube ip):5000/cbreaker:0.0.1 <span class="_">-f</span> cbreaker.dockerfile .</div><div class="line">$ docker push $(minikube ip):5000/cbreaker:0.0.1</div><div class="line">$ kubectl apply <span class="_">-f</span> weather-cbreaker.yaml </div><div class="line">deployment <span class="string">"weather"</span> configured</div><div class="line">$ kubectl apply <span class="_">-f</span> weather-service.yaml</div><div class="line">service <span class="string">"weather"</span> configured</div></pre></td></tr></table></figure>
<p>Circuit breaker会检查服务故障，附加的天气服务不会再引起邮件服务的故障了：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div></pre></td><td class="code"><pre><div class="line">curl http://frontend</div><div class="line"><html></div><div class="line"><body></div><div class="line"> <h3>Weather</h3></div><div class="line"> <p>circuit breaker: service unavailable (tripped)</p></div><div class="line"> <h3>Email</h3></div><div class="line"> <p></div><div class="line"> <ul></div><div class="line"> <li>From: <bob@example.com> Subject: lunch at noon tomorrow</li><br/><li>From: <alice@example.com> Subject: compiler docs</li></div><div class="line"> </ul></div><div class="line"> </p></div><div class="line"></body></div></pre></td></tr></table></figure>
<p><strong>注意：</strong> 一些生产级的代理原生支持circuit breaker模式——比如<a href="http://vulcand.github.io/" target="_blank" rel="external">Vulcand</a>，<a href="https://www.nginx.com/products/" target="_blank" rel="external">Nginx plus</a>和<a href="https://lyft.github.io/envoy/" target="_blank" rel="external">Envoy</a> 。</p>
<h3 id="生产模式：限制速率和连接的sidecar模式"><a href="#生产模式：限制速率和连接的sidecar模式" class="headerlink" title="生产模式：限制速率和连接的sidecar模式"></a>生产模式：限制速率和连接的sidecar模式</h3><p>前面的例子中，我们已经使用了一个sidecar模式——Pod本地的一个特殊代理，给服务添加额外的逻辑，比如错误检测、TLS终止和其他特性。</p>
<p>这个例子使用Nginx代理的sidecar模式，添加了速率和连接的限制：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line">$ <span class="built_in">cd</span> prod/sidecar</div><div class="line">$ docker build -t $(minikube ip):5000/sidecar:0.0.1 <span class="_">-f</span> sidecar.dockerfile .</div><div class="line">$ docker push $(minikube ip):5000/sidecar:0.0.1</div><div class="line">$ docker build -t $(minikube ip):5000/service:0.0.1 <span class="_">-f</span> service.dockerfile .</div><div class="line">$ docker push $(minikube ip):5000/service:0.0.1</div><div class="line">$ kubectl apply <span class="_">-f</span> sidecar.yaml</div><div class="line">deployment <span class="string">"sidecar"</span> configured</div></pre></td></tr></table></figure>
<p>试着以超过每秒一次的频率来请求服务，你将看到速率限制生效：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">$ kubectl run -i -t --rm cli --image=tutum/curl --restart=Never</div><div class="line">curl http://sidecar</div></pre></td></tr></table></figure>
<p><a href="https://istio.io/docs/concepts/what-is-istio/overview.html#architecture" target="_blank" rel="external">Istio</a> 是一个嵌入了这个设计的平台，还可以找到其他的例子。</p>
]]></content>
<summary type="html">
<h1 id="Kubernetes生产模式-。。。及反模式"><a href="#Kubernetes生产模式-。。。及反模式" class="headerlink" title="Kubernetes生产模式 。。。及反模式"></a>Kubernetes生产模式 。。。及反模式</h1><p>本文是一篇英文博客的翻译，<a href="https://gravitational.com/blog/kubernetes-production-patterns/">原文地址</a>。<br>以下开始正文：</p>
<p>我们将探索一些有用的技术来提高Kubernetes部署的弹性和高可用性，同时看看一些和Docker和Kubernetes协作的常见错误，以避免犯错。<br>
</summary>
<category term="Kubernetes" scheme="https://caitianxiong.com/tags/Kubernetes/"/>
<category term="K8s" scheme="https://caitianxiong.com/tags/K8s/"/>
<category term="docker" scheme="https://caitianxiong.com/tags/docker/"/>
</entry>
<entry>
<title>K8s官方文档笔记</title>
<link href="https://caitianxiong.com/2017/08/15/K8s%E5%AE%98%E6%96%B9%E6%96%87%E6%A1%A3%E7%AC%94%E8%AE%B0/"/>
<id>https://caitianxiong.com/2017/08/15/K8s官方文档笔记/</id>
<published>2017-08-14T16:00:00.000Z</published>
<updated>2020-01-17T14:32:39.314Z</updated>
<content type="html"><![CDATA[<p>本篇用于记录K8s官方英文文档的重点笔记。文档地址：<a href="https://kubernetes.io/docs/" target="_blank" rel="external">https://kubernetes.io/docs/</a><br><a id="more"></a></p>
<h3 id="1-Pod-共享的资源包括："><a href="#1-Pod-共享的资源包括：" class="headerlink" title="1. Pod 共享的资源包括："></a>1. Pod 共享的资源包括：</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">Shared storage, as Volumes</div><div class="line">Networking, as a unique cluster IP address</div><div class="line">Information about how to run each container, such as the container image version or specific ports to use</div></pre></td></tr></table></figure>
<h3 id="2-节点上运行的进程："><a href="#2-节点上运行的进程：" class="headerlink" title="2. 节点上运行的进程："></a>2. 节点上运行的进程：</h3><h4 id="Master"><a href="#Master" class="headerlink" title="Master"></a>Master</h4><ul>
<li>kube-apiserver</li>
<li>kube-controller-manager </li>
<li>kube-scheduler<h4 id="Node"><a href="#Node" class="headerlink" title="Node"></a>Node</h4></li>
<li>kubelet, which communicates with the Kubernetes Master.</li>
<li>kube-proxy, a network proxy which reflects Kubernetes networking services on each node.</li>
</ul>
<h3 id="3-K8s常用命令"><a href="#3-K8s常用命令" class="headerlink" title="3. K8s常用命令"></a>3. K8s常用命令</h3><h3 id="4-Scale-UP-DOWN"><a href="#4-Scale-UP-DOWN" class="headerlink" title="4. Scale UP/DOWN"></a>4. Scale UP/DOWN</h3><p>用scale命令，多pod会自动LB</p>
<h3 id="5-滚动更新"><a href="#5-滚动更新" class="headerlink" title="5. 滚动更新"></a>5. 滚动更新</h3>]]></content>
<summary type="html">
<p>本篇用于记录K8s官方英文文档的重点笔记。文档地址：<a href="https://kubernetes.io/docs/">https://kubernetes.io/docs/</a><br>
</summary>
<category term="Kubernetes" scheme="https://caitianxiong.com/tags/Kubernetes/"/>
<category term="K8s" scheme="https://caitianxiong.com/tags/K8s/"/>
</entry>
<entry>
<title>DCOS学习——组件介绍</title>
<link href="https://caitianxiong.com/2017/08/02/dcos%E5%AD%A6%E4%B9%A0%E2%80%94%E2%80%94%E7%BB%84%E4%BB%B6%E4%BB%8B%E7%BB%8D/"/>
<id>https://caitianxiong.com/2017/08/02/dcos学习——组件介绍/</id>
<published>2017-08-01T16:00:00.000Z</published>
<updated>2020-01-17T14:05:43.862Z</updated>
<content type="html"><![CDATA[<p>本篇对DCOS系统的组件进行简单介绍。<br><a id="more"></a></p>
<h3 id="1-docker"><a href="#1-docker" class="headerlink" title="1. docker"></a>1. docker</h3><p>docker是目前最流行的容器隔离技术。</p>
<h3 id="2-etcd"><a href="#2-etcd" class="headerlink" title="2. etcd"></a>2. etcd</h3><p>etcd 是 CoreOS 团队于 2013 年 6 月发起的开源项目，它的目标是构建一个高可用的分布式键值（key-value）数据库，基于 Go 语言实现。<br>在k8s架构中，用于记录所有状态的存储数据库。</p>
<h3 id="3-k8s"><a href="#3-k8s" class="headerlink" title="3. k8s"></a>3. k8s</h3><p>k8s的服务大概分为三类：主节点服务、工作节点服务和其它服务。</p>
<ul>
<li>主节点服务<br>apiserver 是整个系统的对外接口，提供 RESTful 方式供客户端和其它组件调用；<br>scheduler 负责对资源进行调度，分配某个 pod 到某个节点上；<br>controller-manager 负责管理控制器，包括 endpoint-controller（刷新服务和 pod 的关联信息）和 replication-controller（维护某个 pod 的复制为配置的数值）。</li>
<li>工作节点服务<br>kubelet 是工作节点执行操作的 agent，负责具体的容器生命周期管理，根据从数据库中获取的信息来管理容器，并上报 pod 运行状态等；<br>proxy 为 pod 上的服务提供访问的代理。</li>
<li>其它服务<br>etcd 是所有状态的存储数据库；</li>
</ul>
<h3 id="4-Mesos架构组件介绍"><a href="#4-Mesos架构组件介绍" class="headerlink" title="4. Mesos架构组件介绍"></a>4. Mesos架构组件介绍</h3><ul>
<li>Mesos<br>对集群资源进行抽象和管理，类似操作系统的内核。主要由C++实现。</li>
<li>Marathon<br>是可以跟Mesos一起协作的一个framework，基于Scala实现，可以实现保持应用的持续运行。</li>
<li>ZooKeeper<br>一个分布式集群中的信息同步工具，通过自动在多个节点中选举leader，保障多个节点间的某些信息保持一致。</li>
</ul>
]]></content>
<summary type="html">
<p>本篇对DCOS系统的组件进行简单介绍。<br>
</summary>
<category term="docker" scheme="https://caitianxiong.com/tags/docker/"/>
<category term="DCOS" scheme="https://caitianxiong.com/tags/DCOS/"/>
</entry>
<entry>
<title>《Docker——从入门到实践》笔记</title>
<link href="https://caitianxiong.com/2017/07/20/%E3%80%8ADocker%E2%80%94%E2%80%94%E4%BB%8E%E5%85%A5%E9%97%A8%E5%88%B0%E5%AE%9E%E8%B7%B5%E3%80%8B%E7%AC%94%E8%AE%B0/"/>
<id>https://caitianxiong.com/2017/07/20/《Docker——从入门到实践》笔记/</id>
<published>2017-07-19T16:00:00.000Z</published>
<updated>2020-01-17T13:51:54.874Z</updated>
<content type="html"><![CDATA[<p>本篇博客是开源书<a href="https://yeasy.gitbooks.io/docker_practice/" target="_blank" rel="external">《Docker——从入门到实践》</a>的读书笔记。<br><a id="more"></a></p>
<h3 id="1-docker简介"><a href="#1-docker简介" class="headerlink" title="1. docker简介"></a>1. docker简介</h3><ul>
<li>最初实现是基于 LXC，从 0.7 以后开始去除 LXC，转而使用自行开发的 libcontainer，从 1.11 开始，则进一步演进为使用 runC 和 containerd。</li>
</ul>
<h3 id="2-镜像"><a href="#2-镜像" class="headerlink" title="2. 镜像"></a>2. 镜像</h3><ul>
<li>运行容器 docker run –name web3 -d -p 82:80 nginx:v3</li>
<li>可以从仓库pull镜像来使用，也可以自己build镜像，不过要注意删除无用的数据</li>
</ul>
<h3 id="3-容器"><a href="#3-容器" class="headerlink" title="3. 容器"></a>3. 容器</h3><ul>
<li>一些操作：启动run 、后台运行-d、终止stop、导入import、导出export、删除rm<br>用 <code>docker container ls -a</code> 命令可以查看所有已经创建的包括终止状态的容器，如果数量太多要一个个删除可能会很麻烦，用下面的命令可以清理掉所有处于终止状态的容器。<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">$ docker container prune</div></pre></td></tr></table></figure>
</li>
</ul>
<h3 id="4-仓库"><a href="#4-仓库" class="headerlink" title="4. 仓库"></a>4. 仓库</h3><ul>
<li>可以使用公共的Docker Hub</li>
<li>可以建私有仓库，比如用VMware开源的镜像仓库项目Harbor</li>
</ul>
<h3 id="5-使用数据卷"><a href="#5-使用数据卷" class="headerlink" title="5. 使用数据卷"></a>5. 使用数据卷</h3><ul>
<li>数据卷：<br>可以使用本地路径作为数据存储位置，长久存储数据，独立于单个容器的生命周期</li>
<li>数据卷容器<br>用于容器间共享数据</li>
<li>可以使用数据卷容器来备份、恢复、迁移数据卷</li>
</ul>
<h3 id="6-使用网络"><a href="#6-使用网络" class="headerlink" title="6. 使用网络"></a>6. 使用网络</h3><p>docker允许通过外部访问容器或容器互联的方式来提供网络服务。</p>
]]></content>
<summary type="html">
<p>本篇博客是开源书<a href="https://yeasy.gitbooks.io/docker_practice/">《Docker——从入门到实践》</a>的读书笔记。<br>
</summary>
<category term="docker" scheme="https://caitianxiong.com/tags/docker/"/>
</entry>
<entry>
<title>DCOS基本概念学习</title>
<link href="https://caitianxiong.com/2017/07/14/dcos%E5%AD%A6%E4%B9%A0%E2%80%94%E2%80%94%E5%9F%BA%E6%9C%AC%E6%A6%82%E5%BF%B5/"/>
<id>https://caitianxiong.com/2017/07/14/dcos学习——基本概念/</id>
<published>2017-07-13T16:00:00.000Z</published>
<updated>2020-02-02T01:47:15.047Z</updated>
<content type="html"><![CDATA[<h3 id="0-概况"><a href="#0-概况" class="headerlink" title="0 概况"></a>0 概况</h3><ul>
<li>需求来源<br>一级私有云项目组技术储备需要，承蒙杜总的信任，给我分配了学习DCOS的任务。</li>
<li>任务时间<br>半个月</li>
<li>任务内容<br>学习LXC原理、docker机制、k8s，了解DCOS的功能和局限。</li>
</ul>
<a id="more"></a>
<h3 id="1-基本概念"><a href="#1-基本概念" class="headerlink" title="1 基本概念"></a>1 基本概念</h3><ul>
<li>DCOS（DC/OS）<br>是一种数据中心操作系统（Data Center Operation System），通常部署在IDC机房，用于构建和管理面向企业级、云端的大规模业务集群。</li>
<li>容器（封装和隔离）<br>容器是一种轻量级、可移植、自包含的软件打包技术，使应用程序可以在几乎任何地方以相同的方式运行。开发人员在自己笔记本上创建并测试好的容器，无需任何修改就能够在生产系统的虚拟机、物理服务器或公有云主机上运行。<br>容器是一种轻量级的操作系统虚拟化，为App提供独立的、受控的运行环境。容器由两部分组成：应用程序本身和依赖。</li>
<li><p>docker<br>Docker是容器管理工具，创建容器、启停容器、删除容器等。</p>
</li>
<li><p>应用（App）<br>容器服务的核心目标，可以是一个服务软件。</p>
</li>
<li><p>Mesos<br>最初由 UC Berkeley 的 AMP 实验室于 2009 年发起，遵循 Apache 协议，目前已经成立了 Mesosphere 公司进行运营。Mesos 可以将整个数据中心的资源（包括 CPU、内存、存储、网络等）进行抽象和调度，使得多个应用同时运行在集群中分享资源，并无需关心资源的物理分布情况。 Mesos 项目是 Mesosphere 公司 Datacenter Operating System (DCOS) 产品的<strong>核心部件</strong>。</p>
</li>
<li>Kubernetes<br>Google 团队发起的开源项目，它的目标是管理跨多个主机的容器，提供基本的部署，维护以及应用伸缩，主要实现语言为Go语言。</li>
</ul>
<h3 id="2-PPT笔记"><a href="#2-PPT笔记" class="headerlink" title="2 PPT笔记"></a>2 PPT笔记</h3><ul>
<li>学习目标<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line">掌握Docker是什么 </div><div class="line">掌握容器和虚拟机的区别 </div><div class="line">掌握容器实现的原理 </div><div class="line">掌握容器场景化应用 </div><div class="line">掌握Docker工具链使用</div></pre></td></tr></table></figure>
</li>
</ul>
<h4 id="Docker是什么"><a href="#Docker是什么" class="headerlink" title="Docker是什么"></a>Docker是什么</h4><p>Docker是容器管理工具，用于创建容器、启停容器、删除容器等。</p>
<h4 id="容器和虚拟机的区别"><a href="#容器和虚拟机的区别" class="headerlink" title="容器和虚拟机的区别"></a>容器和虚拟机的区别</h4><p><img src="/images/container_vs_vm.png" alt="Container VS VM"><br><img src="/images/why_vm_heavy.png" alt="为什么虚机开销大"><br>谈到容器，就不得不将它与虚拟机进行对比，因为两者都是为应用提供封装和隔离。<br>容器由两部分组成：应用程序本身和依赖：比如应用程序需要的库或其他软件。<br> 容器在 Host 操作系统的用户空间中运行，与操作系统的其他进程隔离。这一点显著区别于的虚拟机。<br>传统的虚拟化技术，比如 VMWare, KVM, Xen，目标是创建完整的虚拟机。为了运行应用，除了部署应用本身及其依赖（通常几十 MB），还得安装整个操作系统（几十 GB）。<br>如图所示，由于所有的容器共享同一个 Host OS，这使得容器在体积上要比虚拟机小很多。另外，启动容器不需要启动整个操作系统，所以容器部署和启动速度更快，开销更小，也更容易迁移。</p>
<h4 id="容器实现的原理"><a href="#容器实现的原理" class="headerlink" title="容器实现的原理"></a>容器实现的原理</h4><ul>
<li><p>docker实现原理<br><img src="/images/docker实现原理.png" alt="docker实现原理"></p>
</li>
<li><p>docker架构<br><img src="/images/docker架构.png" alt="docker架构"></p>
</li>
</ul>
<h4 id="Docker工具链使用"><a href="#Docker工具链使用" class="headerlink" title="Docker工具链使用"></a>Docker工具链使用</h4><p><img src="/images/docker_commands.png" alt="docker命令"></p>
]]></content>
<summary type="html">
<h3 id="0-概况"><a href="#0-概况" class="headerlink" title="0 概况"></a>0 概况</h3><ul>
<li>需求来源<br>一级私有云项目组技术储备需要，承蒙杜总的信任，给我分配了学习DCOS的任务。</li>
<li>任务时间<br>半个月</li>
<li>任务内容<br>学习LXC原理、docker机制、k8s，了解DCOS的功能和局限。</li>
</ul>
</summary>
<category term="docker" scheme="https://caitianxiong.com/tags/docker/"/>
<category term="DCOS" scheme="https://caitianxiong.com/tags/DCOS/"/>
</entry>
<entry>
<title>OLTP场景测试工具——Orion</title>
<link href="https://caitianxiong.com/2017/03/30/oltp-orion/"/>
<id>https://caitianxiong.com/2017/03/30/oltp-orion/</id>
<published>2017-03-30T11:45:01.000Z</published>
<updated>2019-11-07T03:27:52.692Z</updated>
<content type="html"><![CDATA[<h3 id="1-OLTP场景介绍"><a href="#1-OLTP场景介绍" class="headerlink" title="1. OLTP场景介绍"></a>1. OLTP场景介绍</h3><p>关于OLTP和其他相关概念的介绍见上篇，本篇不再赘述。<br><a id="more"></a></p>
<h3 id="2-Orion介绍"><a href="#2-Orion介绍" class="headerlink" title="2. Orion介绍"></a>2. Orion介绍</h3><p>Orion是 Oracle IO Numbers的缩写，是Oracle公司提供的一个测试工具。<a href="http://www.oracle.com/technetwork/cn/topics/index-088165-zhs.html" target="_blank" rel="external">官网</a>介绍：</p>
<blockquote>
<p>ORION (Oracle I/O Calibration Tool) 是校准用于 Oracle 数据库的存储系统 I/O 性能的独立工具。校准结果对于了解存储系统的性能有很大帮助，不仅可以找出影响 Oracle 数据库性能的问题，还能测量新数据库安装的大小。由于 ORION 是一个独立工具，用户不需要创建和运行 Oracle 数据库。 </p>
<p>为了尽可能地模拟 Oracle 数据库，ORION 使用和 Oracle 相同的 I/O 软件集生成一个合成的 I/O 负载。可以将 ORION 配置为生成很多 I/O 负载，包括一个模拟 OLTP 和数据仓库负载的 I/O 负载。</p>
</blockquote>
<h3 id="3-Orion的安装"><a href="#3-Orion的安装" class="headerlink" title="3. Orion的安装"></a>3. Orion的安装</h3><p>Orion的下载需要去Oracle网站。<br>1) 进入www.oracle.com/technetwork/cn/topics/index-088165-zhs.html<br>2) 点击“接受许可协议”，选择Linux(x86_64)版本下载，可能要求输入Oracle ID<br>3) 解压缩安装包 <code>gzip -d orion_linux_x86-64.gz</code><br>4) 解压后得到一个文件，赋予执行权限 <code>chmod +x orion_linux_x86-64</code></p>
<h3 id="4-Orion的用法"><a href="#4-Orion的用法" class="headerlink" title="4. Orion的用法"></a>4. Orion的用法</h3><p>命令行语法 <code>./orion_linux_x86_64 -run [opiton] [-testname] <testname></code></p>
<h3 id="4-1-常用测试参数说明"><a href="#4-1-常用测试参数说明" class="headerlink" title="4.1 常用测试参数说明"></a>4.1 常用测试参数说明</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div></pre></td><td class="code"><pre><div class="line">必选参数：</div><div class="line">run 要运行的负载类型(simple, normal, advanced, dss, oltp)</div><div class="line"> simple - 在不同负载下先测试8k随机小IO，然后测试随机1M大IO。</div><div class="line"> normal - 测试随机8k小IO和随机1M大IO的组合。</div><div class="line"> advanced - 运行用户用可选参数指定的负载。</div><div class="line"> dss - 运行负载渐增的1M随机大 IO 来测定最大的 吞吐量。</div><div class="line"> oltp - 运行负载渐增的8k随机小 IO 来测定最大的 IOPS。</div><div class="line">可选参数:</div><div class="line">testname 测试名</div><div class="line">num_disks 磁盘数（物理主轴数）。默认是 <testname>.lun 中的 LUN 数量。</div><div class="line">size_small 小 IO 的大小 (KB) - 默认8</div><div class="line">size_large 大 IO 的大小 (KB) - 默认 1024</div><div class="line">type 大 IO 类型 (rand, seq) - 默认 rand</div><div class="line"> rand - 随机大 IO</div><div class="line"> seq - 大 IO 的顺序流</div><div class="line">num_streamIO 每个流并发IO数 (只有当type设置为seq时) - 默认 4</div><div class="line">write 写比例 (会损坏已有数据) - 默认 0</div><div class="line">cache_size 阵列缓存的大小（MB）。除非这个选项被设置为0，否则Orion在每个顺序数据点之前会做一定量的随机IO。</div><div class="line"> 如果指定了值，缓存热身会读或写指定大小的IO。默认行为：在每个数据点前填充cache持续2分钟。</div><div class="line">duration 每个数据点测试时长 (秒) - 默认 60</div><div class="line">num_small outstanding 小 IO 数量 (只有当matrix 是point, col, or max时) - 无默认</div><div class="line">num_large 随机读写情况下，outstanding大IO数量。</div><div class="line"> 顺序读写情况下，流数量。 (只有当matrix 是point, col, or max时) - 无默认</div><div class="line">matrix 一个由不同大小IO负载级别组成的Orion测试。这些数据点可以表示为二维矩阵，矩阵的每一列代表一个固定的小IO负载，每一行代表一个固定的大IO负载。第一行没有大IO负载，第一列没有小IO负载。一个Orion测试可以是一个单独的点、一行、一列或者整个矩阵，依赖如下选项值的设定（默认basic）：</div><div class="line"> basic - 测试第一行和第一列</div><div class="line"> detailed - 测试整个矩阵</div><div class="line"> point - 在负载级别 num_small, num_large 这一数据点测试</div><div class="line"> col - 在num_small 指定小IO下遍历大IO</div><div class="line"> row - 在num_large指定大IO下遍历小IO</div><div class="line"> max - 遍历大小IO最大到num_large, num_small</div><div class="line">verbose 如果设定了，打印过程信息到标准输出。默认 -- 未设定</div></pre></td></tr></table></figure>
<p>Orion对于oltp参数的说明：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">oltp - run with random 8K small IOs at increasing loads to determine the maximum IOPS</div><div class="line">用负载渐增的8k随机小 IO 来测定最大的 IOPS</div></pre></td></tr></table></figure></p>
<h3 id="4-2-测试步骤"><a href="#4-2-测试步骤" class="headerlink" title="4.2 测试步骤"></a>4.2 测试步骤</h3><p>1) 创建测试文件<br>文件名为<code><testname>.lun</code>，如 <code>mytest.lun</code> ，在里面写入测试的目标磁盘，每行一个，比如<code>/dev/vdc</code> 。<br>2) 执行OLTP测试<br><code>./orion_linux_x86-64 -run oltp -testname mytest</code><br><code>-testname</code> 后接的参数要与上一步创建的文件名一致，此处为<code>mytest</code>。这个测试会进行几组8K大小的随机读写和1M大小的连续读写测试。前者用来测定IOPS能力，后者测定吞吐量</p>
<h3 id="5-测试注意点"><a href="#5-测试注意点" class="headerlink" title="5. 测试注意点"></a>5. 测试注意点</h3><ul>
<li>Orion的写测试直接在磁盘级进行，会损坏已有的数据，所以最好要用空磁盘进行测试。</li>
<li>Orion模拟的OLTP测试场景无大IO，只是测试随机8k小IO的最大IOPS。可以带-write和-duration参数，可指定-size_small指定IO大小。 -type, -matrix, -num_small or -num_large这些参数不是能设置的。-size_large参数设置了也没有用处。</li>
</ul>
<h3 id="6-测试结果解读"><a href="#6-测试结果解读" class="headerlink" title="6. 测试结果解读"></a>6. 测试结果解读</h3><p>Orion的每次测试，会输出5个测试结果的文件。输出的各个文件说明如下：（以测试名为<code>mytest</code>为例）</p>
<ul>
<li><p>mytest_YYYYMMDD_HHmm_summary.txt<br><code>YYYYMMDD_HHmm</code>表示日期时间，下同。测试结果的概括，该文件包括命令行输入的参数、大随机/顺序负载下的最大吞吐量、小随机负载下的最大IOPS和小随机负载下的最小时延。</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div></pre></td><td class="code"><pre><div class="line">ORION VERSION 11.1.0.7.0</div><div class="line"></div><div class="line">Commandline:</div><div class="line">-run oltp -testname oltp</div><div class="line"></div><div class="line">This maps to this test:</div><div class="line">Test: oltp</div><div class="line">Small IO size: 8 KB</div><div class="line">Large IO size: 1024 KB</div><div class="line">IO Types: Small Random IOs, Large Random IOs</div><div class="line">Simulated Array Type: CONCAT</div><div class="line">Write: 0%</div><div class="line">Cache Size: Not Entered</div><div class="line">Duration for each Data Point: 60 seconds</div><div class="line">Small Columns:, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20</div><div class="line">Large Columns:, 0</div><div class="line">Total Data Points: 21</div><div class="line"></div><div class="line">Name: /dev/vdc Size: 107374182400</div><div class="line">1 FILEs found.</div><div class="line"></div><div class="line">Maximum Small IOPS=24249 @ Small=18 and Large=0 # 小IOPS最大值</div><div class="line">Minimum Small Latency=0.46 @ Small=4 and Large=0 # 最小延迟时间</div></pre></td></tr></table></figure>
</li>
<li><p>mytest_YYYYMMDD_HHmm_mbps.csv<br>大随机/顺序负载下的最大吞吐量数据的csv文件。（OLTP场景下这个文件是没有数据的）</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">arge/Small, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40</div></pre></td></tr></table></figure>
</li>
<li><p>mytest_YYYYMMDD_HHmm_iops.csv<br>IOPS数据的csv文件。OLTP场景的测试，只有小IO的负载，因此只有一行数据。</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">Large/Small, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40</div><div class="line"> 0, 24141, 40656, 37475, 39398, 40970, 40851, 40530, 39910, 39082, 39361, 38264, 37785, 37637, 38017, 38405, 37918, 37968, 37367, 38493, 38823</div></pre></td></tr></table></figure>
</li>
<li><p>mytest_YYYYMMDD_HHmm_lat.csv<br>延时数据的csv文件。同样，只有小IO的负载，因此只有一行数据。</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">Large/Small, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40</div><div class="line"> 0, 0.08, 0.10, 0.16, 0.20, 0.24, 0.29, 0.34, 0.40, 0.46, 0.51, 0.57, 0.63, 0.69, 0.74, 0.78, 0.84, 0.89, 0.96, 0.99, 1.03</div></pre></td></tr></table></figure>
</li>
<li><p>mytest_YYYYMMDD_HHmm_trace.txt<br>跟踪了测试过程中各个数据点的各项数据，最为详细。可以看到随着数据点的增加，small的负载在不断增加，而large的负载保持为0。</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div></pre></td><td class="code"><pre><div class="line">ran (small): VLun = 0 Size = 107372216832</div><div class="line">ran (small): Index = 0 Count = 724213 Avg Lat = 0.08</div><div class="line">ran (small): Index = 1 Count = 724256 Avg Lat = 0.08</div><div class="line">ran (small): nio=1448469 nior=1448469 niow=0 req w%=0 act w%=0</div><div class="line">ran (small): my 2 oth 0 iops 24141 size 8 K lat 0.08 ms bw = 188.60 MBps dur 60.00 s READ</div><div class="line"></div><div class="line"></div><div class="line"></div><div class="line">TEST START</div><div class="line"></div><div class="line">Point 1 (small=2, large=0) of 22</div><div class="line">Valid small 1 Valid large 1</div><div class="line">Valid</div><div class="line"></div><div class="line">ran (small): VLun = 0 Size = 107372216832</div><div class="line">ran (small): Index = 0 Count = 609842 Avg Lat = 0.10</div><div class="line">ran (small): Index = 1 Count = 609883 Avg Lat = 0.10</div><div class="line">ran (small): Index = 2 Count = 609788 Avg Lat = 0.10</div><div class="line">ran (small): Index = 3 Count = 609869 Avg Lat = 0.10</div><div class="line">ran (small): nio=2439382 nior=2439382 niow=0 req w%=0 act w%=0</div><div class="line">ran (small): my 4 oth 0 iops 40656 size 8 K lat 0.10 ms bw = 317.63 MBps dur 60.00 s READ</div><div class="line"></div><div class="line">Point 2 (small=4, large=0) of 22</div><div class="line">Valid small 1 Valid large 1</div><div class="line">Valid</div><div class="line"></div><div class="line">ran (small): VLun = 0 Size = 107372216832</div><div class="line">ran (small): Index = 0 Count = 374727 Avg Lat = 0.16</div><div class="line">ran (small): Index = 1 Count = 374789 Avg Lat = 0.16</div><div class="line">ran (small): Index = 2 Count = 374671 Avg Lat = 0.16</div><div class="line">ran (small): Index = 3 Count = 374833 Avg Lat = 0.16</div><div class="line">ran (small): Index = 4 Count = 374772 Avg Lat = 0.16</div><div class="line">ran (small): Index = 5 Count = 374718 Avg Lat = 0.16</div><div class="line">ran (small): nio=2248510 nior=2248510 niow=0 req w%=0 act w%=0</div><div class="line">ran (small): my 6 oth 0 iops 37475 size 8 K lat 0.16 ms bw = 292.77 MBps dur 60.00 s READ</div><div class="line"></div><div class="line">Point 3 (small=6, large=0) of 22</div><div class="line">...</div><div class="line"></div><div class="line">Point 20 (small=40, large=0) of 22</div><div class="line">Valid small 1 Valid large 1</div><div class="line">Valid</div><div class="line"></div><div class="line"></div><div class="line"></div><div class="line">TEST END</div></pre></td></tr></table></figure>
</li>
</ul>
<h3 id="7-Orion的优缺点"><a href="#7-Orion的优缺点" class="headerlink" title="7. Orion的优缺点"></a>7. Orion的优缺点</h3><h4 id="优点"><a href="#优点" class="headerlink" title="优点"></a>优点</h4><ul>
<li>不需要安装Oracle数据库，不需要准备测试数据</li>
<li>测试结束自动输出csv文件，方便画图分析</li>
</ul>
<h4 id="缺点"><a href="#缺点" class="headerlink" title="缺点"></a>缺点</h4><ul>
<li>闭源工具，只有一个二进制的可执行文件，无法分析测试原理和具体实现</li>
<li>无法自定义测试的总时间、加压幅度，完全是软件自动设置</li>
<li>Oracle不对Orion提供支持</li>
</ul>
<h3 id="8-参考资料"><a href="#8-参考资料" class="headerlink" title="8. 参考资料"></a>8. 参考资料</h3><ul>
<li><a href="http://www.oracle.com/technetwork/cn/topics/index-088165-zhs.html" target="_blank" rel="external">Oracle官网上对Orion的介绍及用户指南</a></li>
<li><a href="https://yq.aliyun.com/articles/62615" target="_blank" rel="external">阿里云ECS使用Oracle</a></li>
<li><a href="http://dbaplus.cn/news-10-190-1.html" target="_blank" rel="external">Oracle IO性能测试工具Orion详解</a></li>
<li><a href="https://wenku.baidu.com/view/a2177b80680203d8ce2f24ff.html" target="_blank" rel="external">orion使用介绍</a></li>
<li><a href="http://blog.csdn.net/shaunfang/article/details/11194289" target="_blank" rel="external">国内公有云对比（2）- 性能篇</a></li>
</ul>
]]></content>
<summary type="html">
<h3 id="1-OLTP场景介绍"><a href="#1-OLTP场景介绍" class="headerlink" title="1. OLTP场景介绍"></a>1. OLTP场景介绍</h3><p>关于OLTP和其他相关概念的介绍见上篇，本篇不再赘述。<br>
</summary>
<category term="测试" scheme="https://caitianxiong.com/tags/%E6%B5%8B%E8%AF%95/"/>
</entry>
<entry>
<title>OLTP场景测试工具——sysbench</title>
<link href="https://caitianxiong.com/2017/03/30/oltp-sysbench/"/>
<id>https://caitianxiong.com/2017/03/30/oltp-sysbench/</id>
<published>2017-03-30T11:38:07.000Z</published>
<updated>2019-11-07T03:27:52.692Z</updated>
<content type="html"><![CDATA[<h3 id="1-OLTP场景介绍"><a href="#1-OLTP场景介绍" class="headerlink" title="1. OLTP场景介绍"></a>1. OLTP场景介绍</h3><p>On-Line Transaction Processing (OLTP) 在线事务处理，是典型的数据库应用场景，随机读写频繁，IOPS是关键指标。在证券交易系统中，单个数据库每秒处理的事务往往超过几百个，设置几千个；select语句的执行数量每秒几千甚至几万个。例如银行、证券交易、机票预订、电子商务系统、大型商场eBay的业务管理数据库系统就是很典型的OLTP数据库。<br><a id="more"></a></p>
<h4 id="相关概念"><a href="#相关概念" class="headerlink" title="相关概念"></a>相关概念</h4><ul>
<li>TPS<br>Transaction Per Second，数据库每秒执行的事务数，以 commit 为准。 </li>
<li>QPS<br>QueryPerSecond，数据库每秒执行的 SQL 数(含 insert、select、update、delete 等)。 </li>
</ul>
<h3 id="2-sysbench介绍"><a href="#2-sysbench介绍" class="headerlink" title="2. sysbench介绍"></a>2. sysbench介绍</h3><p>sysbench是一个跨平台且支持多线程的模块化基准测试工具，可以用来进行CPU、内存、磁盘I/O、线程、数据库的性能测试。目前支持的数据库有MySQL、Oracle和PostgreSQL。<br>本次调研主要关注 sysbench 的数据库性能测试功能。使用sysbench可以绕过复杂的数据库基准设置，甚至在没有安装数据库的前提下，快速了解数据库系统的性能。 </p>
<h3 id="3-sysbench的安装"><a href="#3-sysbench的安装" class="headerlink" title="3. sysbench的安装"></a>3. sysbench的安装</h3><p>调研的 SysBench 版本为 0.5，下载地址为 <a href="https://github.com/akopytov/sysbench" target="_blank" rel="external">https://github.com/akopytov/sysbench</a><br>在CentOS 7.2上，确保已经安装了MariaDB：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line"># mysql --version</div><div class="line">mysql Ver 15.1 Distrib 10.1.20-MariaDB, for Linux (x86_64) using EditLine wrapper</div></pre></td></tr></table></figure></p>
<p>执行如下命令安装 SysBench：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div></pre></td><td class="code"><pre><div class="line"># yum install gcc gcc-c++ autoconf automake make libtool bzr mysql-devel</div><div class="line"># unzip ysbench-0.5.zip</div><div class="line"># cd sysbench-0.5</div><div class="line"># ./autogen.sh</div><div class="line"># ./configure --prefix=/usr --mandir=/usr/share/man</div><div class="line"># make</div></pre></td></tr></table></figure></p>
<h3 id="4-sysbench的用法"><a href="#4-sysbench的用法" class="headerlink" title="4. sysbench的用法"></a>4. sysbench的用法</h3><p>命令行语法：<code>sysbench [options]... [testname] [command]</code> </p>
<h4 id="4-1-常用测试参数说明"><a href="#4-1-常用测试参数说明" class="headerlink" title="4.1 常用测试参数说明"></a>4.1 常用测试参数说明</h4><table>
<thead>
<tr>
<th>参数</th>
<th>说明</th>
<th>默认值</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>--num-threads</code></td>
<td>测试创建的线程数，模拟客户端连接数</td>
<td>1</td>
</tr>
<tr>
<td><code>--max-requests</code></td>
<td>最大总请求数，默认值0表示不限制</td>
<td>0</td>
</tr>
<tr>
<td><code>--max-time</code></td>
<td>测试执行总时间秒数，0表示不限制</td>
<td>10</td>
</tr>
<tr>
<td><code>--test</code></td>
<td>测试类型，可选值有fileio、cpu、memory、threads、mutex、oltp</td>
<td></td>
</tr>
<tr>
<td><code>--oltp-table-size</code></td>
<td>OLTP 测试表数据行数</td>
<td></td>
</tr>
<tr>
<td><code>--oltp-table-count</code></td>
<td>OLTP 测试表数目</td>
<td></td>
</tr>
<tr>
<td><code>–oltp-test-mode</code></td>
<td>指定测试模式，可选值有simeple、complex、nontrx</td>
<td>complex</td>
</tr>
<tr>
<td><code>--db-driver</code></td>
<td>数据库驱动</td>
<td></td>
</tr>
<tr>
<td><code>--mysql-table-engine</code></td>
<td>mysql数据库存储引擎</td>
<td>innodb</td>
</tr>
<tr>
<td><code>--mysql-host</code></td>
<td>mysql服务器主机名</td>
<td>localhost</td>
</tr>
<tr>
<td><code>--mysql-port</code></td>
<td>mysql服务器端口</td>
<td>3306</td>
</tr>
<tr>
<td><code>--mysql-user</code></td>
<td>mysql用户名</td>
<td>sbtest</td>
</tr>
<tr>
<td><code>--mysql-password</code></td>
<td>mysql密码</td>
<td></td>
</tr>
<tr>
<td><code>--report-interval</code></td>
<td>指定输出中间结果的周期秒数，0表示关闭周期输出</td>
<td>0</td>
</tr>
<tr>
<td><code>--percentile</code></td>
<td>有效查询返回时间百分比</td>
<td>95</td>
</tr>
</tbody>
</table>
<h4 id="4-2-测试步骤"><a href="#4-2-测试步骤" class="headerlink" title="4.2 测试步骤"></a>4.2 测试步骤</h4><p>使用的测试参数根据使用的环境进行设置调整。执行以下命令前，需要手动在MariaDB中创建测试用的数据库<code>sbtest</code>，执行准备命令只是在指定的database中创建table，并不会创建database。</p>
<h5 id="创建测试表"><a href="#创建测试表" class="headerlink" title="创建测试表"></a>创建测试表</h5><p>创建时间较长，建议提前准备。<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">sysbench --report-interval=$interval --num-threads=$threads --max-time=$runtime --max-requests=999999999 --test=oltp --oltp-table-size=$rows --oltp-tables-count=$tables --db-driver=mysql --mysql-table-engine=innodb --mysql-host=$DB_HOST --mysql-port=3306 --mysql-user=$DB_USER --mysql-password=$DB_PW prepare</div></pre></td></tr></table></figure></p>
<p>创建的测试表的结构如下：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div></pre></td><td class="code"><pre><div class="line">MariaDB [sbtest]> desc sbtest1;</div><div class="line">+-------+------------------+------+-----+---------+----------------+</div><div class="line">| Field | Type | Null | Key | Default | Extra |</div><div class="line">+-------+------------------+------+-----+---------+----------------+</div><div class="line">| id | int(10) unsigned | NO | PRI | NULL | auto_increment |</div><div class="line">| k | int(10) unsigned | NO | MUL | 0 | |</div><div class="line">| c | char(120) | NO | | | |</div><div class="line">| pad | char(60) | NO | | | |</div><div class="line">+-------+------------------+------+-----+---------+----------------+</div><div class="line">4 rows in set (0.02 sec)</div></pre></td></tr></table></figure></p>
<h5 id="正式测试"><a href="#正式测试" class="headerlink" title="正式测试"></a>正式测试</h5><figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">sysbench --report-interval=$interval --num-threads=$threads --max-time=$runtime --max-requests=999999999 --test=oltp --oltp-table-size=$rows --oltp-tables-count=$tables --db-driver=mysql --mysql-table-engine=innodb --mysql-host=$DB_HOST --mysql-port=3306 --mysql-user=$DB_USER --mysql-password=$DB_PW run</div></pre></td></tr></table></figure>
<h5 id="清理测试数据"><a href="#清理测试数据" class="headerlink" title="清理测试数据"></a>清理测试数据</h5><figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">sysbench --report-interval=$interval --num-threads=$threads --max-time=$runtime --max-requests=999999999 --test=oltp --oltp-table-size=$rows --oltp-tables-count=$tables --db-driver=mysql --mysql-table-engine=innodb --mysql-host=$DB_HOST --mysql-port=3306 --mysql-user=$DB_USER --mysql-password=$DB_PW cleanup</div></pre></td></tr></table></figure>
<h3 id="5-测试注意点"><a href="#5-测试注意点" class="headerlink" title="5. 测试注意点"></a>5. 测试注意点</h3><p>如何评断一个测试的准确性呢？相同的条件下稳定再现。 只有这样，才能用于分析，才是一个可靠的数据。<br>因为sysbench 0.5 没有提供warmup功能，需要提供warmup时间，时间的长短与数据量也有很大关系。需要具体情况具体分析，根据sysbench周期输出的结果，取tps稳定阶段的数据作为有效数据，必要时可借助Excel画曲线图判断。 另外，建议每次测试前重启进程，清空之前的影响（数据库缓存、文件系统缓存等）。</p>
<h3 id="6-测试结果解读"><a href="#6-测试结果解读" class="headerlink" title="6. 测试结果解读"></a>6. 测试结果解读</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div></pre></td><td class="code"><pre><div class="line">sysbench 0.5: multi-threaded system evaluation benchmark</div><div class="line"></div><div class="line">Running the test with following options:</div><div class="line">Number of threads: 1024</div><div class="line">Report intermediate results every 10 second(s)</div><div class="line">Random number generator seed is 0 and will be ignored</div><div class="line"></div><div class="line"></div><div class="line">Initializing worker threads...</div><div class="line"></div><div class="line">Threads started!</div><div class="line"></div><div class="line">OLTP test statistics:</div><div class="line"> queries performed:</div><div class="line"> read: 77109648 # 读总数</div><div class="line"> write: 22031328 # 写总数</div><div class="line"> other: 11015664 # 其他操作总数(SELECT、INSERT、UPDATE、DELETE之外的操作，例如COMMIT等)</div><div class="line"> total: 110156640 # 全部总数</div><div class="line"> transactions: 5507832 (3059.53 per sec.) # 执行的事务总数，括号内的值即为tps，需重点关注</div><div class="line"> read/write requests: 99140976 (55071.48 per sec.) # 读写总数，括号内的值为qps</div><div class="line"> other operations: 11015664 (6119.05 per sec.) # 其他操作总数(每秒其他操作次数)</div><div class="line"> ignored errors: 0 (0.00 per sec.)</div><div class="line"> reconnects: 0 (0.00 per sec.)</div><div class="line"></div><div class="line">General statistics:</div><div class="line"> total time: 1800.2234s # 总耗时</div><div class="line"> total number of events: 5507832 # 共事务数</div><div class="line"> total time taken by event execution: 1843297.8803s # 所有事务耗时相加(不考虑并行因素)</div><div class="line"> response time:</div><div class="line"> min: 24.46ms # 最小耗时</div><div class="line"> avg: 334.67ms # 平均耗时</div><div class="line"> max: 1174.87ms # 最大耗时</div><div class="line"> approx. 95 percentile: 591.16ms # 95%的tps完成的平均时间</div><div class="line"></div><div class="line">Threads fairness:</div><div class="line"> events (avg/stddev): 5378.7422/86.23</div><div class="line"> execution time (avg/stddev): 1800.0956/0.04</div></pre></td></tr></table></figure>
<p>需要重点关注的是tps、qps和响应的平均耗时。</p>
<h3 id="7-sysbench的优缺点"><a href="#7-sysbench的优缺点" class="headerlink" title="7. sysbench的优缺点"></a>7. sysbench的优缺点</h3><p>sysbench的优点是，不用安装数据库，就可以快速了解数据库系统的性能。缺点是测试表结构太简单，并不能百分百模拟实际业务场景；另外，输出的测试结果概括性太强，需要配合编写其他脚本来收集系统资源的数据，若要画图还需要进一步转化为csv文件。<br>总的来说，是一种方便快捷的benchmark工具，有助于快速了解服务器的性能。</p>
<h3 id="8-参考资料"><a href="#8-参考资料" class="headerlink" title="8. 参考资料"></a>8. 参考资料</h3><ol>
<li><a href="http://mingxinglai.com/cn/2013/07/sysbench/" target="_blank" rel="external">sysbench 0.5使用手册</a></li>
<li><a href="http://www.cnblogs.com/zhoujinyi/archive/2013/04/19/3029134.html" target="_blank" rel="external">sysbench 安装、使用和测试</a></li>
<li><a href="https://github.com/akopytov/sysbench" target="_blank" rel="external">sysbench GitHub主页</a></li>
</ol>
]]></content>
<summary type="html">
<h3 id="1-OLTP场景介绍"><a href="#1-OLTP场景介绍" class="headerlink" title="1. OLTP场景介绍"></a>1. OLTP场景介绍</h3><p>On-Line Transaction Processing (OLTP) 在线事务处理，是典型的数据库应用场景，随机读写频繁，IOPS是关键指标。在证券交易系统中，单个数据库每秒处理的事务往往超过几百个，设置几千个；select语句的执行数量每秒几千甚至几万个。例如银行、证券交易、机票预订、电子商务系统、大型商场eBay的业务管理数据库系统就是很典型的OLTP数据库。<br>
</summary>
<category term="测试" scheme="https://caitianxiong.com/tags/%E6%B5%8B%E8%AF%95/"/>
</entry>
<entry>
<title>集成Ceph到DevStack</title>
<link href="https://caitianxiong.com/2017/02/28/integrate-ceph-into-devstack/"/>
<id>https://caitianxiong.com/2017/02/28/integrate-ceph-into-devstack/</id>
<published>2017-02-28T06:35:53.000Z</published>
<updated>2019-11-07T03:27:52.691Z</updated>
<content type="html"><![CDATA[<h1 id="环境"><a href="#环境" class="headerlink" title="环境"></a>环境</h1><ul>
<li>已经搭建好的 <code>Mon*1 + OSD*3</code> 的Ceph-10.2.5测试集群</li>
<li>All-In-One的Newton版DevStack环境，参考<a href="https://docs.openstack.org/developer/devstack/" target="_blank" rel="external">DevStack部署文档</a><a id="more"></a>
</li>
</ul>
<h1 id="步骤"><a href="#步骤" class="headerlink" title="步骤"></a>步骤</h1><h2 id="创建pool"><a href="#创建pool" class="headerlink" title="创建pool"></a>创建pool</h2><ul>
<li>Ceph集群创建几个pool：images/volumes/backups/vms<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line">ceph osd pool create volumes 128 (pg大小根据自己的Ceph集群规模调整）</div><div class="line">ceph osd pool create images 128</div><div class="line">ceph osd pool create backups 128</div><div class="line">ceph osd pool create vms 128</div></pre></td></tr></table></figure>
</li>
</ul>
<h2 id="配置OpenStack-Ceph客户端"><a href="#配置OpenStack-Ceph客户端" class="headerlink" title="配置OpenStack Ceph客户端"></a>配置OpenStack Ceph客户端</h2><ul>
<li>将ceph.conf配置文件复制到openstack节点，包括：运行 glance-api, cinder-volume, nova-compute 和 cinder-backup 的节点，都作为Ceph client。</li>
</ul>
<h3 id="安装Ceph客户端"><a href="#安装Ceph客户端" class="headerlink" title="安装Ceph客户端"></a>安装Ceph客户端</h3><ul>
<li>在运行glance-api的节点，安装<code>python-rbd</code>库，<code>yum install python-rbd -y</code></li>
<li>在运行nova-compute, cinder-backup 和 cinder-volume 节点，安装python-rbd库和ceph命令行工具：<code>yum install python-rbd ceph-common -y</code></li>
</ul>
<h3 id="配置Ceph客户端认证"><a href="#配置Ceph客户端认证" class="headerlink" title="配置Ceph客户端认证"></a>配置Ceph客户端认证</h3><ul>
<li><p>为Nova/Cinder/Glance创建新用户</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">ceph auth get-or-create client.glance mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=images'</div><div class="line">ceph auth get-or-create client.cinder-backup mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=backups'</div></pre></td></tr></table></figure>
</li>
<li><p>Mitaka及之后的版本，引入了RBD快照的支持（做Nova实例的快照），因此需要配置<code>client.cinder</code> key对images pool的写权限：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">ceph auth get-or-create client.cinder mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=vms, allow rwx pool=images'</div></pre></td></tr></table></figure>
</li>
<li><p>将<code>client.cinder, client.glance, 和 client.cinder-backup</code>的 keyring 添加到相应的节点，并改变其属主：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div></pre></td><td class="code"><pre><div class="line">ceph auth get-or-create client.glance | ssh {your-glance-api-server} sudo tee /etc/ceph/ceph.client.glance.keyring</div><div class="line">ssh {your-glance-api-server} sudo chown glance:glance /etc/ceph/ceph.client.glance.keyring</div><div class="line">ceph auth get-or-create client.cinder | ssh {your-volume-server} sudo tee /etc/ceph/ceph.client.cinder.keyring</div><div class="line">ssh {your-cinder-volume-server} sudo chown cinder:cinder /etc/ceph/ceph.client.cinder.keyring</div><div class="line">ceph auth get-or-create client.cinder-backup | ssh {your-cinder-backup-server} sudo tee /etc/ceph/ceph.client.cinder-backup.keyring</div><div class="line">ssh {your-cinder-backup-server} sudo chown cinder:cinder /etc/ceph/ceph.client.cinder-backup.keyring</div></pre></td></tr></table></figure>
</li>
<li><p>运行nova-compute进程的节点也需要client.cinder的keyring文件：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">ceph auth get-or-create client.cinder | ssh {your-nova-compute-server} sudo tee /etc/ceph/ceph.client.cinder.keyring</div></pre></td></tr></table></figure>
</li>
<li><p><strong>计算节点</strong>上，还需要把client.cinder的keyring文件保存到libvirt中。libvirt进程从Cinder挂载一个块设备时，需要访问jCeph集群。</p>
<ul>
<li>创建一个临时文件 <code>ceph auth get-key client.cinder | ssh {your-compute-node} tee client.cinder.key</code></li>
<li><p>将秘钥加到libvirt中，并删除临时文件</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div></pre></td><td class="code"><pre><div class="line">uuidgen</div><div class="line">457eb676-33da-42ec-9a8c-9293d545c337</div><div class="line"></div><div class="line">cat > secret.xml <<EOF</div><div class="line"><secret ephemeral='no' private='no'></div><div class="line"> <uuid>457eb676-33da-42ec-9a8c-9293d545c337</uuid></div><div class="line"> <usage type='ceph'></div><div class="line"> <name>client.cinder secret</name></div><div class="line"> </usage></div><div class="line"></secret></div><div class="line">EOF</div><div class="line">sudo virsh secret-define --file secret.xml</div><div class="line">Secret 457eb676-33da-42ec-9a8c-9293d545c337 created</div><div class="line">sudo virsh secret-set-value --secret 457eb676-33da-42ec-9a8c-9293d545c337 --base64 $(cat client.cinder.key) && rm client.cinder.key secret.xml</div></pre></td></tr></table></figure>
</li>
<li><p>查看libvirt保存的秘钥的uuid的方法：<code>virsh secret-list</code></p>
</li>
<li>查看secret-value: <code>virsh secret-get-value UUID</code></li>
</ul>
</li>
</ul>
<h3 id="配置OpenStack来使用Ceph【改配置文件之前最好备份】"><a href="#配置OpenStack来使用Ceph【改配置文件之前最好备份】" class="headerlink" title="配置OpenStack来使用Ceph【改配置文件之前最好备份】"></a>配置OpenStack来使用Ceph【改配置文件之前最好备份】</h3><h4 id="配置Glance"><a href="#配置Glance" class="headerlink" title="配置Glance"></a>配置Glance</h4><ul>
<li>编辑<code>/etc/glance/glance-api.conf</code> :<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div></pre></td><td class="code"><pre><div class="line">[DEFAULT]</div><div class="line">...</div><div class="line">show_image_direct_url = True # 启用images的copy-on-write克隆</div><div class="line">...</div><div class="line">[glance_store]</div><div class="line">stores = rbd</div><div class="line">default_store = rbd</div><div class="line">rbd_store_pool = images</div><div class="line">rbd_store_user = glance</div><div class="line">rbd_store_ceph_conf = /etc/ceph/ceph.conf</div><div class="line">rbd_store_chunk_size = 8</div><div class="line">...</div><div class="line">[paste_deploy]</div><div class="line">flavor = keystone # 不缓存镜像</div><div class="line">...</div></pre></td></tr></table></figure>
</li>
</ul>
<h4 id="配置Cinder"><a href="#配置Cinder" class="headerlink" title="配置Cinder"></a>配置Cinder</h4><ul>
<li><p>编辑<code>/etc/cinder/cinder.conf</code></p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div></pre></td><td class="code"><pre><div class="line">[DEFAULT]</div><div class="line">...</div><div class="line">enabled_backends = ceph</div><div class="line">...</div><div class="line">[ceph]</div><div class="line">volume_driver = cinder.volume.drivers.rbd.RBDDriver</div><div class="line">volume_backend_name = ceph</div><div class="line">rbd_pool = volumes</div><div class="line">rbd_ceph_conf = /etc/ceph/ceph.conf</div><div class="line">rbd_flatten_volume_from_snapshot = false</div><div class="line">rbd_max_clone_depth = 5</div><div class="line">rbd_store_chunk_size = 4</div><div class="line">rados_connect_timeout = -1</div><div class="line">glance_api_version = 2</div></pre></td></tr></table></figure>
</li>
<li><p>如果开启了cephx认证，还需要libvirt中保存的秘钥的user和uuid</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line">[ceph]</div><div class="line">...</div><div class="line">rbd_user = cinder</div><div class="line">rbd_secret_uuid = $UUID # 使用之前存在libvirt中的UUID</div></pre></td></tr></table></figure>
</li>
</ul>
<h4 id="配置Cinder-Backup"><a href="#配置Cinder-Backup" class="headerlink" title="配置Cinder Backup"></a>配置Cinder Backup</h4><ul>
<li>修改<code>/etc/cinder/cinder.conf</code><figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div></pre></td><td class="code"><pre><div class="line">backup_driver = cinder.backup.drivers.ceph</div><div class="line">backup_ceph_conf = /etc/ceph/ceph.conf</div><div class="line">backup_ceph_user = cinder-backup</div><div class="line">backup_ceph_chunk_size = 134217728</div><div class="line">backup_ceph_pool = backups</div><div class="line">backup_ceph_stripe_unit = 0</div><div class="line">backup_ceph_stripe_count = 0</div><div class="line">restore_discard_excess_bytes = true</div></pre></td></tr></table></figure>
</li>
</ul>
<h4 id="配置Nova"><a href="#配置Nova" class="headerlink" title="配置Nova"></a>配置Nova</h4><ul>
<li>修改<code>/etc/nova/nova.conf</code><figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div></pre></td><td class="code"><pre><div class="line">[libvirt]</div><div class="line">virt_type = qemu</div><div class="line">images_type = rbd</div><div class="line">images_rbd_pool = vms</div><div class="line">images_rbd_ceph_conf = /etc/ceph/ceph.conf</div><div class="line">rbd_user = cinder</div><div class="line">rbd_secret_uuid = $UUID</div><div class="line">disk_cachemodes="network=writeback"</div><div class="line">live_migration_flag="VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE,VIR_MIGRATE_PERSIST_DEST,VIR_MIGRATE_TUNNELLED"</div></pre></td></tr></table></figure>
</li>
</ul>
<h1 id="重启相关服务"><a href="#重启相关服务" class="headerlink" title="重启相关服务"></a>重启相关服务</h1><h2 id="需要重启的服务"><a href="#需要重启的服务" class="headerlink" title="需要重启的服务"></a>需要重启的服务</h2><ul>
<li>glance-api (g-api)</li>
<li>glance-registry (g-reg)</li>
<li>nova-compute (n-cpu)</li>
<li>cinder-volume (<code>c-vol*</code>)</li>
<li>cinder-api (c-api)</li>
<li>cinder-schduler (c-sch)</li>
</ul>
<h2 id="重启方法"><a href="#重启方法" class="headerlink" title="重启方法"></a>重启方法</h2><p>由于devstack启动服务的方式比较特殊，安装的所有服务都运行在screen中，要重启服务，需要找到对应的screen，在其中进行重启操作。参考<a href="http://1.chaoxu.sinaapp.com/archives/3311" target="_blank" rel="external">这篇博客</a>。</p>
<ul>
<li><p>输入<code>screen -list</code>，查看当前用户开启的所有screen</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line">[stack@devstack devstack]$ screen -list</div><div class="line">There is a screen on:</div><div class="line"> 2756.stack (Detached)</div><div class="line">1 Socket in /var/run/screen/S-stack.</div></pre></td></tr></table></figure>
</li>
<li><p>附加到screen：<code>screen -r $screen_id # 上面的2756</code></p>
</li>
<li>依次执行<code>control + a</code> 和 <code>control + d</code> 来退出当前的screen</li>
<li>进入某个screen的方法，<code>control + a + '</code>，然后输入屏幕下方的数字，敲回车，切换到相应的screen。</li>
<li>停止服务<code>control + c</code>，重新启动<code>↑ + enter</code></li>
</ul>
<h1 id="测试验证"><a href="#测试验证" class="headerlink" title="测试验证"></a>测试验证</h1><ul>
<li>用openstack命令创建镜像：<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">[root@devstack devstack]# openstack image create "cirros-raw" --file files/cirros-0.3.4-x86_64-disk.raw --disk-format qcow2 --container-format bare --public</div></pre></td></tr></table></figure>
</li>
</ul>
<p>成功后可以用<code>rbd ls images</code>查看到</p>
<ul>
<li>在控制台创建新卷，成功后可以用<code>rbd ls volumes</code> 查到到</li>
<li>在控制台用新上传的镜像启动云主机，成功后可以用<code>rbs ls vms</code>查看到。【此处不知为何没有生效，新建的云主机镜像文件保存到了volumes pool中】</li>
</ul>
<h1 id="踩坑记录"><a href="#踩坑记录" class="headerlink" title="踩坑记录"></a>踩坑记录</h1><h3 id="控制台创建卷时，卷类型没有CEPH可选，显示无类型"><a href="#控制台创建卷时，卷类型没有CEPH可选，显示无类型" class="headerlink" title="控制台创建卷时，卷类型没有CEPH可选，显示无类型"></a>控制台创建卷时，卷类型没有CEPH可选，显示无类型</h3><ul>
<li>需要在cinder节点上添加：<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">cinder type-list # 只能看到一个lvmdirver-1</div><div class="line">cinder type-create --is-public True ceph</div></pre></td></tr></table></figure>
</li>
</ul>
<p>此时再到控制台创建，就可以看到有CEPH类型可选了。</p>
<h3 id="无法创建VM，报错Block-Device-Mapping-is-Invalid"><a href="#无法创建VM，报错Block-Device-Mapping-is-Invalid" class="headerlink" title="无法创建VM，报错Block Device Mapping is Invalid"></a>无法创建VM，报错<code>Block Device Mapping is Invalid</code></h3><ul>
<li>问题在于用的启动镜像，还是用之前的file镜像，跟ceph配置冲突。使用新上传到glance中的镜像进行启动就可以了</li>
</ul>
<h3 id="创建云主机时报错：No-valid-host-available"><a href="#创建云主机时报错：No-valid-host-available" class="headerlink" title="创建云主机时报错：No valid host available"></a>创建云主机时报错：No valid host available</h3><ul>
<li>首先用<code>free</code>等命令确认物理机资源足够flavor使用</li>
<li>最后发现，问题出在nova.conf配置文件中的virt_type项，参考博客设置的qemu，改回kvm就好了</li>
<li>关于这个配置项，在OpenStack官方文档找到如下的解释：<blockquote>
<p>Describes the virtualization type (or so called domain type) libvirt should use.<br>The choice of this type must match the underlying virtualization strategy you have chosen for this host.</p>
</blockquote>
</li>
</ul>
<p>用来指定libvirt使用的虚拟化类型，必须和宿主机上选择的虚拟化策略一致。</p>
<h1 id="参考"><a href="#参考" class="headerlink" title="参考"></a>参考</h1><ul>
<li>Ceph 官方文档 <a href="http://docs.ceph.com/docs/master/rbd/rbd-openstack/" target="_blank" rel="external">BLOCK DEVICES AND OPENSTACK</a>。文档中是按多节点部署的环境说明的，我部署的环境是All-In-One的单节点环境，所有服务的用户都是<code>stack.stack</code>，在相应的命令中替换掉即可。</li>
<li><a href="https://docs.openstack.org/developer/devstack/" target="_blank" rel="external">部署DevStack文档</a></li>
</ul>
]]></content>
<summary type="html">
<h1 id="环境"><a href="#环境" class="headerlink" title="环境"></a>环境</h1><ul>
<li>已经搭建好的 <code>Mon*1 + OSD*3</code> 的Ceph-10.2.5测试集群</li>
<li>All-In-One的Newton版DevStack环境，参考<a href="https://docs.openstack.org/developer/devstack/">DevStack部署文档</a>
</summary>
<category term="Ceph" scheme="https://caitianxiong.com/tags/Ceph/"/>
<category term="OpenStack" scheme="https://caitianxiong.com/tags/OpenStack/"/>
<category term="DevStack" scheme="https://caitianxiong.com/tags/DevStack/"/>
</entry>
<entry>
<title>手动部署Ceph集群</title>
<link href="https://caitianxiong.com/2017/02/25/deploy-ceph-cluster-manually/"/>
<id>https://caitianxiong.com/2017/02/25/deploy-ceph-cluster-manually/</id>
<published>2017-02-25T11:16:30.000Z</published>
<updated>2019-11-07T03:27:52.691Z</updated>
<content type="html"><![CDATA[<h2 id="环境"><a href="#环境" class="headerlink" title="环境"></a>环境</h2><figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line">[root@node1 ~]# uname -r</div><div class="line">3.10.0-327.28.3.el7.x86_64</div><div class="line">[root@node1 ~]# cat /etc/redhat-release</div><div class="line">CentOS Linux release 7.3.1611 (Core)</div></pre></td></tr></table></figure>
<a id="more"></a>
<h2 id="安装Ceph-10-2-5"><a href="#安装Ceph-10-2-5" class="headerlink" title="安装Ceph-10.2.5"></a>安装Ceph-10.2.5</h2><ul>
<li>添加KEYS <code>sudo rpm --import 'https://download.ceph.com/keys/release.asc'</code></li>
<li><p>添加yum源<code>ceph.repo</code>，其中<code>jewel/el7</code></p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div></pre></td><td class="code"><pre><div class="line">[ceph]</div><div class="line">name=Ceph packages for $basearch</div><div class="line">baseurl=https://download.ceph.com/rpm-{ceph-release}/{distro}/$basearch</div><div class="line">enabled=1</div><div class="line">priority=2</div><div class="line">gpgcheck=1</div><div class="line">gpgkey=https://download.ceph.com/keys/release.asc</div><div class="line"></div><div class="line">[ceph-noarch]</div><div class="line">name=Ceph noarch packages</div><div class="line">baseurl=https://download.ceph.com/rpm-{ceph-release}/{distro}/noarch</div><div class="line">enabled=1</div><div class="line">priority=2</div><div class="line">gpgcheck=1</div><div class="line">gpgkey=https://download.ceph.com/keys/release.asc</div><div class="line"></div><div class="line">[ceph-source]</div><div class="line">name=Ceph source packages</div><div class="line">baseurl=https://download.ceph.com/rpm-{ceph-release}/{distro}/SRPMS</div><div class="line">enabled=0</div><div class="line">priority=2</div><div class="line">gpgcheck=1</div><div class="line">gpgkey=https://download.ceph.com/keys/release.asc</div></pre></td></tr></table></figure>
</li>
<li><p>安装依赖 <code>sudo yum install snappy leveldb gdisk python-argparse gperftools-libs -y</code>【其中的leveldb需要先安装epel的yum源，<code>yum install epel-release -y</code>】</p>
</li>
<li>安装ceph，<code>yum install ceph -y</code></li>
</ul>
<h2 id="安装块设备虚拟化"><a href="#安装块设备虚拟化" class="headerlink" title="安装块设备虚拟化"></a>安装块设备虚拟化</h2><ul>
<li>QEMU <code>yum update && yum install qemu-kvm qemu-kvm-tools qemu-im</code></li>
<li>libvirt <code>yum install libvirt -y</code></li>
</ul>
<h2 id="部署ceph集群"><a href="#部署ceph集群" class="headerlink" title="部署ceph集群"></a>部署ceph集群</h2><p>参考官方文档<a href="http://docs.ceph.com/docs/master/install/manual-deployment/" target="_blank" rel="external">手动部署</a>，其中的启动命令过时了，应该用<code>systemctl</code>。具体步骤如下：</p>
<h3 id="MON"><a href="#MON" class="headerlink" title="MON"></a>MON</h3><ol>
<li>生成uuid</li>
<li><p>创建<code>ceph.conf</code>文件。<code>uuidgen</code> 生成uuid，写到<code>/etc/ceph/ceph.conf</code> 中作为<code>fsid</code>，同时写入mon信息</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">fsid = a7f64266-0894-4f1e-a635-d0aeaca0e993</div><div class="line">mon initial members = node1</div><div class="line">mon host = 192.168.1.43</div></pre></td></tr></table></figure>
</li>
<li><p>创建keyring文件，并生成一个 monitor 秘钥</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">ceph-authtool --create-keyring /tmp/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *'</div></pre></td></tr></table></figure>
</li>
<li><p>生成administrator keyring，生成client.admin用户，并将用户添加到keyring</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --set-uid=0 --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow'</div></pre></td></tr></table></figure>
</li>
<li><p>添加<code>client.admin</code> key到<code>ceph.mon.keyring</code></p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">ceph-authtool /tmp/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring</div></pre></td></tr></table></figure>
</li>
<li><p>使用hostname、IP、FSID生成一个monitor map，保存到<code>/tmp/monmap</code></p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">monmaptool --create --add {hostname} {ip-address} --fsid {uuid} /tmp/monmap</div></pre></td></tr></table></figure>
</li>
<li><p>在monitor主机上创建默认的数据目录，并修改属主为ceph</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">mkdir /var/lib/ceph/mon/{cluster-name}-{hostname}</div><div class="line">chown -R ceph.ceph /var/lib/ceph/mon</div></pre></td></tr></table></figure>
</li>
<li><p>将monitor map和keyring<strong>保存到（populate）</strong>monitor守护进程中</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">sudo -u ceph ceph-mon [--cluster {cluster-name}] --mkfs -i {hostname} --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring</div></pre></td></tr></table></figure>
</li>
<li><p>检查ceph.conf配置文件，根据需求，添加必要项</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div></pre></td><td class="code"><pre><div class="line">[global]</div><div class="line">fsid = {cluster-id}</div><div class="line">mon initial members = {hostname}[, {hostname}]</div><div class="line">mon host = {ip-address}[, {ip-address}]</div><div class="line">public network = {network}[, {network}]</div><div class="line">cluster network = {network}[, {network}]</div><div class="line">auth cluster required = cephx</div><div class="line">auth service required = cephx</div><div class="line">auth client required = cephx</div><div class="line">osd journal size = {n}</div><div class="line">osd pool default size = {n} # Write an object n times.</div><div class="line">osd pool default min size = {n} # Allow writing n copy in a degraded state.</div><div class="line">osd pool default pg num = {n}</div><div class="line">osd pool default pgp num = {n}</div><div class="line">osd crush chooseleaf type = {n}</div></pre></td></tr></table></figure>