master cluster_L File: master File: master Type: cpu Time: Sep 3, 2018 at 7:41pm (CST) Duration: 48s, Total samples = 466.90s (972.70%) Showing nodes accounting for 433.98s, 92.95% of 466.90s total Dropped 609 nodes (cum <= 2.33s) Dropped 62 edges (freq <= 0.47s) Showing top 80 nodes out of 113 N1 runtime systemstack 0.72s (0.15%) of 167.01s (35.77%) N48 runtime gcDrain 0.63s (0.13%) of 97.44s (20.87%) N1->N48 97.44s N80 runtime wbBufFlush func1 0.09s (0.019%) of 45.58s (9.76%) N1->N80 45.58s N76 runtime gcDrainN 0.21s (0.045%) of 13.91s (2.98%) N1->N76 13.91s N49 runtime (*mcentral) cacheSpan 0.23s (0.049%) of 6.81s (1.46%) N1->N49 6.81s N2 github com/pingcap/tidb/executor baseHashAggWorker getPartialResult 3.19s (0.68%) of 186.18s (39.88%) N10 runtime mapassign_faststr 9.35s (2.00%) of 98.82s (21.17%) N2->N10 98.80s N12 runtime mapaccess2_faststr 46.32s (9.92%) of 64.48s (13.81%) N2->N12 64.46s N30 runtime makeslice 0.93s (0.2%) of 10.55s (2.26%) N2->N30 6.60s N77 github com/pingcap/tidb/executor/aggfuncs (*baseAvgFloat64) AllocPartialResult 0.24s (0.051%) of 6.09s (1.30%) N2->N77 6.09s N55 runtime slicebytetostring 0.61s (0.13%) of 4.23s (0.91%) N2->N55 4.22s N13 runtime gcWriteBarrier 3.76s (0.81%) of 31.30s (6.70%) N2->N13 2.31s N3 github com/pingcap/tidb/executor (*HashAggPartialWorker) run 0.04s (0.0086%) of 219.89s (47.10%) N5 github com/pingcap/tidb/executor (*HashAggPartialWorker) updatePartialResult 11.08s (2.37%) of 198.54s (42.52%) N3->N5 198.54s N29 github com/pingcap/tidb/executor (*HashAggPartialWorker) shuffleIntermData 3.35s (0.72%) of 20.66s (4.42%) N3->N29 20.66s N4 runtime scanobject 37.43s (8.02%) of 110.18s (23.60%) N17 runtime greyobject 38.71s (8.29%) of 38.88s (8.33%) N4->N17 38.88s N9 runtime heapBitsForObject 54.31s (11.63%) N4->N9 33.87s N5->N2 151.25s N24 github com/pingcap/tidb/executor (*HashAggPartialWorker) getGroupKey 2.35s (0.5%) of 19.94s (4.27%) N5->N24 19.94s N27 runtime newobject 0.97s (0.21%) of 16.21s (3.47%) N5->N27 8.53s N41 github com/pingcap/tidb/executor/aggfuncs (*avgOriginal4Float64) UpdatePartialResult 2.77s (0.59%) of 5.79s (1.24%) N5->N41 5.79s N5->N13 1.05s N6 github com/pingcap/tidb/executor (*HashAggFinalWorker) consumeIntermData 4.98s (1.07%) of 114.45s (24.51%) N22 github com/pingcap/tidb/executor (*HashAggIntermData) getPartialResultBatch 2.40s (0.51%) of 38.86s (8.32%) N6->N22 38.86s N6->N2 33.69s N20 github com/pingcap/tidb/util/mvmap (*MVMap) Get 8.24s (1.76%) of 26.99s (5.78%) N6->N20 26.99s N39 github com/pingcap/tidb/util/mvmap (*MVMap) Put 0.47s (0.1%) of 6.15s (1.32%) N6->N39 6.15s N42 github com/pingcap/tidb/executor/aggfuncs (*avgPartial4Float64) MergePartialResult 3.63s (0.78%) N6->N42 3.63s N7 runtime evacuate_faststr 26.71s (5.72%) of 74.30s (15.91%) N19 runtime aeshashbody 20.68s (4.43%) N7->N19 16.66s N66 runtime memclrHasPointers 0.09s (0.019%) of 14s (3.00%) N7->N66 14s N25 runtime typedmemmove 1.07s (0.23%) of 15.43s (3.30%) N7->N25 10.55s N7->N13 5.52s N8 runtime mallocgc 13.35s (2.86%) of 43.28s (9.27%) N8->N1 22.37s N54 runtime gcmarknewobject 2.07s (0.44%) of 3.56s (0.76%) N8->N54 3.56s N46 runtime heapBitsSetType 3.32s (0.71%) N8->N46 3.32s N72 runtime growWork_faststr 0.23s (0.049%) of 74.53s (15.96%) N10->N72 74.53s N45 runtime hashGrow 0 of 12.55s (2.69%) N10->N45 11.03s N53 runtime (*hmap) newoverflow 0.62s (0.13%) of 2.36s (0.51%) N10->N53 1.90s N10->N13 1.07s N10->N19 0.92s N11 github com/pingcap/tidb/executor (*HashAggFinalWorker) run 0 of 116.20s (24.89%) N11->N6 114.45s N15 runtime memeqbody 28.61s (6.13%) N12->N15 15.74s N12->N19 1.95s N23 runtime wbBufFlush 0.16s (0.034%) of 45.74s (9.80%) N13->N23 27.54s N14 runtime gcBgMarkWorker 0 of 97.45s (20.87%) N14->N1 97.45s N16 runtime memmove 18.29s (3.92%) N18 runtime wbBufFlush1 24.54s (5.26%) of 45.49s (9.74%) N18->N9 20.41s N32 runtime mapaccess1_fast64 8.53s (1.83%) of 8.74s (1.87%) N20->N32 8.60s N67 github com/pingcap/tidb/util/mvmap (*dataStore) get 0.54s (0.12%) of 6.33s (1.36%) N20->N67 6.33s N20->N13 3.28s N21 runtime bulkBarrierPreWrite 6.75s (1.45%) of 24.95s (5.34%) N21->N23 18.20s N26 runtime mapaccess1_faststr 8.08s (1.73%) of 22.30s (4.78%) N22->N26 22.29s N22->N13 14.17s N23->N1 45.58s N59 github com/pingcap/tidb/util/codec EncodeValue 0.77s (0.16%) of 6.17s (1.32%) N24->N59 6.17s N58 github com/pingcap/tidb/expression (*Column) Eval 0.88s (0.19%) of 5.62s (1.20%) N24->N58 5.62s N24->N25 4.12s N24->N13 0.93s N25->N21 11.39s N25->N16 2.97s N26->N15 12.86s N26->N19 1.14s N27->N8 15.24s N28 github com/pingcap/tidb/util/codec (*Decoder) DecodeOne 2.05s (0.44%) of 13.37s (2.86%) N74 github com/pingcap/tidb/util/codec appendIntToChunk 0.44s (0.094%) of 3.55s (0.76%) N28->N74 3.55s N73 github com/pingcap/tidb/util/codec appendFloatToChunk 0.48s (0.1%) of 3.14s (0.67%) N28->N73 3.14s N62 github com/pingcap/tidb/util/codec DecodeVarint 0.61s (0.13%) of 2.73s (0.58%) N28->N62 2.73s N34 runtime stringtoslicebyte 0.29s (0.062%) of 13.33s (2.86%) N29->N34 13.33s N29->N16 0.82s N30->N8 9.62s N31 github com/pingcap/tidb/executor (*HashAggExec) fetchChildData 0.05s (0.011%) of 16.85s (3.61%) N50 github com/pingcap/tidb/distsql (*selectResult) Next 0.05s (0.011%) of 16.35s (3.50%) N31->N50 16.31s N33 runtime memclrNoHeapPointers 4.06s (0.87%) N34->N16 10s N60 runtime rawbyteslice 0.55s (0.12%) of 3.04s (0.65%) N34->N60 3.04s N35 runtime cmpbody 5.77s (1.24%) N36 runtime futex 2.37s (0.51%) N37 runtime (*mheap) alloc 0.11s (0.024%) of 5.02s (1.08%) N68 runtime (*mheap) alloc_m 0.23s (0.049%) of 2.54s (0.54%) N37->N68 2.54s N37->N33 2.35s N38 github com/pingcap/tidb/vendor/google golang org/grpc/transport (*http2Client) reader 0.03s (0.0064%) of 4.16s (0.89%) N64 github com/pingcap/tidb/vendor/google golang org/grpc/transport (*http2Client) handleData 0.02s (0.0043%) of 2.41s (0.52%) N38->N64 2.41s N70 io ReadFull 0 of 2.81s (0.6%) N38->N70 1.45s N44 runtime mapassign_fast64 0.44s (0.094%) of 4.44s (0.95%) N39->N44 4.43s N39->N13 0.54s N40 github com/pingcap/tidb/util/chunk Row GetDatum 4.22s (0.9%) of 4.39s (0.94%) N52 github com/pingcap/tidb/expression (*Column) EvalReal 3.02s (0.65%) N41->N52 3.02s N43 github com/pingcap/tidb/distsql (*selectResult) readRowsData 1.19s (0.25%) of 14.57s (3.12%) N43->N28 13.37s N44->N45 1.52s N45->N8 12.54s N47 github com/pingcap/tidb/util/codec encode 1.63s (0.35%) of 5.40s (1.16%) N61 github com/pingcap/tidb/util/codec encodeSignedInt 0.91s (0.19%) of 3.20s (0.69%) N47->N61 3.20s N48->N4 96.50s N78 runtime (*mcentral) grow 0.01s (0.0021%) of 4.56s (0.98%) N49->N78 4.56s N49->N36 0.92s N50->N43 14.57s N51 github com/pingcap/tidb/util/chunk (*Chunk) AppendInt64 1.55s (0.33%) of 3.11s (0.67%) N51->N13 1s N51->N16 0.56s N53->N27 1.42s N55->N8 3.07s N55->N16 0.55s N56 github com/pingcap/tidb/util/chunk (*Chunk) AppendFloat32 1.35s (0.29%) of 2.66s (0.57%) N56->N13 0.86s N57 github com/pingcap/tidb/util/goroutine_pool (*goroutine) workLoop 0 of 3.70s (0.79%) N63 github com/pingcap/tidb/vendor/google golang org/grpc (*parser) recvMsg 0 of 3.41s (0.73%) N57->N63 3.38s N58->N40 4.39s N59->N47 5.40s N60->N8 2.21s N63->N30 2.04s N63->N70 1.36s N64->N30 1.27s N64->N16 0.92s N65 github com/pingcap/tidb/server (*Server) onConn 0 of 3.28s (0.7%) N75 github com/pingcap/tidb/server (*clientConn) writeChunks 0.10s (0.021%) of 3.27s (0.7%) N65->N75 3.27s N66->N21 13.56s N67->N35 5.76s N69 runtime mcall 0.01s (0.0021%) of 2.39s (0.51%) N69->N36 0.63s N79 io ReadAtLeast 0.01s (0.0021%) of 2.81s (0.6%) N70->N79 2.81s N71 github com/pingcap/tidb/server dumpTextRow 0.13s (0.028%) of 2.54s (0.54%) N72->N7 74.30s N73->N56 2.66s N74->N51 3.11s N75->N71 2.54s N76->N4 13.68s N77->N27 5.85s N78->N37 4.09s N79->N16 1.25s N80->N18 45.49s