Skip to content

Commit

Permalink
receive: Improved efficiency of multitsdb appends.
Browse files Browse the repository at this point in the history
Release vs current main looks the same:

```
benchstat -delta-test=none ../_dev/thanos/2021/receive/5.txt ../_dev/thanos/2021/receive2/main-go1.15.txt
name                                                                                old time/op    new time/op    delta
HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/OK-12                        1.56ms ± 0%    1.45ms ± 0%  -7.12%
HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/conflict_errors-12           6.49ms ± 0%    7.14ms ± 0%  +9.92%
HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/OK-12                       16.0ms ± 0%    16.4ms ± 0%  +2.79%
HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/conflict_errors-12          71.7ms ± 0%    69.4ms ± 0%  -3.20%
HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK-12                  138ms ± 0%     131ms ± 0%  -4.79%
HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors-12     1.58s ± 0%     1.68s ± 0%  +6.11%

name                                                                                old alloc/op   new alloc/op   delta
HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/OK-12                        1.70MB ± 0%    1.70MB ± 0%  +0.12%
HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/conflict_errors-12           4.84MB ± 0%    4.84MB ± 0%  +0.04%
HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/OK-12                       18.3MB ± 0%    18.2MB ± 0%  -0.19%
HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/conflict_errors-12          49.6MB ± 0%    49.6MB ± 0%  +0.00%
HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK-12                  331MB ± 0%     331MB ± 0%  -0.00%
HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors-12     804MB ± 0%     804MB ± 0%  +0.00%

name                                                                                old allocs/op  new allocs/op  delta
HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/OK-12                         15.6k ± 0%     15.6k ± 0%  +0.04%
HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/conflict_errors-12            35.6k ± 0%     35.6k ± 0%  +0.01%
HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/OK-12                         155k ± 0%      155k ± 0%  -0.08%
HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/conflict_errors-12            355k ± 0%      355k ± 0%  +0.00%
HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK-12                    147 ± 0%       145 ± 0%  -1.36%
HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors-12       417 ± 0%       421 ± 0%  +0.96%
```

Unfortunately go1.16 introduces more allocs overall (not that much more):

```
benchstat -delta-test=none ../_dev/thanos/2021/receive2/main-go1.15.txt ../_dev/thanos/2021/receive2/main-go1.16.3.txt
name                                                                                old time/op    new time/op    delta
HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/OK-12                        1.45ms ± 0%    1.62ms ± 0%  +11.87%
HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/conflict_errors-12           7.14ms ± 0%    6.47ms ± 0%   -9.40%
HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/OK-12                       16.4ms ± 0%    15.8ms ± 0%   -3.87%
HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/conflict_errors-12          69.4ms ± 0%    66.4ms ± 0%   -4.35%
HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK-12                  131ms ± 0%     141ms ± 0%   +7.59%
HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors-12     1.68s ± 0%     1.67s ± 0%   -0.49%

name                                                                                old alloc/op   new alloc/op   delta
HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/OK-12                        1.70MB ± 0%    1.75MB ± 0%   +2.50%
HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/conflict_errors-12           4.84MB ± 0%    4.89MB ± 0%   +0.88%
HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/OK-12                       18.2MB ± 0%    18.8MB ± 0%   +3.07%
HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/conflict_errors-12          49.6MB ± 0%    50.1MB ± 0%   +1.09%
HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK-12                  331MB ± 0%     343MB ± 0%   +3.63%
HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors-12     804MB ± 0%     816MB ± 0%   +1.50%

name                                                                                old allocs/op  new allocs/op  delta
HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/OK-12                         15.6k ± 0%     15.6k ± 0%   -0.01%
HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/conflict_errors-12            35.6k ± 0%     35.6k ± 0%   +0.01%
HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/OK-12                         155k ± 0%      155k ± 0%   +0.08%
HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/conflict_errors-12            355k ± 0%      355k ± 0%   +0.00%
HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK-12                    145 ± 0%       166 ± 0%  +14.48%
HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors-12       421 ± 0%       440 ± 0%   +4.51%
```

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
  • Loading branch information
bwplotka committed Apr 19, 2021
1 parent 2e212f4 commit 3cb75ec
Showing 1 changed file with 6 additions and 3 deletions.
9 changes: 6 additions & 3 deletions pkg/receive/handler_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -1166,15 +1166,18 @@ func benchmarkHandlerMultiTSDBReceiveRemoteWrite(b testutil.TB) {
for i := 0; i < n; i++ {
r := httptest.NewRecorder()
handler.receiveHTTP(r, &http.Request{ContentLength: int64(len(tcase.writeRequest)), Body: ioutil.NopCloser(bytes.NewReader(tcase.writeRequest))})
testutil.Equals(b, http.StatusConflict, r.Code, "%v", i)
testutil.Equals(b, http.StatusConflict, r.Code, "%v-%s", i, func() string {
b, _ := ioutil.ReadAll(r.Body)
return string(b)
}())
}
})
})
}

runtime.GC()
// Take snapshot at the end to reveal how much memory we keep in TSDB.
testutil.Ok(b, Heap("../../"))
testutil.Ok(b, Heap("../../../_dev/thanos/2021/receive2"))

}

Expand All @@ -1183,7 +1186,7 @@ func Heap(dir string) (err error) {
return err
}

f, err := os.Create(filepath.Join(dir, "mem.pprof"))
f, err := os.Create(filepath.Join(dir, "main-go16.pprof"))
if err != nil {
return err
}
Expand Down

0 comments on commit 3cb75ec

Please sign in to comment.