Add some benchmarks

[Hoverbear]

This PR adds benchmarks that:

• Use proptest to generate mildly random values for use.
• Do microbenchmarking with criterion.
• Report (with GNUplot) those results.

I'll be adding proptest related tests and more benchmarks as we go, but this is a good scaffold.

Note: I also had to renormalize some line endings.

Some sample results:

test result: ok. 0 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out

     Running target/release/deps/bench-441a949aeb506fc6
raw::put                time:   [2.3470 ms 2.3924 ms 2.4449 ms]
                        change: [-11.100% +1.8898% +17.550%] (p = 0.81 > 0.05)
                        No change in performance detected.
Found 8 outliers among 100 measurements (8.00%)
  3 (3.00%) high mild
  5 (5.00%) high severe

raw::get                time:   [1.3006 ms 1.3138 ms 1.3275 ms]
                        change: [+0.9404% +2.3687% +3.9148%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) high mild

raw::delete             time:   [2.3424 ms 2.3939 ms 2.4549 ms]
                        change: [-1.8414% +1.9432% +5.5014%] (p = 0.30 > 0.05)
                        No change in performance detected.
Found 9 outliers among 100 measurements (9.00%)
  1 (1.00%) low mild
  3 (3.00%) high mild
  5 (5.00%) high severe

raw::batch_put          time:   [3.3776 ms 3.4550 ms 3.5358 ms]
                        change: [-2.3549% +0.9699% +4.7708%] (p = 0.60 > 0.05)
                        No change in performance detected.
Found 10 outliers among 100 measurements (10.00%)
  7 (7.00%) high mild
  3 (3.00%) high severe

raw::batch_get          time:   [1.8608 ms 1.8804 ms 1.9015 ms]
                        change: [-1.5889% -0.0198% +1.5294%] (p = 0.98 > 0.05)
                        No change in performance detected.
Found 6 outliers among 100 measurements (6.00%)
  3 (3.00%) high mild
  3 (3.00%) high severe

raw::batch_delete       time:   [3.1134 ms 3.1646 ms 3.2197 ms]
                        change: [+0.9958% +3.9570% +6.7947%] (p = 0.01 < 0.05)
                        Change within noise threshold.
Found 4 outliers among 100 measurements (4.00%)
  1 (1.00%) low mild
  2 (2.00%) high mild
  1 (1.00%) high severe

[nrc]

Looks good. How reproducible are the results using the randomisation? I'd prefer to use the same input every time we run a benchmark, but we can use random-ish input if there is not too much noise in the results.

[nrc]

I'd also prefer not to benchmark the roundtrip to the server, and just benchmark the client side of things, but I don't think we can do that yet (or not easily). No harm in having both too.

[Hoverbear]

@nrc Yes! non-integration benchmarks are coming, but I wanted to do some full round-trip tests now. :)

The benchmarks are fairly normal. I think you're right that the same sample each run would give us more predictable performance, but we can't really say that performance each run in these kinds of tests will be consistent anyways (there is way too much going on for a slight change in buffer size/values to matter much, and these aren't necessarily clean-slate or configuration-enforced tests, tikv can be configured however).

This also keeps us honest and prevents us from optimizing on specific values.

That said, I have no strong opinions.

[Hoverbear]

Maybe I found an ICE?

[nrc]

I can repro with today's nightly, but not with my previous one, so it's a new bug in the last week or so.

[nrc]

Looks like rust-lang/rust#61442. It should be fixed by rust-lang/rust#61572 so I think we are better to wait a few days for that to land, rather than work around it.

[jlerche]

I think it would be useful to add to the readme how to run the benchmarks. Here's a skeleton blurb how I might structure it:

---

Running benchmarks

Some of the benchmarks require a connection to a PD and TikV instance. To run them, the addresses of the PD instances must be passed in as environment variable

PD_ADDRS=127.0.0.1:2379 cargo +nightly bench

where PD_ADDRS can also be a comma separated list of addresses to PD instances.

It is possible to specify running specific benchmarks, for example

PD_ADDRS=127.0.0.1:2379 cargo +nightly bench --features integration-tests

[Hoverbear]

@jlerche Great catch, added that!

[Hoverbear]

I'll base this on #68 since it's a better strategy.

[nrc]

I'll base this on #68 since it's a better strategy.

ping @Hoverbear are you still planning to rebase this PR?

[Hoverbear]

@nrc Yes I'd like to rebase this but I need to get fixtures working first! I think the best way is to use a pre-seeded random this way we don't have to ship data.

[Hoverbear]

Nick and I discussed this and we'd like to try another method.