[Performance] mrbench fails snapshotting on heavy load (n=3, p=1000)

[akiradeveloper]

With 3 nodes, it starts to slow down on heavy load. Since MakeSnapshot is a read request, I suspect something is bad in read path.

$ ./target/release/mrbench -t 3600s -w10 -r0 -n3 -p1000
[tests/mrbench/src/main.rs:31:5] &opts = Opts {
    num_nodes: 3,
    num_shards: 1000,
    io_duration: 3600s,
    n_batch_writes: 10,
    n_batch_reads: 0,
    io_size: 1,
    enable_console: false,
    no_compaction: false,
}
cluster setup: 2.490033669s
io done. 1.587973364s. error=(w=0,r=0)
io done. 1.900172713s. error=(w=0,r=0)
io done. 1.727882326s. error=(w=0,r=0)
io done. 1.757022015s. error=(w=0,r=0)
io done. 1.765393621s. error=(w=0,r=0)
io done. 1.689050918s. error=(w=0,r=0)
io done. 1.717950617s. error=(w=0,r=0)
io done. 1.746052428s. error=(w=0,r=0)
io done. 1.74737389s. error=(w=0,r=0)
io done. 1.550884065s. error=(w=0,r=0)
io done. 1.623529568s. error=(w=0,r=0)
io done. 1.512375294s. error=(w=0,r=0)
io done. 1.615274773s. error=(w=0,r=0)
io done. 1.539611146s. error=(w=0,r=0)
io done. 1.764735809s. error=(w=0,r=0)
io done. 1.972855071s. error=(w=0,r=0)
io done. 1.91604916s. error=(w=0,r=0)
io done. 1.78214814s. error=(w=0,r=0)
io done. 1.684460764s. error=(w=0,r=0)
io done. 1.616587642s. error=(w=0,r=0)
io done. 1.943483596s. error=(w=0,r=0)
io done. 2.035023898s. error=(w=0,r=0)
io done. 1.897965368s. error=(w=0,r=0)
io done. 1.91813767s. error=(w=0,r=0)
io done. 2.045074379s. error=(w=0,r=0)
io done. 2.186428005s. error=(w=0,r=0)
io done. 2.060105661s. error=(w=0,r=0)
io done. 2.017666159s. error=(w=0,r=0)
io done. 1.984182341s. error=(w=0,r=0)
io done. 2.123261071s. error=(w=0,r=0)
io done. 2.469992088s. error=(w=0,r=0)
io done. 2.174305499s. error=(w=0,r=0)
io done. 2.051967277s. error=(w=0,r=0)
io done. 2.149033763s. error=(w=0,r=0)
io done. 2.142540899s. error=(w=0,r=0)
io done. 2.547040554s. error=(w=0,r=0)
io done. 1.969417652s. error=(w=0,r=0)
io done. 2.452979924s. error=(w=0,r=0)
io done. 2.500046345s. error=(w=0,r=0)
io done. 2.019879979s. error=(w=0,r=0)
io done. 2.254520963s. error=(w=0,r=0)
io done. 2.540560112s. error=(w=0,r=0)
io done. 2.446165139s. error=(w=0,r=0)
io done. 2.272120114s. error=(w=0,r=0)
io done. 2.36014169s. error=(w=0,r=0)
io done. 2.743515291s. error=(w=0,r=0)
io done. 2.563707706s. error=(w=0,r=0)
io done. 2.742024055s. error=(w=0,r=0)
io done. 2.619447281s. error=(w=0,r=0)
io done. 2.966838171s. error=(w=0,r=0)
io done. 2.884810693s. error=(w=0,r=0)
io done. 3.163284127s. error=(w=0,r=0)
io done. 2.755947927s. error=(w=0,r=0)
failed to make snapshot: status: Cancelled, message: "h2 protocol error: http2 error", details: [], metadata: MetadataMap { headers: {} }

Caused by:
    0: transport error
    1: http2 error
    2: stream error received: stream no longer needed
io done. 3.201212141s. error=(w=0,r=0)
io done. 3.316089523s. error=(w=0,r=0)
io done. 3.885980661s. error=(w=0,r=0)
io done. 3.795237185s. error=(w=0,r=0)
failed to make snapshot: status: Cancelled, message: "h2 protocol error: http2 error", details: [], metadata: MetadataMap { headers: {} }

Caused by:
    0: transport error
    1: http2 error
    2: stream error received: stream no longer needed
io done. 13.669674329s. error=(w=0,r=0)
io done. 22.745909667s. error=(w=0,r=0)
failed to make snapshot: status: Cancelled, message: "h2 protocol error: http2 error", details: [], metadata: MetadataMap { headers: {} }

Caused by:
    0: transport error
    1: http2 error
    2: stream error received: stream no longer needed
io done. 67.428227436s. error=(w=519,r=0)
failed to make snapshot: status: Cancelled, message: "h2 protocol error: http2 error", details: [], metadata: MetadataMap { headers: {} }

Caused by:
    0: transport error
    1: http2 error
    2: stream error received: stream no longer needed