perf(gateway): speed up ratelimiter

[vilgotf]

This PR optimizes available and poll_available based on two observations:

1. The instant queue is ordered, so finding the partition point is enough for counting elapsed and unelapsed ones.
2. Instant::elapsed internally calls Instant::now (which might result in a syscall) so calling it over the queue is inefficient, a single instant should instead be reused.

To simplify the code, the time at which permits elapse is now stored instead of when in was acquired.

[zeylahellyer]

I'm curious what benchmark results would say here. I would also highly prefer the refactoring to be outside of the performance work, it's hard to keep track of what benefits performance and what's just refactoring, and we should probably keep the two separate for record purposes.

[vilgotf]

Thank you for reminding me to always benchmark! I'll get to it shortly. Note that it's sadly impossible to commit a benchmark due to the critical methods being private.

[vilgotf]

I don't know how to add the benchmark to Twilight in a good way. It requires public access to private functions, which I guarded behind a commented out by default "unstable" feature flag. It somewhat breaks RA and CI, and I don't know how to fix that. Anyway, here's the benchmark results:

Previously

command ratelimiter/empty
                        time:   [191.37 ns 212.46 ns 229.33 ns]
command ratelimiter/half elapsed
                        time:   [3.6624 µs 3.6838 µs 3.7117 µs]
Found 9 outliers among 100 measurements (9.00%)
  1 (1.00%) high mild
  8 (8.00%) high severe
command ratelimiter/half full
                        time:   [3.6387 µs 3.6788 µs 3.7245 µs]
Found 11 outliers among 100 measurements (11.00%)
  4 (4.00%) high mild
  7 (7.00%) high severe

With this PR

command ratelimiter/empty
                        time:   [200.13 ns 219.23 ns 234.52 ns]
                        change: [-14.968% +3.0799% +25.051%] (p = 0.76 > 0.05)
                        No change in performance detected.
command ratelimiter/half elapsed
                        time:   [56.646 ns 57.611 ns 58.745 ns]
                        change: [-98.472% -98.448% -98.422%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
  2 (2.00%) high mild
  7 (7.00%) high severe
command ratelimiter/half full
                        time:   [58.656 ns 59.632 ns 60.759 ns]
                        change: [-98.414% -98.388% -98.363%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 12 outliers among 100 measurements (12.00%)
  4 (4.00%) high mild
  8 (8.00%) high severe

[vilgotf]

To make CI pass and therefore show that this PR is ready, I pushed and immediately reverted the benchmark commit (so you can test run the benchmark yourself).

[vilgotf]

This should arguably return the instant at which the next permit is available. Users can get the Duration until then via elapsed.saturating_duration_since(Instant::now()).

[zeylahellyer]

Thanks for the benchmark, that's quite impressive. I'll see if I can take a look and understand what's going on this week.

[zeylahellyer]

Thank you for all the work you did on this. The benchmarks are quite impressive.