Add streaming proxy feature

[kraih]

This is an implementation of the streaming proxy described in #1295.

[kraih]

The core functionality seems pretty solid, and so do the new low level methods and attributes (naming could use some bikeshedding probably).

What i do not quite like yet is the API of the helper, but i can't think of a better alternative. I originally planned to use the returned promise for response rewriting (adding headers and so on). Unfortunately the promise would have to be resolved right in between the body and read events of the forwarded response, which is impossible because resolving a promise requires another reactor tick. So instead i had to use a callback passed as an argument to start_p, which looks rather silly.

The helper API might be a blocker for this feature.

[kraih]

This script illustrates the problem.

use Mojo::Base -strict;

use Mojo::EventEmitter;
use Mojo::IOLoop;
use Mojo::Promise;
use Mojo::Util 'dumper';

my $promise = Mojo::Promise->new;
my $events  = Mojo::EventEmitter->new;

# We want to use a promise to change the "foo" value
my $response = {foo => 'before'};
$promise->then(sub {
  $response->{foo} = 'after';
});

# The first event resolves the promise, and the second prints the "foo" value
$events->on(
  one => sub {
    $promise->resolve();
  }
);
$events->on(
  two => sub {
    say dumper $response;
  }
);

# Events happen in the right order, but the promise only resolves after the
# second event because it requires a reactor tick
$events->emit('one')->emit('two');
Mojo::IOLoop->one_tick;

[kraih]

Having mentioned the negative, on the plus side the streaming proxy has been up to 30 times faster than previously used proxy implementations for Mojolicious.

use Mojolicious::Lite;

my $UPSTREAM = 'http://127.0.0.1:3000';

# Old way (caching the whole response)
get '/proxy1' => sub {
  my $c = shift;

  $c->ua->get_p($UPSTREAM)->then(sub {
    my $tx = shift;
    $c->tx->res($tx->res);
    $c->rendered;
  });
};

# New way (streaming the response as chunks of data arrive)
get '/proxy2' => sub {
  my $c = shift;

  $c->proxy->get_p($UPSTREAM);
};

app->start;

And response size does not matter at all, it can even be an infinite EventSource stream. Throttling works automatically, so if one direction is faster than the other the proxy will not cache more than 1MB of data (this is configurable).

$ wrk -c 100 -d 10 http://127.0.0.1:8080/proxy2
Running 10s test @ http://127.0.0.1:8080/proxy2
  2 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    46.31ms   31.32ms 402.74ms   92.33%
    Req/Sec   309.83     67.12   434.00     78.50%
  6178 requests in 10.01s, 6.04GB read
  Socket errors: connect 0, read 24, write 0, timeout 0
Requests/sec:    617.04
Transfer/sec:    617.52MB

$ wrk -c 100 -d 10 http://127.0.0.1:8080/proxy1
Running 10s test @ http://127.0.0.1:8080/proxy1
  2 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.23s   258.57ms   1.96s    72.62%
    Req/Sec    22.03     25.85   141.00     85.07%
  200 requests in 10.03s, 204.65MB read
  Socket errors: connect 0, read 0, write 0, timeout 116
Requests/sec:     19.95
Transfer/sec:     20.41MB

This test was performed with 1MB responses from another Mojolicious application on a Linux machine with the page cache disabled, to simulate a worst case scenario.

[kraih]

I'm calling for a vote on this @mojolicious/core.

[kraih]

This PR also contains some small optimizations for streaming responses in general. Found those bottlenecks while benchmarking the proxy helper.

[jhthorsen]

I apologise in advance if my questions are idiotic. Please ignore me if that is the case.

I'm very much positive to the change in general and I more than welcome a simple proxy helper.

[kraih]

Ok, the problem with the API has been resolved, this should be fine now. 😀

[kraih]

The solution we settled on for the API problem was to just reuse the body event after the start_p call.

my $tx = $c->ua->build_tx(GET => 'http://mojolicious.org');
$c->proxy->start_p($tx);
$tx->res->content->once(body => sub {
  $c->res->headers->header('X-Proxy' => 'Mojo');
});

This gives us all the flexibility we need for rewriting HTTP responses.

[kraih]

Since there is a lot of potential for bugs, i suggest we keep the feature experimental for some time.

[jhthorsen]

I think the code makes sense and on a high level I'm giving the PR 👍

[jberger]

Looks good to me 👍 .

For posterity I want to leave a note that reminds us to document the timing (and values if any) of the resolution/rejection of the promise. For now, that can be deferred.