Socket Factory API

[Kilobyte22]

The other day i was working on a proxy library and i got an idea for an API that crystal would probably benefit from. The basic idea is to have a central class/mixin/other way to create something that can create a socket according to specifications. Example:

module SocketFactory
  abstract def create_socket(url: String): Socket # Possibly just an IO object?
  abstract def supported_protocols: Array(String)
end

This could be used as a parameter when instanciating objects related to networking. For example:

HTTP::Client.get("http://crystal-lang.org", socket_factory: Proxy::SocksFactory.new("localhost", "8888"))

socket_factory would be an optional parameter, defaulting to SocketFactory.default, which creates raw TCPSockets and supports only a single protocol: tcp. (in other words: only supports urls starting with tcp:)

I've had a suggestion to make this a single callback, but thats less expandable and things like getting the supported protocols would be tough to implement. If we find a way around that, that should be a viable option as well though.

SocketFactories which themselves do more than just creating a socket (like a TLS one) should use another SocketFactory for creating their socket. This allows for interesting combinations, like piping a TLS stream through a proxy or connecting to a proxy through a proxy.

[kumpelblase2]

Not too sure how an actual socket factory is needed, because you could just expect a block and create the socket in there, however, I'm not against the idea itself. Especially in the example you brought up, I am certainly in the need of such a mechanism (since there's no way to either inject or access the socket of a http client right now).

I don't really see a strong reason to have something like "supported protocols" for a socket factory though. You'd either end up with bloated socket factories supporting many protocols or only one and then it serves no purpose what so ever. As far as I'm concerned I'd only want a socket factory to create a socket, no questions asked. If you hand me a specific socket factory, I will only use the sockets created by it. But enlighten me :)

Either way, I'm in support of this 👍

[asterite]

Alternatively, you can subclass HTTP::Client and overwrite the socket method (which is private, but we could document it) to return a socket configured for your needs.

That way we can keep the default implementation, which will be 99% useful out of the box, clean.

[kumpelblase2]

Alternatively, you can subclass HTTP::Client and overwrite the socket method (which is private, but we could document it) to return a socket configured for your needs.

That might be true. However, I don't want to have to do this in order to use other things from the standard library. The exact use case I'm talking about would be to convert a http client socket to a web socket, as you start a websocket via an http request on the client side. If the HTTP::Client would accept a socket directly or a socket factory, that I could specifiy to use a specific socket, this would be possible.

So I would argue that a socket factory or similar would bring a lot of flexibility while keeping complexity to a minimum.

[valpackett]

This would be very useful for using TLS libraries other than OpenSSL!

[ysbaddaden]

I'm not fond of factory classes to pass around. This sounds like unnecessary overhead. I would prefer chainable calls to build a request with lazy evaluation (something like http.rb). For example:

response = HTTP::Client.proxy(host, port).get(uri)

[Kilobyte22]

The problem with that is that it is not remotely as flexible.

[valpackett]

Maybe something like HTTP::Client.get(socket, uri)?

[ysbaddaden]

HTTP::Client is a simple abstraction to quickly execute HTTP requests. Configuring a proxy is legitimate need, and maybe we should implement this in HTTP::Client.

Using another TLS than the default one... is a bit more far fetched —but I already thought about it. As suggested by @asterite we can document the private API and keep it stable (when crystal gets stable, that is). For example, there could be an ssl_socket that would take a socket and return a wrapped SSL socket.

I don't see other use cases. This is a HTTP Client, not a generic IO resource we're talking about. Maybe a UNIX socket? Bit that would be a very specific use case...

[Kilobyte22]

Please be aware this is not just about HTTP Clients, but for anything.
Examples of things that could use a SocketFactory, like SSH, IRC, IMAP - basicly anything that runs via tcp.

This would mean (given everyone properly uses this api) that you could just plug in the TLS implementation of author A into the IRC library of author B even if they don't even know about each other

[valpackett]

There's a lot of uses for custom sockets!

• TLS implementations (I want to use LibreSSL's new libtls!)
• Other encrypted transports (Noise Protocol!)
• UNIX sockets (more common than you think!)
• SOCKS, other TCP-level proxy protocols

All of this should be done through the same API, whether it's "plug a socket into a protocol implementation" or "plug a socket factory…" or both.

I think this looks good:

HTTP::Client.new(socket_factory:
  TLSSocketFactory.new(SocksProxyFactory.new(UnixSocket.new("/var/run/proxy"))))
# http client with tls support, proxying connections through
# a socks proxy, which is accessed through a unix socket

[valpackett]

This is how golang does this:

For control over proxies, TLS configuration, keep-alives, compression, and other settings, create a Transport:

tr := &http.Transport{
    TLSClientConfig:    &tls.Config{RootCAs: pool},
    DisableCompression: true,
}
client := &http.Client{Transport: tr}
resp, err := client.Get("https://example.com")

You can have Transports that call other Transports, etc.
Here's a UNIX socket transport I wrote. A transport that adds OAuth.

Similarly, on the server side there are composable Listeners. Here's a listener I wrote that intercepts SSH connections.

I think these names (Transport and Listener) are good, much better than SocketFactory.

[Kilobyte22]

I don't really care how it is named, i do care about how it works :)

I assume most people in here agree with that