Support load balancing HTTP requests across worker threads

[mcollina]

I think we can support load balancing HTTP requests across worker threads by leveraging the same mechanisms that allows Cluster to do it. If a libuv handle can cross processes, it definitely can cross threads.

I've tried to do it outside of core in https://github.com/mcollina/tbal, but unfortunately it does not work because it requires some mangling of libuv handles that is only possible from within core.

The idea is to allow worker threads to have an libuv IPC channel similar to child processes, so that they can exchange libuv handles through it.

This should be opt-in.

[juanarbol]

Interesting. Could you please be a bit more specific with the implementation? Currently, moving sockets between workers is not possible.

[theanarkh]

Maybe we can achieve this by sharing fd 🤔(Or listen on the port in the main thread and distribute fd to child threads?).

server.js

const { Worker, isMainThread, parentPort, threadId } = require('worker_threads');
const os = require('os');
const net = require('net');
const http = require('http');

if (isMainThread) {
    const handle = net._createServerHandle('127.0.0.1', '9999')
    if (typeof handle === 'number') {
        console.error('exit', handle);
        process.exit();
    }
    const workers = [];
    for (let i = 0; i < os.cpus().length; i++) {
        new Worker(__filename).postMessage(handle.fd)
    }
} else {
    parentPort.on('message', (fd) => {
        console.log(`worker ${threadId}`)
        http.createServer((_, res) => {
            res.end(`${threadId}`)
        }).listen({fd});
    });
}

client.js

const http = require('http')
for (let i = 0; i < 10; i++) {
    http.get("http://127.0.0.1:9999", res => {
        let chunk
        res.on('data', (data) => {
            chunk = chunk ? Buffer.concat([chunk, data]) : data;
        });
        res.on('end', () => {
            console.log(chunk.toString())
        })
    })
}

[juanarbol]

The thing is. That handle you create on the main thread, is a socket (under the hood), and that socket/handle will be managed on your main event loop; you can't transfer that handle to another event loop.

[theanarkh]

All threads share the fd among process, so main thread just need to send the fd to other worker threads, then the worker threads create a socket based on fd.

[mcollina]

Sending the fd is possible, but we must tell the server on the main process that it should not respond to that fd, or doing anything with it.

Plus the tricky part is managing the "shutdown" of the server.

[flakey5]

If this is something y'all would be interested in pursuing I'd be willing to take a swing at it 👀

[mcollina]

cc @nodejs/workers

I think this would be great.

[gireeshpunathil]

IIRC, events that libuv manages have extended / unused attributes. one idea can be to use one of those attribute to attach the owner thread info. loops of all the threads will poll for the event, but only the one whose thread id matches with the thread info field will process the callback, the rest (including main thread) ignore the event.

[juanarbol]

I think I'm not considering something in the upfront.

https://github.com/libuv/libuv/blob/d2d92b74a8327daf9652a9732454937f3529bd30/src/unix/stream.c#L539

Every time uv_accept is called (at least in unix) the first operation made is check that the client's loop is associated with the server's loop.

Is this idea more like a reversed proxy with spawned workers in a IP range? And those workers will write into the reverse proxy socket? I'm very confused with this one.

Are the workers just responsable of creating the buffers and writing into the "main" process client sockets? If that is the case, I would consider this as a passable to achieve. IMHO.

[juanarbol]

it may be possible to pass like an option for "http handlers" to the workers, they will compute your request, but the streams will remain on front. Not quite sure if that is what you would like to have