Passing sockets over pipes fails on windows

[stakach]

I'm working on implementing an FFI implementation of libuv for ruby compatible with the v0.11.13 release of libuv: https://github.com/cotag/libuv

All of my tests run on unix without issue however I cannot pass sockets over pipes on Windows. I continually receive: EAGAIN: resource temporarily unavailable no matter how much I delay or retry the accept. Once again this works flawlessly on all unix variants so I don't think I am doing anything wrong.

Also regular sockets and regular pipes work on windows, just not having any luck with write2 or read2...

ping @piscisaureus this may be relevant, I noticed that you've been working on windows compatibility.
Also, and unrelated, having an issue using unlink on windows after creating and writing a file with 777 permissions.

[stakach]

Also @bnoordhuis you were working on the win write2 implementation recently. Any ideas?
I'm working through the libuv code atm

[saghul]

Does the libuv test suite pass for you? Last time I checked IPC worked fine on Windows.

[stakach]

I'm trying to pass sockets from one uv loop to another in the same process.

I looked at the test suite however was a little confused as to if it was passing sockets across different processes. Then if so, does this mean Windows doesn't support same process IPC only inter-process. As this would explain the issue.

[piscisaureus]

@stakach I don't think I actually ever tested passing sockets within the same process. However there is no theoretical reason it shouldn't work. However it could be that the socket isn't getting duplicated properly.

Just, guessing here, it's a long time ago: uv_spawn() sets some magic property on the uv_pipe_t handle so uv_write2 knows which process to duplicate the handle to. If you aren't spawning then this property might not set and WSADuplicateSocket could be failing.

Also note that passing uv_pipe_t handles over the IPC channel is currently not supported on windows.

[stakach]

Thanks for the pointer. I'll fiddle around with the libuv code and see if I can make it work then post any finding here / a pull request.

TCP connections are the only handles my tests are passing over IPC.

[stakach]

Ok, it seems: "Sockets can be shared among threads in a given process without using the WSADuplicateSocket function because a socket descriptor is valid in all threads of a process."

Source http://msdn.microsoft.com/en-us/library/windows/desktop/ms741565(v=vs.85).aspx
So a special case may need to be made when process ids match? Although not entirely sure how to go about this.. If it is possible.

[bnoordhuis]

@stakach What the MSDN documentation is saying is that you don't need to call WSADuplicateSocket(), not that you can't.

Can you test if test/benchmark-multi-accept.c works for you?

[piscisaureus]

@stakach

Ok, it seems: "Sockets can be shared among threads in a given process without using the WSADuplicateSocket function because a socket descriptor is valid in all threads of a process."

You need to duplicate in order not to mess up libuv's internal bookkeeping.

[grittygrease]

I'm having the same problems on Windows. Also, the tests named tcp_multi_accept* all fail in test/benchmark-multi-accept.c Is there any progress on a fix?

[jgrahamc]

I am working with @grittygrease above on this and we are seeing the following.

We have a TCP server written using libuv with threads that works perfectly on Linux and Mac OS X but fails on Windows. The TCP server binds to a port in the main program, spawns a number of threads and passes the bound socket to them using pipes. Our code is similar in operation to test/benchmark-multi-accept.c.

In the threads we uv_listen() on the now duplicated socket and that call is successful. We then uv_run() but never receive a callback when a connection is made and, therefore, never get to call uv_accept() etc.

[jgrahamc]

I have a suspicion that the duplication of the socket is failing when in the same process because pipe->ipc_pid is not set.

The code win/tcp.c does the following:

/* Use the IPC framing protocol. */
if (send_handle) {
  tcp_send_handle = (uv_tcp_t*)send_handle;

  err = uv_tcp_duplicate_socket(tcp_send_handle, handle->ipc_pid,
      &ipc_frame.socket_info);
  if (err) {
    return err;
  }

ipc_pid only gets set in uv_pipe_open or uv_spawn. Neither will happen if sending a uv_tcp_t via a pipe within the same process (but to a different thread). Thus handle->ipc_pid will be 0 and an error will occur.

[stakach]

I did some basic multiprocess socket passing on windows, to see if it would make a difference, and had no luck there either. The same code worked on OSX - however haven't had the time to dig into the internals of libuv

[indutny]

Will work on it

[indutny]

Basically, the idea is to send pids in both ways at the pipe initialization and to use them as ipc_pid value

[indutny]

That was note for myself ^

[jgrahamc]

Thanks. The other question I had about the source code is that when the uv_write2 happens the socket handle is duplicated (done in uv_pipe_write_impl via a call to uv_tcp_duplicate_socket). uv_duplicate_socket will do a listen() on the 'parent' socket.

Then uv_pipe_accept (in the subprocess/other thread) will call uv_tcp_import which does tcp->flags |= UV_HANDLE_SHARED_TCP_SOCKET; When I do a uv_listen on the socket in the subprocess/other thread this flag prevents a listen() from happening on the duplicated socket.

I'm unsure if that matters to the problem at hand but was something I spotted while reviewing the code.

[indutny]

After considering this, I found that sending pid over the IPC pipe is quite racey as users may start sending handles on that pipe before the PID message will arrive. I'll need to think about it for a bit...

[jgrahamc]

Is there any news on this?

[indutny]

Not really, I don't think that this is going to work on windows (i.e. sending handles inside the same process), I'll consider working on another problem somewhere later, but I'm not really a windows guy here :) cc @piscisaureus

[indutny]

Though, I'll review and land a PR if you'll send one ;)

[jgrahamc]

How far did you get? I'm interested in picking this up and seeing if we can patch it ourselves.

[indutny]

@jgrahamc I got to this https://github.com/indutny/libuv/compare/fix;gh-926 . I think it should be possible to wait for other side's ack before sending handles, just need to find time to implement it.

[jgrahamc]

Thanks. I'll see what I can do.

John.

On Mon, Feb 17, 2014 at 1:18 PM, Fedor Indutny notifications@github.comwrote:

@jgrahamc https://github.com/jgrahamc I got to this
https://github.com/indutny/libuv/compare/fix;gh-926 . I think it should
be possible to wait for other side's ack before sending handles, just need
to find time to implement it.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/926#issuecomment-35255871
.

My book: The Geek Atlas, 128 Places Where Science and Technology Come Alive
http://geekatlas.com/ Signed copies: http://bit.ly/eSvmsV

[stakach]

Was just having a play around with the latest commits.
Looks like now when we have a pipe that supports write2 the pipe fails to accept: "accept error resource temporarily unavailable"

Regular pipes continue to work fine in Windows for me.

[saghul]

No IPC tests fail for me on Windows. Using pripes to send TCP handles across processes, that is. Unfortunately passing handles across threads continues to be an unsolved problem.

[neoxic]

Fedor, I wonder if your pid sending patch is somewhere on the way.

Another (easier) option would be to leave it to the user, i.e. provide a windows-only function to set the ipc_pid field on Windows before they can send handles. Yeah, clumsy, but more robust and can be "officially" backed by platform "security requirements".:-)

[jgrahamc]

CloudFlare is willing to pay a small bounty for proper support for this on
Windows. We would like to use libuv cross platform for a networked
application that we will be open sourcing. If there's someone who is wants
to work on this and has the skillset please get in contact with me directly.

John.

On Thu, Mar 27, 2014 at 6:20 PM, Arseny Vakhrushev <notifications@github.com

wrote:

Fedor, I wonder if your pid sending patch is somewhere on the way.

Another (easier) option would be to leave it to the user, i.e. provide a
windows-only function to set the ipc_pid field on Windows before they can
send handles. Yeah, clumsy, but more robust and can be "officially"
supported by platform "security requirements".:-)

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/926#issuecomment-38841522
.

My book: The Geek Atlas, 128 Places Where Science and Technology Come Alive
http://geekatlas.com/ Signed copies: http://bit.ly/eSvmsV

[retep998]

@indutny Did you intend to close this issue with your commit?

[tojocky]

Any news on this topic?

[piscisaureus]

What are the use cases that people need this for?
Load balancing incoming connections, or something else?

@jgrahamc and @stakach

[jgrahamc]

Specifically, we (CloudFlare) have a piece of software based on libuv that uses OpenSSL and we use libuv for threading and for handling a listen socket with multiple connections.

See my original comment above: #926 (comment)

[txdv]

If the bounty is still up, I can look at it at the weekend.

[juanper]

Is possible setup ipc_pid in every call write2()?
I would like create my own pipe server side in my parent processes and control when pass tcp handles to my child processes.
Using uv_spawn() to pass pipe to child processes, libuv create 1 temporary pipe handle server side to each child process. Creating 1000 child processes, I'll have 1000 temporary pipes server handles and 1000 locally pipe handles passes to child processes using stdio struct in uv_spawn(), getting a total of 2000 handles.
If I use my own created server pipe handle this number is reduced to 1001 handles in parent process. Today I need set ipc_pid = GetCurrentProcessId() to use write2() and don't get error. (tested in libuv v1.4.2)

[stakach]

@juanper I'm not sure I understand the question and/or how it relates to the issue of passing TCP handles over pipes on Windows within the one process?

Or are you saying that if you create a proxy process and pass the handles to it you can then have the second process pass the sockets back to the original process?

[saghul]

You should be able to create the pipe yourself (pass 1 for IPC in
uv_pipe_init) and then in uv_spawn use the UV_INHERIT_STREAM flag.
On Apr 30, 2015 6:22 AM, "Juan José Pereira" notifications@github.com
wrote:

Is possible setup ipc_pid in every call write2()?
I would like create my own pipe server side in my parent processes and
control when pass tcp handles to my child processes.
Using uv_spawn() to pass pipe to child processes, libuv create 1 temporary
pipe handle server side to each child process. Creating 1000 child
processes, I'll have 1000 temporary pipes server handles and 1000 locally
pipe handles passes to child processes using stdio struct in uv_spawn(),
getting a total of 2000 handles.
If I use my own created server pipe handle this number is reduced to 1001
handles in parent process. Today I need set ipc_pid = GetCurrentProcessId()
to use write2() and don't get error. (tested in libuv v1.4.2)

—
Reply to this email directly or view it on GitHub
#926 (comment).

[juanper]

@stakach in my case, I create a process who accept tcp connection, read first data block and according content of data block transfers tpc stream handle to a child process.

[juanper]

@saghul I used UV_INHERIT_STREAM but all my attemps fails.
In uv__stdio_create() there is a test if "stream->flags & UV_HANDLE_CONNECTED" that allway fails in my tests. Have you a example code to follow? Thanks.

[saghul]

Fixed in libuv/libuv#540

[stakach]

Amazing work! Thanks!

[juanper]

Great, thanks!

[KeyserSoSay0]

Sorry to revive a dead thread, but this still doesn't work by default and this is the first thread that comes up in searches.

So if you're wondering how to pass sockets between two different processes on windows: the trick is that WSADuplicateSocket(used inside uv_write2) actually requires the PID of the process that you want to pass the socket to, not the process that you're passing the socket from(which is what libuv currently assumes).

There's not really a way to do this internally to libuv, so you end up having to send a few handshake messages where you trade PIDs when you're setting up your connections. Then you can just manually set YourStream.pipe.conn.ipc_pid = WhateverThePIDOfTheOtherProcessIs sometime before you call uv_write2.