[Docker] [macOS] moby/vpnkit exhausts docker resources

[yiwangwuqian]

In macOS 10.14.2, run with docker

2019-02-18 10:16:35,884 - ERROR - run:604 - Exception while handling connection <socket.socket fd=1024, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('172.17.0.2', 8899), raddr=('172.17.0.1', 50078)> with reason ValueError('filedescriptor out of range in select()')
Traceback (most recent call last):
  File "./proxy.py", line 600, in run
    self._process()
  File "./proxy.py", line 575, in _process
    r, w, x = select.select(rlist, wlist, xlist, 1)
ValueError: filedescriptor out of range in select()

[abhinavsingh]

I did run into a similar issue during development. This seems to be a MacOS specific issue, that too when proxy.py is set up as a system-wide MacOS HTTP/HTTPS proxy.

Example:

1. Start the docker container
2. From command line stress test proxy.py, example:
   hey -x http://127.0.0.1:8899 -c 10 -n 100 http://httpbin.org/get
   See https://github.com/rakyll/hey to setup hey or
   one can also use apache benchmark ab
3. Check open file descriptors by proxy.py in container:
   $ docker exec -it <container-id> /bin/sh
/app # lsof -p 1 | wc -l
53
4. Now update MacOS system proxy settings to use proxy.py. Check open file descriptors again. There are thousands of socket connection open in CLOSE_WAIT state, within a few seconds.

My initial investigation hinted that client connection somehow gets dropped/closed right after proxy.py has received the request. Unsure if the connection is closed by the client or it has to do with docker network settings. Anyways, this results in clients making calls for the same URL repeatedly in a tight loop, overwhelming proxy.py and eventually running into file descriptor out of range error. You will notice access log for same host:port combination again and again.

2019-02-21 02:33:47,089 - INFO - 9:access_log:695 - 172.17.0.1:47872 - b'CONNECT' b'17.249.9.246':443
2019-02-21 02:33:47,089 - INFO - 8:access_log:695 - 172.17.0.1:47864 - b'CONNECT' b'17.249.9.246':443
2019-02-21 02:33:47,090 - INFO - 9:access_log:695 - 172.17.0.1:47856 - b'CONNECT' b'17.249.9.246':443
2019-02-21 02:33:47,091 - INFO - 8:access_log:695 - 172.17.0.1:47850 - b'CONNECT' b'17.249.9.246':443
2019-02-21 02:33:47,092 - INFO - 8:access_log:695 - 172.17.0.1:47834 - b'CONNECT' b'17.249.9.246':443
2019-02-21 02:33:47,093 - INFO - 9:access_log:695 - 172.17.0.1:47840 - b'CONNECT' b'17.249.9.246':443
2019-02-21 02:33:47,095 - INFO - 9:access_log:695 - 172.17.0.1:47826 - b'CONNECT' b'17.249.9.246':443
2019-02-21 02:33:47,095 - INFO - 8:access_log:695 - 172.17.0.1:47820 - b'CONNECT' b'17.249.9.246':443

In above case, IP 17.249.9.246 belongs to Apple. Sometimes destination IP may belong to Google or other service providers running on your system.

Client IP 172.17.0.1 is docker bridge IP address:

$ docker network inspect bridge | grep -i gateway
                    "Gateway": "172.17.0.1"

[yiwangwuqian]

I don't know how to solve this,but thanks for your reply!