Investigate and possibly downgrade socket hang ups

[tylersmalley]

 log   [18:51:30.985] [error][client][connection] Error: socket hang up
   at TLSSocket.<anonymous> (_tls_wrap.js:822:25)
   at emitOne (events.js:101:20)
   at TLSSocket.emit (events.js:188:7)
   at Socket._handle.close (net.js:498:12)
   at Socket.g (events.js:291:16)
   at emitOne (events.js:101:20)
   at Socket.emit (events.js:188:7)
   at TCP._handle.close [as _onclose] (net.js:498:12)

[gmoskovicz]

Thanks for opening this @tylersmalley . At a high level it would be great if we can just turn of specific logs such as we do with log4j in Java (Elasticsearch and Logstash)? What do you think?

[tylersmalley]

We have an interceptor in place to handle downgrading these issues. It should be a matter of verifying these can be safely ignored and adding to the interceptor.

[gmoskovicz]

Roger that, thanks for confirming @tylersmalley

[turtlemonvh]

Hello,

I'm the customer who was experiencing this error, so let me know if you need some debug stuff run. For now I can tell you that we are accessing this service over an ssh tunnel using self signed TLS certs. We are updating the service so we can also access it via an nginx proxy, and I'll let you know if that changes the error I'm seeing.

[tylersmalley]

Thanks @turtlemonvh, let me know your findings with the proxy.

[turtlemonvh]

@tylersmalley I'm still seeing the same log line warnings when accessing through the nginx proxy that I saw when accessing via the http tunnel.

However, I don't think that these errors are due to the configuration of clients connecting in to kibana, since even after I kill all connections coming in to port 5601 and restart kibana I still see the same lines in the kibana logs. (I originally thought the client was an issue because this seemed to be the cause of most of the errors I saw other people discussing when they saw the socket hang up line in their logs.)

Maybe has to do with the outbound connection to elasticsearch instead?

[tylersmalley]

@turtlemonvh still have not been able to reproduce this. What do you have configured in your kibana.yml file?

The error is coming from Hapi, which is the Node server we use. It's possible it could be bubbling up from Elasticsearch through the proxy we create. Do you see anything abnormal in your ES logs?

I will continue to investigate.

[tylersmalley]

Resolved in #31742