Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Downgrade ECONNRESET Error: socket hang up in logs to debug #22225

Closed
inqueue opened this issue Aug 21, 2018 · 3 comments · Fixed by #31742
Closed

Downgrade ECONNRESET Error: socket hang up in logs to debug #22225

inqueue opened this issue Aug 21, 2018 · 3 comments · Fixed by #31742
Labels
bug Fixes for quality problems that affect the customer experience Team:Operations Team label for Operations Team v6.5.1

Comments

@inqueue
Copy link
Member

inqueue commented Aug 21, 2018

Kibana version: 6.3.1
Elasticsearch version: 6.3.1

Kibana logs this error when connections are made that do not establish an actual TLS session; e.g., connection checks either with netcat/telnet or load balancers:

{"type":"error","@timestamp":"2018-08-15T22:09:52Z","tags":
["connection","client","error"],"pid":14699,"level":"error","error":{"message":"socket hang 
up","name":"Error","stack":"Error: socket hang up\n    at TLSSocket.<anonymous> 
(_tls_wrap.js:876:25)\n    at emitOne (events.js:121:20)\n    at TLSSocket.emit (events.js:211:7)\n    at 
_handle.close (net.js:557:12)\n    at Socket.done (_tls_wrap.js:356:7)\n    at Object.onceWrapper 
(events.js:315:30)\n    at emitOne (events.js:121:20)\n    at Socket.emit (events.js:211:7)\n    at 
TCP._handle.close [as _onclose] (net.js:557:12)","code":"ECONNRESET"},"message":"socket hang up"}

Reproduce on a TLS enabled Kibana with nc kibana.host 5601, then CTRL+C. If a load balancer causes it, the log message populates the log at an uncomfortable rate.

Could this logging be suppressed to debug level only?

@lukasolson lukasolson added Team:Operations Team label for Operations Team triage_needed labels Aug 21, 2018
@bhansenmis
Copy link

I am also having this issue. Happening on kibana nodes behind an AWS loadbalancer. It would be great if these errors could be suppressed

@kobelb kobelb added the bug Fixes for quality problems that affect the customer experience label Sep 10, 2018
@tylersmalley
Copy link
Contributor

We're looking into this.

@bhansenmis you can terminate SSL at the LB to resolve this. With the upside of having centralized SSL management.

@bhansenmis
Copy link

@tylersmalley Thanks for the reply. I am currently terminating SSL at the load balancer with a company cert, and then we have self-signed certs between the load balancer and the kibana nodes. We didn't want to have unencrypted traffic between the lb and the kibana nodes. So with that setup, this issue still occurs.

@spalger spalger added v6.5.1 and removed v6.5.0 labels Nov 14, 2018
tylersmalley pushed a commit to tylersmalley/kibana that referenced this issue Feb 21, 2019
errno is not set for ECONNRESET

https://github.com/hapijs/wreck/blob/29d8f4b5eaa12c5d44c642130cd6002c1d83ffd2/lib/index.js#L290

Closes elastic#22225

Example:

{ Error: socket hang up
  at TLSSocket.onSocketClose (_tls_wrap.js:761:23)
  at TLSSocket.emit (events.js:194:15)
  at _handle.close (net.js:600:12)
  at Socket.done (_tls_wrap.js:388:7)
  at Object.onceWrapper (events.js:277:13)
  at Socket.emit (events.js:189:13)
  at TCP._handle.close (net.js:600:12) code: 'ECONNRESET'

Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
tylersmalley added a commit that referenced this issue Feb 23, 2019
errno is not set for ECONNRESET

https://github.com/hapijs/wreck/blob/29d8f4b5eaa12c5d44c642130cd6002c1d83ffd2/lib/index.js#L290

Closes #22225

Example:

{ Error: socket hang up
  at TLSSocket.onSocketClose (_tls_wrap.js:761:23)
  at TLSSocket.emit (events.js:194:15)
  at _handle.close (net.js:600:12)
  at Socket.done (_tls_wrap.js:388:7)
  at Object.onceWrapper (events.js:277:13)
  at Socket.emit (events.js:189:13)
  at TCP._handle.close (net.js:600:12) code: 'ECONNRESET'

Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
tylersmalley added a commit to tylersmalley/kibana that referenced this issue Feb 23, 2019
errno is not set for ECONNRESET

https://github.com/hapijs/wreck/blob/29d8f4b5eaa12c5d44c642130cd6002c1d83ffd2/lib/index.js#L290

Closes elastic#22225

Example:

{ Error: socket hang up
  at TLSSocket.onSocketClose (_tls_wrap.js:761:23)
  at TLSSocket.emit (events.js:194:15)
  at _handle.close (net.js:600:12)
  at Socket.done (_tls_wrap.js:388:7)
  at Object.onceWrapper (events.js:277:13)
  at Socket.emit (events.js:189:13)
  at TCP._handle.close (net.js:600:12) code: 'ECONNRESET'

Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
tylersmalley added a commit to tylersmalley/kibana that referenced this issue Feb 23, 2019
errno is not set for ECONNRESET

https://github.com/hapijs/wreck/blob/29d8f4b5eaa12c5d44c642130cd6002c1d83ffd2/lib/index.js#L290

Closes elastic#22225

Example:

{ Error: socket hang up
  at TLSSocket.onSocketClose (_tls_wrap.js:761:23)
  at TLSSocket.emit (events.js:194:15)
  at _handle.close (net.js:600:12)
  at Socket.done (_tls_wrap.js:388:7)
  at Object.onceWrapper (events.js:277:13)
  at Socket.emit (events.js:189:13)
  at TCP._handle.close (net.js:600:12) code: 'ECONNRESET'

Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
tylersmalley added a commit that referenced this issue Feb 23, 2019
errno is not set for ECONNRESET

https://github.com/hapijs/wreck/blob/29d8f4b5eaa12c5d44c642130cd6002c1d83ffd2/lib/index.js#L290

Closes #22225

Example:

{ Error: socket hang up
  at TLSSocket.onSocketClose (_tls_wrap.js:761:23)
  at TLSSocket.emit (events.js:194:15)
  at _handle.close (net.js:600:12)
  at Socket.done (_tls_wrap.js:388:7)
  at Object.onceWrapper (events.js:277:13)
  at Socket.emit (events.js:189:13)
  at TCP._handle.close (net.js:600:12) code: 'ECONNRESET'

Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
tylersmalley added a commit that referenced this issue Feb 23, 2019
errno is not set for ECONNRESET

https://github.com/hapijs/wreck/blob/29d8f4b5eaa12c5d44c642130cd6002c1d83ffd2/lib/index.js#L290

Closes #22225

Example:

{ Error: socket hang up
  at TLSSocket.onSocketClose (_tls_wrap.js:761:23)
  at TLSSocket.emit (events.js:194:15)
  at _handle.close (net.js:600:12)
  at Socket.done (_tls_wrap.js:388:7)
  at Object.onceWrapper (events.js:277:13)
  at Socket.emit (events.js:189:13)
  at TCP._handle.close (net.js:600:12) code: 'ECONNRESET'

Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
tylersmalley added a commit to tylersmalley/kibana that referenced this issue Feb 28, 2019
errno is not set for ECONNRESET

https://github.com/hapijs/wreck/blob/29d8f4b5eaa12c5d44c642130cd6002c1d83ffd2/lib/index.js#L290

Closes elastic#22225

Example:

{ Error: socket hang up
  at TLSSocket.onSocketClose (_tls_wrap.js:761:23)
  at TLSSocket.emit (events.js:194:15)
  at _handle.close (net.js:600:12)
  at Socket.done (_tls_wrap.js:388:7)
  at Object.onceWrapper (events.js:277:13)
  at Socket.emit (events.js:189:13)
  at TCP._handle.close (net.js:600:12) code: 'ECONNRESET'

Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
tylersmalley added a commit that referenced this issue Mar 14, 2019
errno is not set for ECONNRESET

https://github.com/hapijs/wreck/blob/29d8f4b5eaa12c5d44c642130cd6002c1d83ffd2/lib/index.js#L290

Closes #22225

Example:

{ Error: socket hang up
  at TLSSocket.onSocketClose (_tls_wrap.js:761:23)
  at TLSSocket.emit (events.js:194:15)
  at _handle.close (net.js:600:12)
  at Socket.done (_tls_wrap.js:388:7)
  at Object.onceWrapper (events.js:277:13)
  at Socket.emit (events.js:189:13)
  at TCP._handle.close (net.js:600:12) code: 'ECONNRESET'

Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Fixes for quality problems that affect the customer experience Team:Operations Team label for Operations Team v6.5.1
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants