Console not falling back to running es

[TomonoriSoejima]

I am testing support for multiple elasticsearch_nodes and found it is working when I stop one of the es node but Dev tools -> Console is not falling back to the alive es node.

This is how I tested it.

1. start 2 es nodes

ES_PATH_CONF=./conf1 elasticsearch/bin/elasticsearch -Enode.name=node1 -Ecluster.name=elasticsearch -Epath.data=node1_data -d 
ES_PATH_CONF=./conf2 elasticsearch/bin/elasticsearch -Enode.name=node2 -Ecluster.name=elasticsearch -Epath.data=node2_data -d

2. start kibana with the following set in kibana.yml
   elasticsearch.hosts: ["http://localhost:9200","http://localhost:9201"]

3. make sure you can use the most feature of kibana such as Console and Discover.

For instance, I did GET / in http://localhost:5601/app/kibana#/dev_tools/console?_g=()

response

{
  "name" : "node1",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "xk8_BJHFQU-4NXSA_TBcdQ",
  "version" : {
    "number" : "6.6.0",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "a9861f4",
    "build_date" : "2019-01-24T11:27:09.439740Z",
    "build_snapshot" : false,
    "lucene_version" : "7.6.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

GET /_cat/nodes?v

ip        heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
127.0.0.1           32          99  15    2.03                  mdi       -      node2
127.0.0.1           36          99  15    2.03                  mdi       *      node1

4. check kibana connection.

[wave:tools]$ k_pid=$(lsof -i:5601 | grep node | awk -F ' ' '{print $2}')
[wave:tools]$ lsof -P -n -p $k_pid | grep TCP
node    96456 surfer   25u    IPv4 0x716ccc6fe8bedf5f      0t0       TCP 127.0.0.1:5601 (LISTEN)
node    96456 surfer   26u    IPv4 0x716ccc7000d04f5f      0t0       TCP 127.0.0.1:56327->127.0.0.1:9201 (ESTABLISHED)
node    96456 surfer   27u    IPv4 0x716ccc6ff6c7e25f      0t0       TCP 127.0.0.1:56338->127.0.0.1:9200 (ESTABLISHED)
node    96456 surfer   28u    IPv4 0x716ccc6fd42375df      0t0       TCP 127.0.0.1:56328->127.0.0.1:9200 (ESTABLISHED)
node    96456 surfer   29u    IPv4 0x716ccc6ffa3d125f      0t0       TCP 127.0.0.1:56211->127.0.0.1:9200 (ESTABLISHED)
node    96456 surfer   30u    IPv4 0x716ccc6ffb25a5df      0t0       TCP 127.0.0.1:56212->127.0.0.1:9201 (ESTABLISHED)
node    96456 surfer   31u    IPv4 0x716ccc6ffab9ef5f      0t0       TCP 127.0.0.1:56644->127.0.0.1:9201 (ESTABLISHED)
node    96456 surfer   32u    IPv4 0x716ccc6ff8affc5f      0t0       TCP 127.0.0.1:56683->127.0.0.1:9200 (ESTABLISHED)
node    96456 surfer   33u    IPv4 0x716ccc7004072f5f      0t0       TCP 127.0.0.1:56645->127.0.0.1:9201 (ESTABLISHED)
node    96456 surfer   34u    IPv4 0x716ccc70011635df      0t0       TCP 127.0.0.1:56215->127.0.0.1:9200 (ESTABLISHED)
node    96456 surfer   35u    IPv4 0x716ccc6ff6c01f5f      0t0       TCP 127.0.0.1:56684->127.0.0.1:9200 (ESTABLISHED)
node    96456 surfer   36u    IPv4 0x716ccc6ffb55425f      0t0       TCP 127.0.0.1:56216->127.0.0.1:9201 (ESTABLISHED)
node    96456 surfer   37u    IPv4 0x716ccc70045a05df      0t0       TCP 127.0.0.1:56217->127.0.0.1:9200 (ESTABLISHED)
node    96456 surfer   38u    IPv4 0x716ccc700459fc5f      0t0       TCP 127.0.0.1:56218->127.0.0.1:9201 (ESTABLISHED)
node    96456 surfer   39u    IPv4 0x716ccc6fe80f125f      0t0       TCP 127.0.0.1:56219->127.0.0.1:9200 (ESTABLISHED)
node    96456 surfer   41u    IPv4 0x716ccc6ff0c7d5df      0t0       TCP 127.0.0.1:56228->127.0.0.1:9200 (ESTABLISHED)
node    96456 surfer   42u    IPv4 0x716ccc6feb2545df      0t0       TCP 127.0.0.1:56237->127.0.0.1:9201 (ESTABLISHED)
node    96456 surfer   43u    IPv4 0x716ccc6ff71715df      0t0       TCP 127.0.0.1:56238->127.0.0.1:9201 (ESTABLISHED)
node    96456 surfer   44u    IPv4 0x716ccc6fd10fef5f      0t0       TCP 127.0.0.1:56239->127.0.0.1:9201 (ESTABLISHED)
node    96456 surfer   47u    IPv4 0x716ccc6fd423925f      0t0       TCP 127.0.0.1:56329->127.0.0.1:9201 (ESTABLISHED)
node    96456 surfer   48u    IPv4 0x716ccc6fd4236c5f      0t0       TCP 127.0.0.1:56330->127.0.0.1:9200 (ESTABLISHED)
node    96456 surfer   49u    IPv4 0x716ccc6ffd05e8df      0t0       TCP 127.0.0.1:56331->127.0.0.1:9201 (ESTABLISHED)
node    96456 surfer   52u    IPv4 0x716ccc6ffb552f5f      0t0       TCP 127.0.0.1:56389->127.0.0.1:9200 (ESTABLISHED)
[wave:tools]$ 

5. stop the es whose name is node1 by locating the pid by jps -v | grep -i conf1

[wave:6.6.0]$ jps -v  | grep -i conf1
96419 Elasticsearch -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp2 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:logs/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -Des.path.home=/Users/surfer/elastic/labs/6.6.0/elasticsearch -Des.path.conf=/Users/surfer/elastic/labs/6.6.0/conf1 -Des.distribution.flavor=default -Des.distribution.type=tar
[wave:6.6.0]$ kill -9 96419

6. check kibana connection again and see that kibana is now exclusively connected to port 9201 which is the second es node.

[wave:tools]$ lsof -P -n -p $k_pid | grep TCP
node    96456 surfer   25u    IPv4 0x716ccc6fe8bedf5f      0t0       TCP 127.0.0.1:5601 (LISTEN)
node    96456 surfer   26u    IPv4 0x716ccc7000d04f5f      0t0       TCP 127.0.0.1:56327->127.0.0.1:9201 (ESTABLISHED)
node    96456 surfer   27u    IPv4 0x716ccc6ff6c015df      0t0       TCP 127.0.0.1:56803->127.0.0.1:9201 (ESTABLISHED)
node    96456 surfer   28u    IPv4 0x716ccc6ffd05df5f      0t0       TCP 127.0.0.1:56813->127.0.0.1:9201 (ESTABLISHED)
node    96456 surfer   29u    IPv4 0x716ccc6ffd05d5df      0t0       TCP 127.0.0.1:56814->127.0.0.1:9201 (ESTABLISHED)
node    96456 surfer   30u    IPv4 0x716ccc6ffb25a5df      0t0       TCP 127.0.0.1:56212->127.0.0.1:9201 (ESTABLISHED)
node    96456 surfer   31u    IPv4 0x716ccc6ffafbcc5f      0t0       TCP 127.0.0.1:56722->127.0.0.1:9201 (ESTABLISHED)
node    96456 surfer   32u    IPv4 0x716ccc6fd4b8d25f      0t0       TCP 127.0.0.1:56815->127.0.0.1:9201 (ESTABLISHED)
node    96456 surfer   33u    IPv4 0x716ccc6fe905025f      0t0       TCP 127.0.0.1:56723->127.0.0.1:9201 (ESTABLISHED)
node    96456 surfer   34u    IPv4 0x716ccc6fe07a4f5f      0t0       TCP 127.0.0.1:56816->127.0.0.1:9201 (ESTABLISHED)
node    96456 surfer   36u    IPv4 0x716ccc6ffb55425f      0t0       TCP 127.0.0.1:56216->127.0.0.1:9201 (ESTABLISHED)
node    96456 surfer   38u    IPv4 0x716ccc700459fc5f      0t0       TCP 127.0.0.1:56218->127.0.0.1:9201 (ESTABLISHED)
node    96456 surfer   39u    IPv4 0x716ccc6ffafddc5f      0t0       TCP 127.0.0.1:56842->127.0.0.1:9201 (ESTABLISHED)
node    96456 surfer   40u    IPv4 0x716ccc6fdf8005df      0t0       TCP 127.0.0.1:56841->127.0.0.1:9201 (ESTABLISHED)
node    96456 surfer   42u    IPv4 0x716ccc6feb2545df      0t0       TCP 127.0.0.1:56237->127.0.0.1:9201 (ESTABLISHED)
node    96456 surfer   43u    IPv4 0x716ccc6ff71715df      0t0       TCP 127.0.0.1:56238->127.0.0.1:9201 (ESTABLISHED)
node    96456 surfer   44u    IPv4 0x716ccc6fd10fef5f      0t0       TCP 127.0.0.1:56239->127.0.0.1:9201 (ESTABLISHED)
node    96456 surfer   47u    IPv4 0x716ccc6fd423925f      0t0       TCP 127.0.0.1:56329->127.0.0.1:9201 (ESTABLISHED)
node    96456 surfer   49u    IPv4 0x716ccc6ffd05e8df      0t0       TCP 127.0.0.1:56331->127.0.0.1:9201 (ESTABLISHED)

7. Doing GET / in http://localhost:5601/app/kibana#/dev_tools/console?_g=() again comes back with this resposne, but Kibana itself is working such as Discover is fully functional.

{
  "message": "Client request error: connect ECONNREFUSED 127.0.0.1:9200",
  "statusCode": 502,
  "error": "Bad Gateway"
}

[elasticmachine]

Pinging @elastic/es-ui

[cjcenizal]

Just wanted to add that I ran into this during stack upgrade testing. I set up a multi-node cluster, enabled the 6.7 settings for node sniffing, and performed a rolling upgrade. Mid-upgrade, Console stopped working when the node to which Kibana was connected was taken offline.

Ideally, Console should connect to the same node Kibana connects to by default. We could also add support for selecting a node via a combobox (#11866).

[kobelb]

I was working with another customer today who ran into this issue as well, and would like me to be their proxy for +1'ing this issue.

[sulmansarwar]

+1.
I ran into this today during Elasticsearch Engineer 2 training session....

[jloleysens]

This should be addressed by #50185