Test: Integrate skywalking-ui and oap-server

[hanahmily]

@apache/skywalking-committers
I'm excited to link UI with OAP server firstly and will have been on this work for a long time cause both sides are so unreliable.
This issue will record some results, discussion, and related issues or pr related to this topics.
I will use the code to build the test package in the following branches:

https://github.com/apache/incubator-skywalking
https://github.com/apache/incubator-skywalking-ui/tree/6.0.0/dev

All patches should be merged into them.

[wu-sheng]

@hanahmily I just move the milestone to 6.0.0-alpha. 6.0.0 milestone should be for the stable version. :)

[hanahmily]

Thx for reminding this @wu-sheng.

[hanahmily]

@peng-yongsheng
The mesh-receiver module doesn't start up with buffer cache error, logs is shown details:

2018-09-27 15:18:53,061 - org.apache.skywalking.oap.server.library.module.ModuleDefine -1556 [main] INFO  [] - Prepare the elasticsearch provider in storage module.
2018-09-27 15:18:53,102 - org.apache.skywalking.oap.server.library.module.ModuleDefine -1597 [main] INFO  [] - Prepare the standalone provider in cluster module.
2018-09-27 15:18:53,111 - org.apache.skywalking.oap.server.library.module.ModuleDefine -1606 [main] INFO  [] - Prepare the default provider in core module.
2018-09-27 15:18:53,191 - org.apache.skywalking.oap.server.library.server.grpc.GRPCServer -1686 [main] INFO  [] - Server started, host 0.0.0.0 listening on 11800
2018-09-27 15:18:53,222 - org.eclipse.jetty.util.log -1717 [main] INFO  [] - Logging initialized @1782ms to org.eclipse.jetty.util.log.Slf4jLog
2018-09-27 15:18:53,322 - org.apache.skywalking.oap.server.library.server.jetty.JettyServer -1817 [main] INFO  [] - http server root context path: /
2018-09-27 15:18:53,447 - org.apache.skywalking.oap.server.library.module.ModuleDefine -1942 [main] INFO  [] - Prepare the graphql provider in query module.
2018-09-27 15:18:53,861 - com.coxautodev.graphql.tools.SchemaClassScanner -2356 [main] WARN  [] - Schema type was defined but can never be accessed, and can be safely deleted: NodeType
2018-09-27 15:18:54,204 - org.apache.skywalking.oap.server.library.module.ModuleDefine -2699 [main] INFO  [] - Prepare the default provider in alarm module.
2018-09-27 15:18:54,226 - org.apache.skywalking.oap.server.library.module.ModuleDefine -2721 [main] INFO  [] - Prepare the default provider in service-mesh module.
2018-09-27 15:18:54,250 - org.apache.skywalking.oap.server.library.buffer.BufferStream -2745 [main] INFO  [] - Try to lock buffer directory, directory is: ../buffermesh/
2018-09-27 15:18:54,257 - org.apache.skywalking.oap.server.library.buffer.BufferStream -2752 [main] INFO  [] - Lock buffer directory successfully, directory is: ../buffermesh/
Exception in thread "main" java.lang.AssertionError: Still in preparing stage.
	at org.apache.skywalking.oap.server.library.module.ModuleManager.assertPreparedStage(ModuleManager.java:82)
	at org.apache.skywalking.oap.server.library.module.ModuleManager.find(ModuleManager.java:73)
	at org.apache.skywalking.aop.server.receiver.mesh.TelemetryDataDispatcher.setCache(TelemetryDataDispatcher.java:56)
	at org.apache.skywalking.aop.server.receiver.mesh.MeshReceiverProvider.prepare(MeshReceiverProvider.java:53)
	at org.apache.skywalking.oap.server.library.module.ModuleDefine.prepare(ModuleDefine.java:87)
	at org.apache.skywalking.oap.server.library.module.ModuleManager.init(ModuleManager.java:49)
	at org.apache.skywalking.oap.server.starter.OAPServerStartUp.main(OAPServerStartUp.java:38)

The application.yml is

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

cluster:
  standalone:
#  zookeeper:
#    hostPort: localhost:2181
#    # Retry Policy
#    baseSleepTimeMs: 1000 # initial amount of time to wait between retries
#    maxRetries: 3 # max number of times to retry
#  kubernetes:
#    watchTimeoutSeconds: 60
#    namespace: default
#    labelSelector: app=collector,release=skywalking
#    uidEnvName: SKYWALKING_COLLECTOR_UID
core:
  default:
    restHost: 0.0.0.0
    restPort: 12800
    restContextPath: /
    gRPCHost: 0.0.0.0
    gRPCPort: 11800
storage:
  elasticsearch:
    clusterNodes: localhost:9200
    indexShardsNumber: 2
    indexReplicasNumber: 0
    # Batch process setting, refer to https://www.elastic.co/guide/en/elasticsearch/client/java-api/5.5/java-docs-bulk-processor.html
    bulkActions: 2000 # Execute the bulk every 2000 requests
    bulkSize: 20 # flush the bulk every 20mb
    flushInterval: 10 # flush the bulk every 10 seconds whatever the number of requests
    concurrentRequests: 2 # the number of concurrent requests
    # Set a timeout on metric data. After the timeout has expired, the metric data will automatically be deleted.
    traceDataTTL: 90 # Unit is minute
    minuteMetricDataTTL: 90 # Unit is minute
    hourMetricDataTTL: 36 # Unit is hour
    dayMetricDataTTL: 45 # Unit is day
    monthMetricDataTTL: 18 # Unit is month
receiver-register:
  default:
receiver-trace:
  default:
    bufferPath: ../buffer/  # Path to trace buffer files, suggest to use absolute path
    bufferOffsetMaxFileSize: 100 # Unit is MB
    bufferDataMaxFileSize: 500 # Unit is MB
    bufferFileCleanWhenRestart: false
receiver-jvm:
  default:
service-mesh:
  default:
    bufferPath: ../buffermesh/  # Path to trace buffer files, suggest to use absolute path
    bufferOffsetMaxFileSize: 100 # Unit is MB
    bufferDataMaxFileSize: 500 # Unit is MB
    bufferFileCleanWhenRestart: false
istio-telemetry:
  default:
query:
  graphql:
    path: /graphql
alarm:
  default:

code base is 69cccd1

[wu-sheng]

Provide fix #1711 . This is my fault, an untest PR merged. @hanahmily

[hanahmily]

Some errors in Dashboard page after running AgentDataMock.

getThermodynamic

2018-09-27 18:50:38,860 - graphql.execution.SimpleDataFetcherExceptionHandler -6806144 [qtp1753150176-206] WARN  [] - Exception while fetching data (/getThermodynamic) : null
java.lang.NullPointerException: null
	at org.apache.skywalking.oap.server.core.storage.annotation.ValueColumnIds$ValueColumn.access$100(ValueColumnIds.java:44) ~[classes/:?]
	at org.apache.skywalking.oap.server.core.storage.annotation.ValueColumnIds.getValueCName(ValueColumnIds.java:37) ~[classes/:?]
	at org.apache.skywalking.oap.server.core.query.MetricQueryService.getThermodynamic(MetricQueryService.java:82) ~[classes/:?]
	at org.apache.skywalking.oap.query.graphql.resolver.MetricQuery.getThermodynamic(MetricQuery.java:66) ~[classes/:?]
	at org.apache.skywalking.oap.query.graphql.resolver.MetricQueryMethodAccess.invoke(Unknown Source) ~[reflectasm-1.11.7.jar:?]
	at com.coxautodev.graphql.tools.MethodFieldResolverDataFetcher.get(MethodFieldResolver.kt:147) ~[graphql-java-tools-5.2.3.jar:?]
	at graphql.execution.ExecutionStrategy.fetchField(ExecutionStrategy.java:227) [graphql-java-8.0.jar:?]
	at graphql.execution.ExecutionStrategy.resolveField(ExecutionStrategy.java:170) [graphql-java-8.0.jar:?]
	at graphql.execution.AsyncExecutionStrategy.execute(AsyncExecutionStrategy.java:59) [graphql-java-8.0.jar:?]

getAllEndpointTopN( name: "endpoint_Avg")

2018-09-27 18:50:38,869 - graphql.execution.SimpleDataFetcherExceptionHandler -6806153 [qtp1753150176-206] WARN  [] - Exception while fetching data (/getTopNSlowEndpoint) : Elasticsearch exception [type=index_not_found_exception, reason=no such index]
org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=index_not_found_exception, reason=no such index]
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:653) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:628) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:535) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:508) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
	at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:404) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
	at org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient.search(ElasticSearchClient.java:122) ~[classes/:?]
	at org.apache.skywalking.oap.server.storage.plugin.elasticsearch.query.AggregationQueryEsDAO.aggregation(AggregationQueryEsDAO.java:123) ~[classes/:?]
	at org.apache.skywalking.oap.server.storage.plugin.elasticsearch.query.AggregationQueryEsDAO.getAllEndpointTopN(AggregationQueryEsDAO.java:87) ~[classes/:?]
	at org.apache.skywalking.oap.server.core.query.AggregationQueryService.getAllEndpointTopN(AggregationQueryService.java:89) ~[classes/:?]
	at org.apache.skywalking.oap.query.graphql.resolver.AggregationQuery.getAllEndpointTopN(AggregationQuery.java:75) ~[classes/:?]
	at org.apache.skywalking.oap.query.graphql.resolver.AggregationQueryMethodAccess.invoke(Unknown Source) ~[reflectasm-1.11.7.jar:?]
	at com.coxautodev.graphql.tools.MethodFieldResolverDataFetcher.get(MethodFieldResolver.kt:147) ~[graphql-java-tools-5.2.3.jar:?]

[hanahmily]

{
"variables":{
"duration":{
"start":"2018-09-27 1958",
"end":"2018-09-27 2013",
"step":"MINUTE"
}
},
"query":"\n query Dashboard($duration: Duration!) {\n getGlobalBrief(duration: $duration) {\n numOfService\n numOfEndpoint\n numOfDatabase\n numOfCache\n numOfMQ\n }\n getAlarmTrend(duration: $duration) {\n numOfAlarm\n }\n getThermodynamic(duration: $duration, metric: {\n name: "all_heatmap"\n }) {\n nodes\n responseTimeStep: axisYStep\n }\n getTopNSlowEndpoint: getAllEndpointTopN(\n duration: $duration,\n name: "endpoint_avg",\n topN: 10,\n order: DES\n ) {\n key: id\n label: name\n value\n }\n getTopNServiceThroughput: getServiceTopN(\n duration: $duration,\n name: "service_cpm",\n topN: 10,\n order: DES\n ) {\n key: id\n label: name\n value\n }\n }\n "
}

[wu-sheng]

All metric name should be in lower case. The generation tool causes that.

[wu-sheng]

Submit this PR #1714 @hanahmily Could you try to run a test on this? This should fix the numberOfEndpoint not right. And remember to clear all existing data in ElasticSearch.

[wu-sheng]

@peng-yongsheng

Issue

1. No downstream data, including hour/day/month
2. TopologyQueryEsDAO#load can't get any data from service_relation_server/client_side index.
   
3. numOfService, numOfEndpoint query(s) use time range as condition, but in analysis side, there is no cores to update that. For more, read (4)
4. Hearbeat, Ping(new) uplink protocols have not processed, so, I assume this is the root cause of (3)

[wu-sheng]

I am closing this issue. Look like we passed local integration tests. Let's move to integration test in real env. FYI @ascrutae @hanahmily