server: add diagnostics service

[lonng]

Signed-off-by: Lonng heng@lonng.org

What problem does this PR solve?

This PR embed Diagnostics service to the PD to make it diagnosable. (Part of pingcap/tidb#13567)

What is changed and how it works?

Register Diagnostics service to PD at server startup.

Check List

Tests

• Manual test (add detailed scripts or steps below)

mysql> select * from cluster_log where type='pd' and level='warn' and message like '%';
+-------------------------+------+-----------------+-------+---------------------------------------------------------------------------------------------------------------------------+
| TIME                    | TYPE | ADDRESS         | LEVEL | MESSAGE                                                                                                                   |
+-------------------------+------+-----------------+-------+---------------------------------------------------------------------------------------------------------------------------+
| 2019/12/16 16:28:28.357 | pd   | 127.0.0.1:49901 | Warn  | [store.go:1288] ["simple token is not cryptographically signed"]                                                          |
| 2019/12/16 16:28:29.852 | pd   | 127.0.0.1:49901 | Warn  | [history_buffer.go:138] ["load history index failed"] [error="leveldb: not found"]                                        |
| 2019/12/16 16:29:03.773 | pd   | 127.0.0.1:49901 | Warn  | [cluster.go:383] ["store does not have enough disk space"] [store-id=1] [capacity=1574498656256] [available=280283320320] |
| 2019/12/16 16:29:13.774 | pd   | 127.0.0.1:49901 | Warn  | [cluster.go:383] ["store does not have enough disk space"] [store-id=1] [capacity=1574498656256] [available=280283267072] |
| 2019/12/16 16:29:23.775 | pd   | 127.0.0.1:49901 | Warn  | [cluster.go:383] ["store does not have enough disk space"] [store-id=1] [capacity=1574498656256] [available=280283267072] |
| 2019/12/16 16:29:33.776 | pd   | 127.0.0.1:49901 | Warn  | [cluster.go:383] ["store does not have enough disk space"] [store-id=1] [capacity=1574498656256] [available=280283267072] |
| 2019/12/16 16:29:43.777 | pd   | 127.0.0.1:49901 | Warn  | [cluster.go:383] ["store does not have enough disk space"] [store-id=1] [capacity=1574498656256] [available=280283267072] |
| 2019/12/16 16:29:53.778 | pd   | 127.0.0.1:49901 | Warn  | [cluster.go:383] ["store does not have enough disk space"] [store-id=1] [capacity=1574498656256] [available=280283267072] |
| 2019/12/16 16:30:03.779 | pd   | 127.0.0.1:49901 | Warn  | [cluster.go:383] ["store does not have enough disk space"] [store-id=1] [capacity=1574498656256] [available=280283267072] |
| 2019/12/16 16:30:13.780 | pd   | 127.0.0.1:49901 | Warn  | [cluster.go:383] ["store does not have enough disk space"] [store-id=1] [capacity=1574498656256] [available=280283267072] |
| 2019/12/16 16:30:23.781 | pd   | 127.0.0.1:49901 | Warn  | [cluster.go:383] ["store does not have enough disk space"] [store-id=1] [capacity=1574498656256] [available=280283262976] |
+-------------------------+------+-----------------+-------+---------------------------------------------------------------------------------------------------------------------------+
11 rows in set (0.00 sec)

mysql> select * from cluster_log where type='pd' and message like '%scheduler%' and message regexp '.*coordinator.*';
+-------------------------+------+-----------------+-------+-----------------------------------------------------------------------------------------+
| TIME                    | TYPE | ADDRESS         | LEVEL | MESSAGE                                                                                 |
+-------------------------+------+-----------------+-------+-----------------------------------------------------------------------------------------+
| 2019/12/16 16:28:56.764 | pd   | 127.0.0.1:49901 | Info  | [coordinator.go:172] ["coordinator starts to run schedulers"]                           |
| 2019/12/16 16:28:56.765 | pd   | 127.0.0.1:49901 | Info  | [coordinator.go:236] ["create scheduler"] [scheduler-name=balance-region-scheduler]     |
| 2019/12/16 16:28:56.766 | pd   | 127.0.0.1:49901 | Info  | [coordinator.go:236] ["create scheduler"] [scheduler-name=balance-leader-scheduler]     |
| 2019/12/16 16:28:56.766 | pd   | 127.0.0.1:49901 | Info  | [coordinator.go:236] ["create scheduler"] [scheduler-name=balance-hot-region-scheduler] |
| 2019/12/16 16:28:56.767 | pd   | 127.0.0.1:49901 | Info  | [coordinator.go:236] ["create scheduler"] [scheduler-name=label-scheduler]              |
+-------------------------+------+-----------------+-------+-----------------------------------------------------------------------------------------+
5 rows in set (0.01 sec)

[lhy1024]

LGTM. But there are some error with ci.

[crazycs520]

LGTM

[disksing]

/merge

[sre-bot]

/run-all-tests

[sre-bot]

@lonng merge failed.

[disksing]

CI not run. retry...

/merge

[disksing]

/merge

[sre-bot]

/run-all-tests

[lonng]

/run-all-tests

[sre-bot]

@lonng merge failed.