Domain: sometimes print warn log when domain closed

[3pointer]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do?
   TiDB domain integrated into BR pkg, when BR finished and close the domain sometimes it will print warn log

{"level":"warn","ts":"2020-02-27T14:38:56.499+0800","caller":"clientv3/retry_interceptor.go:61","msg":"retrying of unary invoker failed","target":"endpoint://client-9177a4a5-a0a9-4dae-b5a3-83e51c4aada7/127.0.0.1:2379","attempt":0,"error":"rpc error: code = Canceled desc = context canceled"}

the reason as I can tell is when dom.Close(), there exists concurrent close, and if etcd session closed first, it will retry sometimes and print above warn log.
it will make user confused whether BR is success or not. please help fix it.

2. What did you expect to see?
   no warn log

3. What did you see instead?
   sometimes warn log

4. What version of TiDB are you using (tidb-server -V or run select tidb_version(); on TiDB)?
   master

[zimulala]

@3pointer
As you said, if it is caused by concurrent close, it may be that etcdClient is repeatedly closed. We need to ensure that etcdClient is not closed before domain.Close. Maybe we can confirm with etcdClient.Ctx (). Done.

If it is caused by etcd being closed, then make sure etcd is closed before domain.Close. The specific method needs to be confirmed.

[3pointer]

@zimulala After I tried, I found that we can reduce the warn log in this way, but it's hard to solve it completely in restore cmd.

because when we are doing restore

1. we will get a domain to start DDL job.
2. we start a new goroutine to run campaignLoop, we can not control it exit as we want, even if we use ctx.Cancel in ddl.Stop
3. when we run revokeSession, it will sometimes print warn log, when etcdClient closed.

[zimulala]

@3pointer
Or whether to consider changing the log level of the etcd client to error?