Docker Engine can not connect to etcd after the cluster change endpoint. #35859
Description
Description
Docker use etcd as store and use etcd1.domain,etcd2.domain,etcd3.domain as it's config. libkv will AutoSync etcd cluster endpoint every 5 minutes. But if a etcd cluster recovered from unhealty state and the libkv will exit the AutoSync goroutine.Then if you change the endpoint of the etcd node in cluster. Docker will never know the real ip address.
Steps to reproduce the issue:
1.config docker daemon with etcd endpoint (domain name).
2.cut down the connection between docker and etcd cluster to trigger the error of AutoSync.
3.change the etcd node endpoint and refresh the domain name resolution.
Describe the results you received:
Docker will still use the client before,because the etcd AutoSync is failed.
Describe the results you expected:
libkv will AutoSync the new endpoint and renew the etcd client,so that Docker will connect to the real etcd endpoint
Additional information you deem important (e.g. issue happens only occasionally):
Output of docker version:
Docker version 17.06.2-ce, build a176465
Output of docker info:
Server Version: 17.06.2-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 55
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: acd local nas ossfs
Network: bridge host macvlan null overlay vpc
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 6e23458c129b551d5c9871e5174f6b1b7f6d1170
runc version: 810190ceaa507aa2727d7ae6f4790c76ec150bd2
init version: 949e6fa
Security Options:
apparmor
Kernel Version: 4.4.0-63-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.859GiB
Name: iZm5eadbj7exchu19krn3uZ
ID: ENFJ:TJHY:KJX3:NKDE:4IBB:GVMX:JFRA:FZMT:OIZV:DFLV:K5PK:MPLM
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
**Additional environment details (AWS, VirtualBox, physical, etc.):
Aliyun Container Service