无法使用覆盖在多主机网络中向 consul store 注册 dockerhost
Unable to register dockerhost with consul store in multihost networking using overlay
我正在尝试在多主机 docker 网络中测试覆盖驱动程序概念，但在向 consul store 注册 dochost2 时出现错误
dochost1:
root@dochost1:/usr/lib/systemd# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 08:00:27:38:6a:0b brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic noprefixroute enp0s3
valid_lft 68198sec preferred_lft 68198sec
inet6 fe80::a00:27ff:fe38:6a0b/64 scope link
valid_lft forever preferred_lft forever
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 08:00:27:c7:bc:23 brd ff:ff:ff:ff:ff:ff
inet 192.168.56.9/24 brd 192.168.56.255 scope global noprefixroute enp0s8
valid_lft forever preferred_lft forever
inet6 fe80::a00:27ff:fec7:bc23/64 scope link
valid_lft forever preferred_lft forever
4: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:3c:4a:00:d9 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80::42:3cff:fe4a:d9/64 scope link
valid_lft forever preferred_lft forever
24: vetha7949d3@if23: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default
link/ether 4e:48:66:9e:d7:14 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::4c48:66ff:fe9e:d714/64 scope link
valid_lft forever preferred_lft forever
dochost2:
root@dochost2:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 08:00:27:a0:17:7a brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic noprefixroute enp0s3
valid_lft 82941sec preferred_lft 82941sec
inet6 fe80::c88:231a:9eb7:7d7c/64 scope link noprefixroute
valid_lft forever preferred_lft forever
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 08:00:27:b8:91:45 brd ff:ff:ff:ff:ff:ff
inet 192.168.50.0/24 brd 192.168.50.255 scope global noprefixroute enp0s8
valid_lft forever preferred_lft forever
inet6 fe80::a00:27ff:feb8:9145/64 scope link
valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:49:2b:5e:ef brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
dochost1 - 具有领事商店和详细信息如下
内核版本：
root@dochost1:/usr/lib/systemd# uname -r
5.0.0-23-generic
ubuntu版本：
root@dochost1:/usr/lib/systemd# cat /etc/*release*
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS"
NAME="Ubuntu"
VERSION="18.04.3 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.3 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
docker版本：
root@dochost1:/usr/lib/systemd# docker --version
Docker version 19.03.5, build 633a0ea838
root@dochost1:/usr/lib/systemd#
领事容器信息：
root@dochost1:/usr/lib/systemd# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
237b81df3720 progrium/consul "/bin/start -server …" 2 hours ago Up 2 hours 53/tcp, 53/udp, 8300-8302/tcp, 8400/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp consul
root@dochost1:/usr/lib/systemd#
docker dochost1 的信息:
root@dochost1:/usr/lib/systemd# docker info
Client:
Debug Mode: false
Server:
Containers: 4
Running: 1
Paused: 0
Stopped: 3
Images: 7
Server Version: 19.03.5
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.0.0-23-generic
Operating System: Ubuntu 18.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.944GiB
Name: dochost1
ID: OCT2:CMAB:WPLU:VDL6:MZNH:CWXM:XMRU:CCHH:NK6S:XYRZ:RCWF:52PQ
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Cluster Store: consul://192.168.56.9:8500/network
Cluster Advertise: 192.168.56.9:0
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
在第二个 docker 主机节点 - dochost2，我正在尝试如下编辑 /lib/systemd/system/docker.service 文件 execstart 并重新启动 docker
ExecStart=/usr/bin/dockerd -H fd:// --cluster-store=consul://192.168.56.9:8500/network --cluster-advertise=enp0s8:2376 --containerd=/run/containerd/containerd.sock
我在 journalctl -u docker 输出中收到以下错误
Dec 17 13:34:41 dochost2 dockerd[2370]: time="2019-12-17T13:34:41.713159844+05:30" level=error msg="discovery error: Get http://192.168.56.9:8500/v1/kv/network/docker/nodes?consistent=: dial tcp 192.168.56.9:8500: i/o timeout"
Dec 17 13:35:11 dochost2 dockerd[2370]: time="2019-12-17T13:35:11.714004242+05:30" level=error msg="discovery error: Put http://192.168.56.9:8500/v1/kv/network/docker/nodes?flags=3304740253564472344: dial tcp 192.168.56.9:8500: i/o timeout"
Dec 17 13:35:41 dochost2 dockerd[2370]: time="2019-12-17T13:35:41.714024951+05:30" level=error msg="discovery error: Unexpected watch error"
两台主机上也没有配置防火墙
root@dochost1:/usr/lib/systemd# ufw status
Status: inactive
root@dochost1:/usr/lib/systemd#
root@dochost2:~# ufw status
Status: inactive
root@dochost2:~#
纯网络相关 issue.Since 这 2 docker 主机在不同的子网中，它们之间没有网络连接 nodes.So 我使用了 linux 网桥概念并添加到网桥下方以在这 2 docker 台主机
之间建立连接
在 dochost 2 上：
首先我将 enp0s8 接口 ip 更改为 192.168.50.9，因为我不能使用 192.168.50.0/24 作为接口 ip，因为它指向整个子网 (192.168.50.0 - 255) ip，我们需要给 bridge
分配一个 ip
root@dochost2:~# ip addr show enp0s8
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master host1_bridge1 state UP group default qlen 1000
link/ether 08:00:27:b8:91:45 brd ff:ff:ff:ff:ff:ff
inet 192.168.50.9/24 brd 192.168.50.255 scope global noprefixroute enp0s8
valid_lft forever preferred_lft foreverdebug2: channel 0: window 999381 sent adjust 49195
root@dochost2:~#
ip link add host1_bridge1 type bridge
ip address add 192.168.56.8/24 dev host1_bridge1
ip link set dev enp0s8 master host1_bridge1
ip link set host1_bridge1 up
完成上述步骤后，我能够 ping 和 ssh dochost 1 - 192.168.56.9
在 dochost1 上：
在指向 dochost2 上的网桥接口 ip 的路由下方添加
route add -host 192.168.50.9 gw 192.168.56.8
经过上面的路由，我可以从dochost1连接到dochost2
完成上述步骤后，我刚刚在 dochost2
上重新加载 docker 守护进程
systemctl daemon-reload
在 dochost2 上使用 journalctl -u docker 检查了日志，现在没有看到任何错误
4346354+05:30" level=info msg="2019/12/19 14:22:59 [INFO] serf: EventMemberJoin: dochost2 192.168.50.9\n"
8140350+05:30" level=info msg="2019/12/19 14:22:59 [INFO] serf: EventMemberJoin: dochost1 192.168.56.9\n"
即使是为了测试这个，我在 dochost1 上创建了一个覆盖网络 myoverlay，我在 dochost2
上也看到了
root@dochost1:~# docker network ls
NETWORK ID NAME DRIVER SCOPE
28e8ae4d1273 bridge bridge local
b5b6ab1e41d2 host host local
e49864108832 myoverlay overlay global
8c156d319aa8 none null local
root@dochost1:~#
root@dochost2:~# docker network ls
NETWORK ID NAME DRIVER SCOPE
e3c773160654 bridge bridge local
8bb2cb8d6e5c host host local
e49864108832 myoverlay overlay global
3427614365c3 none null local
root@dochost2:~#
我正在尝试在多主机 docker 网络中测试覆盖驱动程序概念，但在向 consul store 注册 dochost2 时出现错误
dochost1:
root@dochost1:/usr/lib/systemd# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 08:00:27:38:6a:0b brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic noprefixroute enp0s3
valid_lft 68198sec preferred_lft 68198sec
inet6 fe80::a00:27ff:fe38:6a0b/64 scope link
valid_lft forever preferred_lft forever
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 08:00:27:c7:bc:23 brd ff:ff:ff:ff:ff:ff
inet 192.168.56.9/24 brd 192.168.56.255 scope global noprefixroute enp0s8
valid_lft forever preferred_lft forever
inet6 fe80::a00:27ff:fec7:bc23/64 scope link
valid_lft forever preferred_lft forever
4: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:3c:4a:00:d9 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80::42:3cff:fe4a:d9/64 scope link
valid_lft forever preferred_lft forever
24: vetha7949d3@if23: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default
link/ether 4e:48:66:9e:d7:14 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::4c48:66ff:fe9e:d714/64 scope link
valid_lft forever preferred_lft forever
dochost2:
root@dochost2:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 08:00:27:a0:17:7a brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic noprefixroute enp0s3
valid_lft 82941sec preferred_lft 82941sec
inet6 fe80::c88:231a:9eb7:7d7c/64 scope link noprefixroute
valid_lft forever preferred_lft forever
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 08:00:27:b8:91:45 brd ff:ff:ff:ff:ff:ff
inet 192.168.50.0/24 brd 192.168.50.255 scope global noprefixroute enp0s8
valid_lft forever preferred_lft forever
inet6 fe80::a00:27ff:feb8:9145/64 scope link
valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:49:2b:5e:ef brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
dochost1 - 具有领事商店和详细信息如下
内核版本：
root@dochost1:/usr/lib/systemd# uname -r
5.0.0-23-generic
ubuntu版本：
root@dochost1:/usr/lib/systemd# cat /etc/*release*
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS"
NAME="Ubuntu"
VERSION="18.04.3 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.3 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
docker版本：
root@dochost1:/usr/lib/systemd# docker --version
Docker version 19.03.5, build 633a0ea838
root@dochost1:/usr/lib/systemd#
领事容器信息：
root@dochost1:/usr/lib/systemd# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
237b81df3720 progrium/consul "/bin/start -server …" 2 hours ago Up 2 hours 53/tcp, 53/udp, 8300-8302/tcp, 8400/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp consul
root@dochost1:/usr/lib/systemd#
docker dochost1 的信息:
root@dochost1:/usr/lib/systemd# docker info
Client:
Debug Mode: false
Server:
Containers: 4
Running: 1
Paused: 0
Stopped: 3
Images: 7
Server Version: 19.03.5
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.0.0-23-generic
Operating System: Ubuntu 18.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.944GiB
Name: dochost1
ID: OCT2:CMAB:WPLU:VDL6:MZNH:CWXM:XMRU:CCHH:NK6S:XYRZ:RCWF:52PQ
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Cluster Store: consul://192.168.56.9:8500/network
Cluster Advertise: 192.168.56.9:0
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
在第二个 docker 主机节点 - dochost2，我正在尝试如下编辑 /lib/systemd/system/docker.service 文件 execstart 并重新启动 docker
ExecStart=/usr/bin/dockerd -H fd:// --cluster-store=consul://192.168.56.9:8500/network --cluster-advertise=enp0s8:2376 --containerd=/run/containerd/containerd.sock
我在 journalctl -u docker 输出中收到以下错误
Dec 17 13:34:41 dochost2 dockerd[2370]: time="2019-12-17T13:34:41.713159844+05:30" level=error msg="discovery error: Get http://192.168.56.9:8500/v1/kv/network/docker/nodes?consistent=: dial tcp 192.168.56.9:8500: i/o timeout"
Dec 17 13:35:11 dochost2 dockerd[2370]: time="2019-12-17T13:35:11.714004242+05:30" level=error msg="discovery error: Put http://192.168.56.9:8500/v1/kv/network/docker/nodes?flags=3304740253564472344: dial tcp 192.168.56.9:8500: i/o timeout"
Dec 17 13:35:41 dochost2 dockerd[2370]: time="2019-12-17T13:35:41.714024951+05:30" level=error msg="discovery error: Unexpected watch error"
两台主机上也没有配置防火墙
root@dochost1:/usr/lib/systemd# ufw status
Status: inactive
root@dochost1:/usr/lib/systemd#
root@dochost2:~# ufw status
Status: inactive
root@dochost2:~#
纯网络相关 issue.Since 这 2 docker 主机在不同的子网中，它们之间没有网络连接 nodes.So 我使用了 linux 网桥概念并添加到网桥下方以在这 2 docker 台主机
之间建立连接在 dochost 2 上：
首先我将 enp0s8 接口 ip 更改为 192.168.50.9，因为我不能使用 192.168.50.0/24 作为接口 ip，因为它指向整个子网 (192.168.50.0 - 255) ip，我们需要给 bridge
分配一个 iproot@dochost2:~# ip addr show enp0s8
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master host1_bridge1 state UP group default qlen 1000
link/ether 08:00:27:b8:91:45 brd ff:ff:ff:ff:ff:ff
inet 192.168.50.9/24 brd 192.168.50.255 scope global noprefixroute enp0s8
valid_lft forever preferred_lft foreverdebug2: channel 0: window 999381 sent adjust 49195
root@dochost2:~#
ip link add host1_bridge1 type bridge
ip address add 192.168.56.8/24 dev host1_bridge1
ip link set dev enp0s8 master host1_bridge1
ip link set host1_bridge1 up
完成上述步骤后，我能够 ping 和 ssh dochost 1 - 192.168.56.9
在 dochost1 上：
在指向 dochost2 上的网桥接口 ip 的路由下方添加
route add -host 192.168.50.9 gw 192.168.56.8
经过上面的路由，我可以从dochost1连接到dochost2
完成上述步骤后，我刚刚在 dochost2
上重新加载 docker 守护进程systemctl daemon-reload
在 dochost2 上使用 journalctl -u docker 检查了日志，现在没有看到任何错误
4346354+05:30" level=info msg="2019/12/19 14:22:59 [INFO] serf: EventMemberJoin: dochost2 192.168.50.9\n"
8140350+05:30" level=info msg="2019/12/19 14:22:59 [INFO] serf: EventMemberJoin: dochost1 192.168.56.9\n"
即使是为了测试这个，我在 dochost1 上创建了一个覆盖网络 myoverlay，我在 dochost2
上也看到了root@dochost1:~# docker network ls
NETWORK ID NAME DRIVER SCOPE
28e8ae4d1273 bridge bridge local
b5b6ab1e41d2 host host local
e49864108832 myoverlay overlay global
8c156d319aa8 none null local
root@dochost1:~#
root@dochost2:~# docker network ls
NETWORK ID NAME DRIVER SCOPE
e3c773160654 bridge bridge local
8bb2cb8d6e5c host host local
e49864108832 myoverlay overlay global
3427614365c3 none null local
root@dochost2:~#