Docker 容器内的 Macvlan 网络没有到达它自己的主机
Docker Macvlan network inside container is not reaching to its own host
我已经在 2 docker 主机之间设置了 Macvlan 网络，如下所示：
主机设置：HOST_1 ens192: 172.18.0.21
创建 macvlan 网桥接口
docker network create -d macvlan \
--subnet=172.18.0.0/22 \
--gateway=172.18.0.1 \
--ip-range=172.18.1.0/28 \
-o macvlan_mode=bridge \
-o parent=ens192 macvlan
创建macvlan接口HOST_1
ip link add ens192.br link ens192 type macvlan mode bridge
ip addr add 172.18.1.0/28 dev ens192.br
ip link set dev ens192.br up
主机设置：HOST_2 ens192: 172.18.0.23
创建 macvlan 网桥接口
docker network create -d macvlan \
--subnet=172.18.0.0/22 \
--gateway=172.18.0.1 \
--ip-range=172.18.1.16/28 \
-o macvlan_mode=bridge \
-o parent=ens192 macvlan
在HOST_2
中创建macvlan接口
ip link add ens192.br link ens192 type macvlan mode bridge
ip addr add 172.18.1.16/28 dev ens192.br
ip link set dev ens192.br up
容器设置
在两个主机中创建容器
HOST_1# docker run --net=macvlan -it --name macvlan_1 --rm alpine /bin/sh
HOST_2# docker run --net=macvlan -it --name macvlan_1 --rm alpine /bin/sh
CONTAINER_1 在 HOST_1
24: eth0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UNKNOWN
link/ether 02:42:ac:12:01:00 brd ff:ff:ff:ff:ff:ff
inet 172.18.1.0/22 brd 172.18.3.255 scope global eth0
valid_lft forever preferred_lft forever
CONTAINER_2 在 HOST_2
21: eth0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UNKNOWN
link/ether 02:42:ac:12:01:10 brd ff:ff:ff:ff:ff:ff
inet 172.18.1.16/22 brd 172.18.3.255 scope global eth0
valid_lft forever preferred_lft forever
CONTAINER_1 和 CONTAINER_2
中的路线 table
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 172.18.0.1 0.0.0.0 UG 0 0 0 eth0
172.18.0.0 0.0.0.0 255.255.252.0 U 0 0 0 eth0
场景
HOST_1 (172.18.0.21) <-> HOST_2 (172.18.0.23) = OK（反之亦然）
HOST_1 (172.18.0.21) -> CONTAINER_1 (172.18.1.0) and CONTAINER_2 (172.18.1.16) = 好的
HOST_2 (172.18.0.23) -> CONTAINER_1 (172.18.1.0) and CONTAINER_2 (172.18.1.16) = 好的
CONTAINER_1 (172.18.1.0) -> HOST_2 (172.18.0.23) = 好的
CONTAINER_2 (172.18.1.16) -> HOST_1 (172.18.0.21) = 好的
CONTAINER_1 (172.18.1.0) <-> CONTAINER_2 (172.18.1.16) = OK（反之亦然）
CONTAINER_1 (172.18.1.0) -> HOST_1 (172.18.0.21) = 失败
CONTAINER_2 (172.18.1.16) -> HOST_2 (172.18.0.23) = 失败
问题
我非常接近我想要实现的解决方案，除了这 1 个问题。我怎样才能让容器连接到它自己的主机。如果有解决办法，我想知道如何在 ESXi 虚拟化角度和裸机上进行配置，如果有什么不同
这是为 macvlan 定义的行为，是设计使然。参见 Docker Macvlan Documentation
When using macvlan, you cannot ping or communicate with the default namespace IP address. For example, if you create a container and try to ping the Docker host’s eth0, it will not work. That traffic is explicitly filtered by the kernel modules themselves to offer additional provider isolation and security.
A macvlan subinterface can be added to the Docker host, to allow traffic between the Docker host and containers. The IP address needs to be set on this subinterface and removed from the parent address.
这个问题“有点老”，但其他人可能会觉得它有用。在 USING DOCKER MACVLAN NETWORKS BY LARS KELLOGG-STEDMAN 的 主机访问 部分中描述了一个解决方法。我可以确认 - 它正在运行。
Host access With a container attached to a macvlan network, you will
find that while it can contact other systems on your local network
without a problem, the container will not be able to connect to your
host (and your host will not be able to connect to your container).
This is a limitation of macvlan interfaces: without special support
from a network switch, your host is unable to send packets to its own
macvlan interfaces.
Fortunately, there is a workaround for this problem: you can create
another macvlan interface on your host, and use that to communicate
with containers on the macvlan network.
First, I’m going to reserve an address from our network range for use
by the host interface by using the --aux-address option to docker
network create. That makes our final command line look like:
docker network create -d macvlan -o parent=eno1 \
--subnet 192.168.1.0/24 \
--gateway 192.168.1.1 \
--ip-range 192.168.1.192/27 \
--aux-address 'host=192.168.1.223' \
mynet
This will prevent Docker from assigning that address to a container.
Next, we create a new macvlan interface on the host. You can call it
whatever you want, but I’m calling this one mynet-shim:
ip link add mynet-shim link eno1 type macvlan mode bridge
Now we need to configure the interface with the address we reserved
and bring it up:
ip addr add 192.168.1.223/32 dev mynet-shim
ip link set mynet-shim up
The last thing we need to do is to tell our host to use that interface
when communicating with the containers. This is relatively easy
because we have restricted our containers to a particular CIDR subset
of the local network; we just add a route to that range like this:
ip route add 192.168.1.192/27 dev mynet-shim
With that route in place, your host will automatically use ths
mynet-shim interface when communicating with containers on the mynet
network.
Note that the interface and routing configuration presented here is
not persistent – you will lose if if you were to reboot your host. How
to make it persistent is distribution dependent.
在我的情况下，我向容器中添加了一个网络。
因此 CONTAINER_1 -> HOST_1 可以通过不同的 IP (10.123.0.2) 访问。
CONTAINER_2 或 HOST_2 可以到达 172.18.1.0.
以下是 docker-compose 示例，希望这是一个解决方法。
version: "3"
services:
macvlan_1:
image: alpine
container: macvlan_1
command: ....
restart: always
networks:
macvlan:
ipv4_address: 172.18.1.0
internalbr:
ipv4_address: 10.123.0.2
networks:
macvlan:
driver: macvlan
driver_opts:
parent: ens192
macvlan_mode: bridge
ipam:
driver: default
config:
- subnet: 172.18.0.0/22
gateway: 172.18.0.1
ip_range: 172.18.1.0/28
internalbr:
driver: bridge
ipam:
config:
- subnet: 10.123.0.0/24
我已经在 2 docker 主机之间设置了 Macvlan 网络，如下所示：
主机设置：HOST_1 ens192: 172.18.0.21
创建 macvlan 网桥接口
docker network create -d macvlan \
--subnet=172.18.0.0/22 \
--gateway=172.18.0.1 \
--ip-range=172.18.1.0/28 \
-o macvlan_mode=bridge \
-o parent=ens192 macvlan
创建macvlan接口HOST_1
ip link add ens192.br link ens192 type macvlan mode bridge
ip addr add 172.18.1.0/28 dev ens192.br
ip link set dev ens192.br up
主机设置：HOST_2 ens192: 172.18.0.23
创建 macvlan 网桥接口
docker network create -d macvlan \
--subnet=172.18.0.0/22 \
--gateway=172.18.0.1 \
--ip-range=172.18.1.16/28 \
-o macvlan_mode=bridge \
-o parent=ens192 macvlan
在HOST_2
中创建macvlan接口ip link add ens192.br link ens192 type macvlan mode bridge
ip addr add 172.18.1.16/28 dev ens192.br
ip link set dev ens192.br up
容器设置
在两个主机中创建容器
HOST_1# docker run --net=macvlan -it --name macvlan_1 --rm alpine /bin/sh
HOST_2# docker run --net=macvlan -it --name macvlan_1 --rm alpine /bin/sh
CONTAINER_1 在 HOST_1
24: eth0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UNKNOWN
link/ether 02:42:ac:12:01:00 brd ff:ff:ff:ff:ff:ff
inet 172.18.1.0/22 brd 172.18.3.255 scope global eth0
valid_lft forever preferred_lft forever
CONTAINER_2 在 HOST_2
21: eth0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UNKNOWN
link/ether 02:42:ac:12:01:10 brd ff:ff:ff:ff:ff:ff
inet 172.18.1.16/22 brd 172.18.3.255 scope global eth0
valid_lft forever preferred_lft forever
CONTAINER_1 和 CONTAINER_2
中的路线 tableKernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 172.18.0.1 0.0.0.0 UG 0 0 0 eth0
172.18.0.0 0.0.0.0 255.255.252.0 U 0 0 0 eth0
场景
HOST_1 (172.18.0.21) <-> HOST_2 (172.18.0.23) = OK（反之亦然）
HOST_1 (172.18.0.21) -> CONTAINER_1 (172.18.1.0) and CONTAINER_2 (172.18.1.16) = 好的
HOST_2 (172.18.0.23) -> CONTAINER_1 (172.18.1.0) and CONTAINER_2 (172.18.1.16) = 好的
CONTAINER_1 (172.18.1.0) -> HOST_2 (172.18.0.23) = 好的
CONTAINER_2 (172.18.1.16) -> HOST_1 (172.18.0.21) = 好的
CONTAINER_1 (172.18.1.0) <-> CONTAINER_2 (172.18.1.16) = OK（反之亦然）
CONTAINER_1 (172.18.1.0) -> HOST_1 (172.18.0.21) = 失败
CONTAINER_2 (172.18.1.16) -> HOST_2 (172.18.0.23) = 失败
问题
我非常接近我想要实现的解决方案，除了这 1 个问题。我怎样才能让容器连接到它自己的主机。如果有解决办法，我想知道如何在 ESXi 虚拟化角度和裸机上进行配置，如果有什么不同
这是为 macvlan 定义的行为，是设计使然。参见 Docker Macvlan Documentation
When using macvlan, you cannot ping or communicate with the default namespace IP address. For example, if you create a container and try to ping the Docker host’s eth0, it will not work. That traffic is explicitly filtered by the kernel modules themselves to offer additional provider isolation and security.
A macvlan subinterface can be added to the Docker host, to allow traffic between the Docker host and containers. The IP address needs to be set on this subinterface and removed from the parent address.
这个问题“有点老”，但其他人可能会觉得它有用。在 USING DOCKER MACVLAN NETWORKS BY LARS KELLOGG-STEDMAN 的 主机访问 部分中描述了一个解决方法。我可以确认 - 它正在运行。
Host access With a container attached to a macvlan network, you will find that while it can contact other systems on your local network without a problem, the container will not be able to connect to your host (and your host will not be able to connect to your container). This is a limitation of macvlan interfaces: without special support from a network switch, your host is unable to send packets to its own macvlan interfaces.
Fortunately, there is a workaround for this problem: you can create another macvlan interface on your host, and use that to communicate with containers on the macvlan network.
First, I’m going to reserve an address from our network range for use by the host interface by using the --aux-address option to docker network create. That makes our final command line look like:
docker network create -d macvlan -o parent=eno1 \ --subnet 192.168.1.0/24 \ --gateway 192.168.1.1 \ --ip-range 192.168.1.192/27 \ --aux-address 'host=192.168.1.223' \ mynetThis will prevent Docker from assigning that address to a container.
Next, we create a new macvlan interface on the host. You can call it whatever you want, but I’m calling this one mynet-shim:
ip link add mynet-shim link eno1 type macvlan mode bridgeNow we need to configure the interface with the address we reserved and bring it up:
ip addr add 192.168.1.223/32 dev mynet-shim ip link set mynet-shim upThe last thing we need to do is to tell our host to use that interface when communicating with the containers. This is relatively easy because we have restricted our containers to a particular CIDR subset of the local network; we just add a route to that range like this:
ip route add 192.168.1.192/27 dev mynet-shimWith that route in place, your host will automatically use ths mynet-shim interface when communicating with containers on the mynet network.
Note that the interface and routing configuration presented here is not persistent – you will lose if if you were to reboot your host. How to make it persistent is distribution dependent.
在我的情况下，我向容器中添加了一个网络。
因此 CONTAINER_1 -> HOST_1 可以通过不同的 IP (10.123.0.2) 访问。
CONTAINER_2 或 HOST_2 可以到达 172.18.1.0.
以下是 docker-compose 示例，希望这是一个解决方法。
version: "3"
services:
macvlan_1:
image: alpine
container: macvlan_1
command: ....
restart: always
networks:
macvlan:
ipv4_address: 172.18.1.0
internalbr:
ipv4_address: 10.123.0.2
networks:
macvlan:
driver: macvlan
driver_opts:
parent: ens192
macvlan_mode: bridge
ipam:
driver: default
config:
- subnet: 172.18.0.0/22
gateway: 172.18.0.1
ip_range: 172.18.1.0/28
internalbr:
driver: bridge
ipam:
config:
- subnet: 10.123.0.0/24