无法使用 testpmd 通过 virtio 将 pkts 发送到 qemu 进程中的 VM
Cannot use testpmd to send pkts to VM in qemu process though virtio
我正在尝试测试 vhost-user/virtio-net。我使用 testpmd 将 pkts（在 txonly 模式下）发送到 qemu VM。但是 testpmd 显示所有 pkts 都被丢弃了。这是我的环境：
DPDK version: 19.08
(HOST) Hugepagesize=1GB Hugepages=16
# testpmd cmd
testpmd -l 0-3 -n 4 --socket-mem 1024 --vdev 'net_vhost0,iface=/tmp/sock0,queues=1' -- -i
# qemu cmd
qemu-system-x86_64 /opt/vm/centos/vm.img \
-cpu qemu64,+ssse3,+sse4.1,+sse4.2 \
--enable-kvm \
--nographic -vnc :0 \
-smp 4 \
-m 4096 -mem-path /dev/hugepages,share=on -mem-prealloc \
-chardev socket,id=chr0,path=/tmp/sock0 \
-netdev vhost-user,id=net0,chardev=chr0,queues=1,vhostforce \
-device virtio-net-pci,netdev=net0,ioeventfd=on,mac=52:54:00:00:00:14 \
-netdev tap,id=tapnet0,ifname=tap1,script=no,downscript=no \
-device e1000,netdev=tapnet0
link 已建立并且 virt-queue 已初始化：
VHOST_CONFIG: new vhost user connection is 31
VHOST_CONFIG: new device, handle is 0
VHOST_CONFIG: read message VHOST_USER_GET_FEATURES
VHOST_CONFIG: read message VHOST_USER_GET_PROTOCOL_FEATURES
VHOST_CONFIG: read message VHOST_USER_SET_PROTOCOL_FEATURES
VHOST_CONFIG: negotiated Vhost-user protocol features: 0x7
VHOST_CONFIG: read message VHOST_USER_GET_QUEUE_NUM
VHOST_CONFIG: read message VHOST_USER_SET_OWNER
VHOST_CONFIG: read message VHOST_USER_GET_FEATURES
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:0 file:32
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:1 file:33
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 1 to qp idx: 0
Port 0: queue state event
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 1 to qp idx: 1
Port 0: queue state event
VHOST_CONFIG: read message VHOST_USER_SET_FEATURES
VHOST_CONFIG: negotiated Virtio features: 0x7820ffc3
VHOST_CONFIG: read message VHOST_USER_SET_MEM_TABLE
VHOST_CONFIG: guest memory region 0, size: 0x40000000
guest physical addr: 0x100000000
guest virtual addr: 0x7f9f7fe00000
host virtual addr: 0x7f83d8000000
mmap addr : 0x7f8318000000
mmap size : 0x100000000
mmap align: 0x1000
mmap off : 0xc0000000
VHOST_CONFIG: guest memory region 1, size: 0xa0000
guest physical addr: 0x0
guest virtual addr: 0x7f9ebfe00000
host virtual addr: 0x7f8421788000
mmap addr : 0x7f8421788000
mmap size : 0xa0000
mmap align: 0x1000
mmap off : 0x0
VHOST_CONFIG: guest memory region 2, size: 0xbff40000
guest physical addr: 0xc0000
guest virtual addr: 0x7f9ebfec0000
host virtual addr: 0x7f82580c0000
mmap addr : 0x7f8258000000
mmap size : 0xc0000000
mmap align: 0x1000
mmap off : 0xc0000
VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM
VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK
VHOST_CONFIG: vring kick idx:0 file:37
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:0 file:38
VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM
VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK
VHOST_CONFIG: vring kick idx:1 file:32
VHOST_CONFIG: virtio is now ready for processing.
Port 0: link state change event
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:1 file:39
在 vm 中设置 nic promisc:
# ifconfig eth0 promisc up
testpmd 显示 link 状态已启动：
testpmd> show port info all
********************* Infos for port 0 *********************
MAC address: 56:48:4F:53:54:00
Device name: net_vhost0
Driver name: net_vhost
Devargs: iface=/tmp/sock0,queues=1
Connect to socket: 0
memory allocation on the socket: 0
Link status: up
Link speed: 10000 Mbps
Link duplex: full-duplex
MTU: 1500
Promiscuous mode: disabled
Allmulticast mode: disabled
Maximum number of MAC addresses: 1
Maximum number of MAC addresses of hash filtering: 0
VLAN offload:
strip off
filter off
qinq(extend) off
No RSS offload flow type is supported.
Minimum size of RX buffer: 0
Maximum configurable length of RX packet: 4294967295
Current number of RX queues: 1
Max possible RX queues: 1
Max possible number of RXDs per queue: 65535
Min possible number of RXDs per queue: 0
RXDs number alignment: 1
Current number of TX queues: 1
Max possible TX queues: 1
Max possible number of TXDs per queue: 65535
Min possible number of TXDs per queue: 0
TXDs number alignment: 1
Max segment number per packet: 65535
Max segment number per MTU/TSO: 65535
testpmd 开始发送 pkts:
testpmd> set fwd txonly
Set txonly packet forwarding mode
testpmd> start
停止测试pmd:
Waiting for lcores to finish...
---------------------- Forward statistics for port 0 ----------------------
RX-packets: 0 RX-dropped: 0 RX-total: 0
TX-packets: 0 TX-dropped: 17056768 TX-total: 17056768
----------------------------------------------------------------------------
+++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
RX-packets: 0 RX-dropped: 0 RX-total: 0
TX-packets: 0 TX-dropped: 17056768 TX-total: 17056768
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Done.
所有 pkts 都被丢弃了。
我错过了什么吗？
看起来它有 DPDK 或 NUMA 支持的页面问题。这同样适用于 DPDK 版本 19.11 LTS 和 20.11 LTS。
DPDK 应用程序： rm /tmp/sock0; sudo ./build/l2fwd --legacy-mem -l 1-2 --no-pci --vdev=net_vhost0,iface=/tmp/sock0 --vdev=net_tap0 -m 1024 -- -p 3 -T 1 --no-mac-updating
QEMU： taskset -c 4-9 qemu-system-x86_64 -cpu host -enable-kvm -m 1024 -smp 4,sockets=1,cores=4,threads=1 \ -object memory-backend-file,id=mem,size=1024M,mem-path=/mnt/huge,share=on \ -numa node,memdev=mem,nodeid=0 -mem-prealloc \ -name test \ -no-reboot \ -vnc none \ -nographic \ -net user,hostfwd=tcp::10023-:22 -net nic \ -chardev socket,id=charnet0,path=/tmp/sock0 \ -netdev type=vhost-user,chardev=charnet0,queues=1,id=hostnet0 \ -device virtio-net-pci,mq=on,vectors=18,netdev=hostnet0,id=net0,mac=fa:16:3e:52:30:73 \ -hda [disk name]
VM 启动后，您可以通过ssh port 10023登录。
我正在尝试测试 vhost-user/virtio-net。我使用 testpmd 将 pkts（在 txonly 模式下）发送到 qemu VM。但是 testpmd 显示所有 pkts 都被丢弃了。这是我的环境：
DPDK version: 19.08
(HOST) Hugepagesize=1GB Hugepages=16
# testpmd cmd
testpmd -l 0-3 -n 4 --socket-mem 1024 --vdev 'net_vhost0,iface=/tmp/sock0,queues=1' -- -i
# qemu cmd
qemu-system-x86_64 /opt/vm/centos/vm.img \
-cpu qemu64,+ssse3,+sse4.1,+sse4.2 \
--enable-kvm \
--nographic -vnc :0 \
-smp 4 \
-m 4096 -mem-path /dev/hugepages,share=on -mem-prealloc \
-chardev socket,id=chr0,path=/tmp/sock0 \
-netdev vhost-user,id=net0,chardev=chr0,queues=1,vhostforce \
-device virtio-net-pci,netdev=net0,ioeventfd=on,mac=52:54:00:00:00:14 \
-netdev tap,id=tapnet0,ifname=tap1,script=no,downscript=no \
-device e1000,netdev=tapnet0
link 已建立并且 virt-queue 已初始化：
VHOST_CONFIG: new vhost user connection is 31
VHOST_CONFIG: new device, handle is 0
VHOST_CONFIG: read message VHOST_USER_GET_FEATURES
VHOST_CONFIG: read message VHOST_USER_GET_PROTOCOL_FEATURES
VHOST_CONFIG: read message VHOST_USER_SET_PROTOCOL_FEATURES
VHOST_CONFIG: negotiated Vhost-user protocol features: 0x7
VHOST_CONFIG: read message VHOST_USER_GET_QUEUE_NUM
VHOST_CONFIG: read message VHOST_USER_SET_OWNER
VHOST_CONFIG: read message VHOST_USER_GET_FEATURES
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:0 file:32
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:1 file:33
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 1 to qp idx: 0
Port 0: queue state event
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 1 to qp idx: 1
Port 0: queue state event
VHOST_CONFIG: read message VHOST_USER_SET_FEATURES
VHOST_CONFIG: negotiated Virtio features: 0x7820ffc3
VHOST_CONFIG: read message VHOST_USER_SET_MEM_TABLE
VHOST_CONFIG: guest memory region 0, size: 0x40000000
guest physical addr: 0x100000000
guest virtual addr: 0x7f9f7fe00000
host virtual addr: 0x7f83d8000000
mmap addr : 0x7f8318000000
mmap size : 0x100000000
mmap align: 0x1000
mmap off : 0xc0000000
VHOST_CONFIG: guest memory region 1, size: 0xa0000
guest physical addr: 0x0
guest virtual addr: 0x7f9ebfe00000
host virtual addr: 0x7f8421788000
mmap addr : 0x7f8421788000
mmap size : 0xa0000
mmap align: 0x1000
mmap off : 0x0
VHOST_CONFIG: guest memory region 2, size: 0xbff40000
guest physical addr: 0xc0000
guest virtual addr: 0x7f9ebfec0000
host virtual addr: 0x7f82580c0000
mmap addr : 0x7f8258000000
mmap size : 0xc0000000
mmap align: 0x1000
mmap off : 0xc0000
VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM
VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK
VHOST_CONFIG: vring kick idx:0 file:37
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:0 file:38
VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM
VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK
VHOST_CONFIG: vring kick idx:1 file:32
VHOST_CONFIG: virtio is now ready for processing.
Port 0: link state change event
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:1 file:39
在 vm 中设置 nic promisc:
# ifconfig eth0 promisc up
testpmd 显示 link 状态已启动：
testpmd> show port info all
********************* Infos for port 0 *********************
MAC address: 56:48:4F:53:54:00
Device name: net_vhost0
Driver name: net_vhost
Devargs: iface=/tmp/sock0,queues=1
Connect to socket: 0
memory allocation on the socket: 0
Link status: up
Link speed: 10000 Mbps
Link duplex: full-duplex
MTU: 1500
Promiscuous mode: disabled
Allmulticast mode: disabled
Maximum number of MAC addresses: 1
Maximum number of MAC addresses of hash filtering: 0
VLAN offload:
strip off
filter off
qinq(extend) off
No RSS offload flow type is supported.
Minimum size of RX buffer: 0
Maximum configurable length of RX packet: 4294967295
Current number of RX queues: 1
Max possible RX queues: 1
Max possible number of RXDs per queue: 65535
Min possible number of RXDs per queue: 0
RXDs number alignment: 1
Current number of TX queues: 1
Max possible TX queues: 1
Max possible number of TXDs per queue: 65535
Min possible number of TXDs per queue: 0
TXDs number alignment: 1
Max segment number per packet: 65535
Max segment number per MTU/TSO: 65535
testpmd 开始发送 pkts:
testpmd> set fwd txonly
Set txonly packet forwarding mode
testpmd> start
停止测试pmd:
Waiting for lcores to finish...
---------------------- Forward statistics for port 0 ----------------------
RX-packets: 0 RX-dropped: 0 RX-total: 0
TX-packets: 0 TX-dropped: 17056768 TX-total: 17056768
----------------------------------------------------------------------------
+++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
RX-packets: 0 RX-dropped: 0 RX-total: 0
TX-packets: 0 TX-dropped: 17056768 TX-total: 17056768
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Done.
所有 pkts 都被丢弃了。
我错过了什么吗？
看起来它有 DPDK 或 NUMA 支持的页面问题。这同样适用于 DPDK 版本 19.11 LTS 和 20.11 LTS。
DPDK 应用程序：
rm /tmp/sock0; sudo ./build/l2fwd --legacy-mem -l 1-2 --no-pci --vdev=net_vhost0,iface=/tmp/sock0 --vdev=net_tap0 -m 1024 -- -p 3 -T 1 --no-mac-updatingQEMU：
taskset -c 4-9 qemu-system-x86_64 -cpu host -enable-kvm -m 1024 -smp 4,sockets=1,cores=4,threads=1 \ -object memory-backend-file,id=mem,size=1024M,mem-path=/mnt/huge,share=on \ -numa node,memdev=mem,nodeid=0 -mem-prealloc \ -name test \ -no-reboot \ -vnc none \ -nographic \ -net user,hostfwd=tcp::10023-:22 -net nic \ -chardev socket,id=charnet0,path=/tmp/sock0 \ -netdev type=vhost-user,chardev=charnet0,queues=1,id=hostnet0 \ -device virtio-net-pci,mq=on,vectors=18,netdev=hostnet0,id=net0,mac=fa:16:3e:52:30:73 \ -hda [disk name]
VM 启动后，您可以通过ssh port 10023登录。