在虚拟化环境中使用 DPDK 内核 NIC 接口

Question

我要开发一个DPDK Linux application on a laptop, but the laptop's hardware is not supported by DPDK. Furtunately, DPDK supports paravirtualized devices including QEMU的virtio-net。

所以我正在尝试为运行在 virtio-net-pci 设备上配置 Kernel NIC Interface(KNI) QEMU 来宾。问题是 KNI 示例应用程序不接受 virtio-net-pci 驱动程序。

QEMU 命令

exec qemu-system-x86_64 -enable-kvm \
  -cpu host -smp 2 \
  -vga std \
  -mem-prealloc -mem-path /dev/hugepages \
  -drive file=GentooVM.img,if=virtio \
  -netdev user,id=vmnic,hostname=gentoo \
  -device virtio-net-pci,netdev=vmnic \
  -m 1024M \
  -monitor stdio \
  -name "Gentoo VM"

运行客户机中的 KNI 示例应用程序

sudo ./examples/kni/build/app/kni -c 0x3 -n 4 -- \
-P -p 0x1 --config="(0,0,1)"

EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 0 on socket 0
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 2 lcore(s)
EAL: Probing VFIO support...
EAL:   IOMMU type 1 (Type 1) is supported
EAL:   IOMMU type 8 (No-IOMMU) is not supported
EAL: VFIO support initialized
EAL: Setting up physically contiguous memory...
...
EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using
unreliable clock cycles !
EAL: Master lcore 0 is ready (tid=657d58c0;cpuset=[0])
PMD: rte_igbvf_pmd_init():  >>
EAL: lcore 1 is ready (tid=305ff700;cpuset=[1])
EAL: PCI device 0000:00:03.0 on NUMA socket -1
EAL:   probe driver: 1af4:1000 rte_virtio_pmd
EAL:   Not managed by a supported kernel driver(0), skipped
PMD: virtio_read_caps(): failed to map pci device!
PMD: vtpci_init(): trying with legacy virtio pci.
Segmentation fault

客户机中lspci命令的输出

...
00:03.0 Ethernet controller: Red Hat, Inc Virtio network device

我注意到 pci_scan_one() 函数设置 dev->kdrv = RTE_KDRV_NONE，而驱动程序被检测为 virtio-pci（来自 /sys/bus/pci/devices/0000:00:03.0/driver）。

TAP 网络

TAP 网络仍然存在同样的问题。在主机上，我从 Wi-Fi 接口配置了一个网桥，并将其连接到 TAP 接口：

wifi_iface=wlp3s0
br_iface=br0
br_network='172.20.0.1/16'
br_dhcp_range='172.20.0.2,172.20.255.254'
tap_iface=tap0
user=ruslan

ip link add name $br_iface type bridge
ip addr add "$br_network" dev $br_iface
ip link set $br_iface up
dnsmasq --interface=$br_iface --bind-interfaces \
  --dhcp-range=$br_dhcp_range

modprobe tun
chmod 0666 /dev/net/tun

ip tuntap add dev $tap_iface mode tap user "$user"
ip link set $tap_iface up promisc on
ip link set $tap_iface master $br_iface

sysctl net.ipv4.ip_forward=1
sysctl net.ipv6.conf.default.forwarding=1
sysctl net.ipv6.conf.all.forwarding=1

iptables -t nat -A POSTROUTING -o $wifi_iface -j MASQUERADE
iptables -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i $tap_iface -o $wifi_iface -j ACCEPT

QEMU 命令：

sudo qemu-system-x86_64 -enable-kvm \
  -cpu host -smp 2 \
  -vga std \
  -mem-prealloc -mem-path /dev/hugepages \
  -drive file=GentooVM.img,if=virtio \
  -netdev tap,id=vm1_p1,ifname=tap0,script=no,downscript=no,vhost=on \
  -device virtio-net-pci,netdev=vm1_p1,bus=pci.0,addr=0x3,ioeventfd=on \
  -m 1024M \
  -monitor stdio \
  -name "Gentoo VM" \
  $@

ifconfig 在客户机中输出：

enp0s3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.20.196.253  netmask 255.255.0.0  broadcast 172.20.255.255
        inet6 fe80::59c1:f175:aeb3:433  prefixlen 64  scopeid 0x20<link>
        ether 52:54:00:12:34:56  txqueuelen 1000  (Ethernet)
        RX packets 9  bytes 1039 (1.0 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 21  bytes 1802 (1.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

以下命令失败的方式与上述 "user" 网络的情况相同：

sudo ./examples/kni/build/app/kni -c 0x3 -n 4 -- \
-P -p 0x1 --config="(0,0,1)"
...
EAL: PCI device 0000:00:03.0 on NUMA socket -1
EAL:   probe driver: 1af4:1000 rte_virtio_pmd
EAL:   Not managed by a supported kernel driver(0), skipped
PMD: virtio_read_caps(): failed to map pci device!
PMD: vtpci_init(): trying with legacy virtio pci.

问题

甚至可以运行 KNI 使用 virtio-net-pci 设备吗？

如果不可能，是否还有其他选择可以在虚拟化环境中开发 DPDK KNI 应用程序？

Answer 1

我很确定 -netdev user 与您想要的不兼容，因为它传递的是 TCP 数据而不是整个以太网帧。根据http://wiki.qemu.org/Documentation/Networking

User Networking is implemented using "slirp", which provides a full TCP/IP stack within QEMU and uses that stack to implement a virtual NAT'd network.

您希望您的选项更像 http://www.dpdk.org/doc/guides/nics/virtio.html

中指定的内容

-netdev tap,id=vm1_p1,ifname=tap0,script=no,vhost=on
-device virtio-net-pci,netdev=vm1_p1,bus=pci.0,addr=0x3,ioeventfd=on

Answer 2

virtio_read_caps() 无法映射 PCI 设备，因为 DPDK 需要 binding network ports 到以下内核模块之一：

uio_pci_generic,
igb_uio,
vfio-pci:

As of release 1.4, DPDK applications no longer automatically unbind all supported network ports from the kernel driver in use. Instead, all ports that are to be used by an DPDK application must be bound to the uio_pci_generic, igb_uio or vfio-pci module before the application is run. Any network ports under Linux* control will be ignored by the DPDK poll-mode drivers and cannot be used by the application.

因此，在运行 DPDK KNI 应用程序之前，我们应该

加载至少一个受支持的 I/O 内核模块（uio_pci_generic、igb_uio 或 vfio-pci）
加载$RTE_SDK/$RTE_TARGET/kmod/rte_kni.ko
通过$RTE_SDK/tools/dpdk_nic_bind.py脚本将网络端口绑定到内核模块。

例如：

$ sudo insmod $RTE_SDK/$RTE_TARGET/kmod/igb_uio.ko
$ sudo insmod $RTE_SDK/$RTE_TARGET/kmod/rte_kni.ko
# Obtain ID of the network port
$ python2 $RTE_SDK/tools/dpdk_nic_bind.py --status
Network devices using DPDK-compatible driver
============================================
0000:00:03.0 'Virtio network device' drv=igb_uio unused=vfio-pci,uio_pci_generic
$ sudo python2 $RTE_SDK/tools/dpdk_nic_bind.py --bind=igb_uio 0000:00:03.0

哪里

$RTE_SDK - 指向DPDK安装目录。
$RTE_TARGET - 指向DPDK目标环境目录。

在虚拟化环境中使用 DPDK 内核 NIC 接口

Using DPDK Kernel NIC Interface in a virtualized environment

linux

virtual-machine

qemu

dpdk