dpdk testpmd 数据包转发大量数据包丢弃与 fm10420 NIC (fm10k)
dpdk testpmd packet forwarding huge amount of packet drop with fm10420 NIC (fm10k)
我正在尝试确定使用 DPDK 转发 20Mp/s 所需的资源量。我使用两个 FM10420 100G 双 NIC 适配器来生成和转发流量。由于我只有一台服务器用于测试，因此我在主机上使用 pktgen 生成数据包，并在虚拟机上使用 testpmd 转发它们。我的设置看起来像这样，
但是，当我 运行 testpmd 和 pktgen 时，我可以看到有大量的数据包丢失。以下是生成并转发数据包60秒后抓取的结果。
Pktgen,
*** Copyright (c) <2010-2019>, Intel Corporation. All rights reserved.
*** Pktgen created by: Keith Wiles -- >>> Powered by DPDK <<<
Port: Name IfIndex Alias NUMA PCI
0: net_fm10k 0 0 8086:15a5/3b:00.1
1: net_fm10k 0 0 8086:15a5/3d:00.1
Initialize Port 0 -- TxQ 1, RxQ 1, Src MAC 06:f0:c7:c0:d7:8c
Initialize Port 1 -- TxQ 1, RxQ 1, Src MAC 8a:8c:f3:27:4b:bc
WARNING: Nothing to do on lcore 1: exiting
WARNING: Nothing to do on lcore 3: exiting
WARNING: Nothing to do on lcore 5: exiting
WARNING: Nothing to do on lcore 6: exiting
WARNING: Nothing to do on lcore 7: exiting
WARNING: Nothing to do on lcore 8: exiting
WARNING: Nothing to do on lcore 9: exiting
WARNING: Nothing to do on lcore 12: exiting
WARNING: Nothing to do on lcore 13: exiting
WARNING: Nothing to do on lcore 10: exiting
WARNING: Nothing to do on lcore 11: exiting
WARNING: Nothing to do on lcore 14: exiting
WARNING: Nothing to do on lcore 15: exiting
WARNING: Nothing to do on lcore 16: exiting
RX/TX processing lcore: 2 rx: 1 tx: 1
RX/TX processing lcore: 4 rx: 1 tx: 1
\ Ports 0-1 of 2 <Main Page> Copyright (c) <2010-2019>, Intel Corporation
Flags:Port : -------Single :0 -------Single :1
Link State : <UP-50000-FD> <UP-50000-FD> ---Total Rate---
Pkts/s Max/Rx : 8177732/0 8242150/0 16419882/0
Max/Tx : 23905867/0 23897506/0 47803373/0
MBits/s Rx/Tx : 0/0 0/0 0/0
Broadcast : 0 0
Multicast : 0 0
Sizes 64 : 475784553 480039690
65-127 : 0 0
128-255 : 0 0
256-511 : 0 0
512-1023 : 0 0
1024-1518 : 0 0
Runts/Jumbos : 0/0 0/0
ARP/ICMP Pkts : 0/0 0/0
Errors Rx/Tx : 0/0 0/0
Total Rx Pkts : 475784553 480039690
Tx Pkts : 1402606367 1403399296
Rx MBs : 319727 322586
Tx MBs : 942551 943084
:
Pattern Type : abcd... abcd...
Tx Count/% Rate : Forever /100% Forever /100%
Pkt Size/Tx Burst : 64 / 64 64 / 64
TTL/Port Src/Dest : 4/ 1234/ 5678 4/ 1234/ 5678
Pkt Type:VLAN ID : IPv4 / TCP:0001 IPv4 / TCP:0001
802.1p CoS/DSCP/IPP : 0/ 0/ 0 0/ 0/ 0
VxLAN Flg/Grp/vid : 0000/ 0/ 0 0000/ 0/ 0
IP Destination : 192.168.1.1 192.168.0.1
Source : 192.168.0.1/24 192.168.1.1/24
MAC Destination : 8a:8c:f3:27:4b:bc 06:f0:c7:c0:d7:8c
Source : 06:f0:c7:c0:d7:8c 8a:8c:f3:27:4b:bc
PCI Vendor/Addr : 8086:15a5/3b:00.1 8086:15a5/3d:00.1
-- Pktgen 19.08.0 (DPDK 19.08.2) Powered by DPDK (pid:24868) ----------------
testpmd,
testpmd> start
io packet forwarding - ports=2 - cores=8 - streams=8 - NUMA support enabled, MP allocation mode: native
Logical Core 1 (socket 0) forwards packets on 1 streams:
RX P=0/Q=0 (socket 0) -> TX P=1/Q=0 (socket 0) peer=02:00:00:00:00:01
Logical Core 2 (socket 0) forwards packets on 1 streams:
RX P=1/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00
Logical Core 3 (socket 0) forwards packets on 1 streams:
RX P=0/Q=1 (socket 0) -> TX P=1/Q=1 (socket 0) peer=02:00:00:00:00:01
Logical Core 4 (socket 0) forwards packets on 1 streams:
RX P=1/Q=1 (socket 0) -> TX P=0/Q=1 (socket 0) peer=02:00:00:00:00:00
Logical Core 5 (socket 0) forwards packets on 1 streams:
RX P=0/Q=2 (socket 0) -> TX P=1/Q=2 (socket 0) peer=02:00:00:00:00:01
Logical Core 6 (socket 0) forwards packets on 1 streams:
RX P=1/Q=2 (socket 0) -> TX P=0/Q=2 (socket 0) peer=02:00:00:00:00:00
Logical Core 7 (socket 0) forwards packets on 1 streams:
RX P=0/Q=3 (socket 0) -> TX P=1/Q=3 (socket 0) peer=02:00:00:00:00:01
Logical Core 8 (socket 0) forwards packets on 1 streams:
RX P=1/Q=3 (socket 0) -> TX P=0/Q=3 (socket 0) peer=02:00:00:00:00:00
io packet forwarding packets/burst=32
nb forwarding cores=8 - nb forwarding ports=2
port 0: RX queue number: 4 Tx queue number: 4
Rx offloads=0x0 Tx offloads=0x0
RX queue: 0
RX desc=12288 - RX free threshold=32
RX threshold registers: pthresh=8 hthresh=8 wthresh=0
RX Offloads=0x0
TX queue: 0
TX desc=12288 - TX free threshold=32
TX threshold registers: pthresh=32 hthresh=0 wthresh=0
TX offloads=0x0 - TX RS bit threshold=32
port 1: RX queue number: 4 Tx queue number: 4
Rx offloads=0x0 Tx offloads=0x0
RX queue: 0
RX desc=12288 - RX free threshold=32
RX threshold registers: pthresh=8 hthresh=8 wthresh=0
RX Offloads=0x0
TX queue: 0
TX desc=12288 - TX free threshold=32
TX threshold registers: pthresh=32 hthresh=0 wthresh=0
TX offloads=0x0 - TX RS bit threshold=32
testpmd> stop
Telling cores to stop...
Waiting for lcores to finish...
------- Forward Stats for RX Port= 1/Queue= 2 -> TX Port= 0/Queue= 2 -------
RX-packets: 658591780 TX-packets: 480381142 TX-dropped: 178210638
------- Forward Stats for RX Port= 0/Queue= 3 -> TX Port= 1/Queue= 3 -------
RX-packets: 663573664 TX-packets: 476130290 TX-dropped: 187443374
---------------------- Forward statistics for port 0 ----------------------
RX-packets: 663573664 RX-dropped: 739825632 RX-total: 1403399296
TX-packets: 480381142 TX-dropped: 178210638 TX-total: 658591780
----------------------------------------------------------------------------
---------------------- Forward statistics for port 1 ----------------------
RX-packets: 658591780 RX-dropped: 744014587 RX-total: 1402606367
TX-packets: 476130290 TX-dropped: 187443374 TX-total: 663573664
----------------------------------------------------------------------------
+++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
RX-packets: 1322165444 RX-dropped: 1483840219 RX-total: 2806005663
TX-packets: 956511432 TX-dropped: 365654012 TX-total: 1322165444
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
我使用以下命令 运行 pktgen 和 testpmd，
sudo ./app/x86_64-native-linuxapp-gcc/pktgen -l 0-16 -n 2 -b 0000:af:00.1 -b 0000:b1:00.1 -- P -m "2.0, 4.1
sudo ./x86_64-native-linuxapp-gcc/app/testpmd -c 0xfff -n 4 -- -i --rxq=4 --rxd=12288 --txq=4 --txd=12288 --nb-cores=8
这是我的虚拟机和主机的详细信息，
#Host Computer
CPU Model - Intel(R) Xeon(R) Silver 4214 CPU @ 2.20GHz
Memory - 24 GB (64 GB Total, 40GB allocated for VM)
Numa nodes - 2,32GB memory per each, Cores on Numa 0 are used by processes on host and Numa 1 cores are used by VM.
Cores - 24 (Hyperthreading enabled)
Hugepages - 13 x 1GB pages (All allocated on Numa 0)
Pktgen Version - 19.08.0
DPDK Version - 19.08.2
VF Driver - igb_uio
PF Driver - fm10k
PF Driver Version - 0.26.1
Iommu settings - iommu=pt intel_iommu=on
Kernel Version - 4.4.0-190-generic
#Virtual machine
Memory - 40GB
Numa nodes - 1
Cores - 24 (Hyperthreading enabled)
Hugepages - 25 x 1GB pages
DPDK Version - 19.11.5 (LTS)
VF Driver - igb_uio
Kernel Version - 3.13.0-170-generic
谁能告诉我这个异常丢包的原因是什么？会不会是固件问题？
编辑：这里是htop的截图
主机（运行ning pktgen）
VM 运行 testpmd
cat /proc/cmdline 主机和 VM 的输出，
#Host
BOOT_IMAGE=/vmlinuz-4.4.0-190-generic root=/dev/mapper/paraqum--vg-root ro default_hugepagesz=1G hugepagesz=1G iommu=pt intel_iommu=on
#VM
BOOT_IMAGE=/boot/vmlinuz-3.13.0-170-generic root=UUID=d419ae60-9d4c-4fe5-a4c8-6ca111e762c5 ro default_hugepagesz=1G hugepagesz=1G hugepages=25
有多种因素会影响 NIC PMD 的性能。下面列出了其中一些
- cpu 核心隔离明确地使用户-space 线程成为 CPU 核心时间
的唯一用户
- 内核看门狗计时器回调减少
- 禁用透明大页面（尤其是 1GB）
- 网卡固件
- DPDK 版本
- RX-TX 的矢量代码
- PCIe 通道（直接连接到 CPU 比南桥性能更高）
- CPU时钟频率
- 网卡的DDIO能力
- 流量模式（在 RX 队列或 FLow DIrector 上使用 RSS）
- 用于防止缓存定位的 Resoruce Director。
我强烈推荐@Anuradha 检查 FM10K PMD capacity, BIOS，并使用 smap_affinity, isol_cpu, rcu_callback etc。
注意：我已经能够使用单核和 DPDK 示例 skeleton 使用 X710 NIC 实现 29 Mpps (64B) 数据包。
我正在尝试确定使用 DPDK 转发 20Mp/s 所需的资源量。我使用两个 FM10420 100G 双 NIC 适配器来生成和转发流量。由于我只有一台服务器用于测试，因此我在主机上使用 pktgen 生成数据包，并在虚拟机上使用 testpmd 转发它们。我的设置看起来像这样，
但是，当我 运行 testpmd 和 pktgen 时，我可以看到有大量的数据包丢失。以下是生成并转发数据包60秒后抓取的结果。
Pktgen,
*** Copyright (c) <2010-2019>, Intel Corporation. All rights reserved.
*** Pktgen created by: Keith Wiles -- >>> Powered by DPDK <<<
Port: Name IfIndex Alias NUMA PCI
0: net_fm10k 0 0 8086:15a5/3b:00.1
1: net_fm10k 0 0 8086:15a5/3d:00.1
Initialize Port 0 -- TxQ 1, RxQ 1, Src MAC 06:f0:c7:c0:d7:8c
Initialize Port 1 -- TxQ 1, RxQ 1, Src MAC 8a:8c:f3:27:4b:bc
WARNING: Nothing to do on lcore 1: exiting
WARNING: Nothing to do on lcore 3: exiting
WARNING: Nothing to do on lcore 5: exiting
WARNING: Nothing to do on lcore 6: exiting
WARNING: Nothing to do on lcore 7: exiting
WARNING: Nothing to do on lcore 8: exiting
WARNING: Nothing to do on lcore 9: exiting
WARNING: Nothing to do on lcore 12: exiting
WARNING: Nothing to do on lcore 13: exiting
WARNING: Nothing to do on lcore 10: exiting
WARNING: Nothing to do on lcore 11: exiting
WARNING: Nothing to do on lcore 14: exiting
WARNING: Nothing to do on lcore 15: exiting
WARNING: Nothing to do on lcore 16: exiting
RX/TX processing lcore: 2 rx: 1 tx: 1
RX/TX processing lcore: 4 rx: 1 tx: 1
\ Ports 0-1 of 2 <Main Page> Copyright (c) <2010-2019>, Intel Corporation
Flags:Port : -------Single :0 -------Single :1
Link State : <UP-50000-FD> <UP-50000-FD> ---Total Rate---
Pkts/s Max/Rx : 8177732/0 8242150/0 16419882/0
Max/Tx : 23905867/0 23897506/0 47803373/0
MBits/s Rx/Tx : 0/0 0/0 0/0
Broadcast : 0 0
Multicast : 0 0
Sizes 64 : 475784553 480039690
65-127 : 0 0
128-255 : 0 0
256-511 : 0 0
512-1023 : 0 0
1024-1518 : 0 0
Runts/Jumbos : 0/0 0/0
ARP/ICMP Pkts : 0/0 0/0
Errors Rx/Tx : 0/0 0/0
Total Rx Pkts : 475784553 480039690
Tx Pkts : 1402606367 1403399296
Rx MBs : 319727 322586
Tx MBs : 942551 943084
:
Pattern Type : abcd... abcd...
Tx Count/% Rate : Forever /100% Forever /100%
Pkt Size/Tx Burst : 64 / 64 64 / 64
TTL/Port Src/Dest : 4/ 1234/ 5678 4/ 1234/ 5678
Pkt Type:VLAN ID : IPv4 / TCP:0001 IPv4 / TCP:0001
802.1p CoS/DSCP/IPP : 0/ 0/ 0 0/ 0/ 0
VxLAN Flg/Grp/vid : 0000/ 0/ 0 0000/ 0/ 0
IP Destination : 192.168.1.1 192.168.0.1
Source : 192.168.0.1/24 192.168.1.1/24
MAC Destination : 8a:8c:f3:27:4b:bc 06:f0:c7:c0:d7:8c
Source : 06:f0:c7:c0:d7:8c 8a:8c:f3:27:4b:bc
PCI Vendor/Addr : 8086:15a5/3b:00.1 8086:15a5/3d:00.1
-- Pktgen 19.08.0 (DPDK 19.08.2) Powered by DPDK (pid:24868) ----------------
testpmd,
testpmd> start
io packet forwarding - ports=2 - cores=8 - streams=8 - NUMA support enabled, MP allocation mode: native
Logical Core 1 (socket 0) forwards packets on 1 streams:
RX P=0/Q=0 (socket 0) -> TX P=1/Q=0 (socket 0) peer=02:00:00:00:00:01
Logical Core 2 (socket 0) forwards packets on 1 streams:
RX P=1/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00
Logical Core 3 (socket 0) forwards packets on 1 streams:
RX P=0/Q=1 (socket 0) -> TX P=1/Q=1 (socket 0) peer=02:00:00:00:00:01
Logical Core 4 (socket 0) forwards packets on 1 streams:
RX P=1/Q=1 (socket 0) -> TX P=0/Q=1 (socket 0) peer=02:00:00:00:00:00
Logical Core 5 (socket 0) forwards packets on 1 streams:
RX P=0/Q=2 (socket 0) -> TX P=1/Q=2 (socket 0) peer=02:00:00:00:00:01
Logical Core 6 (socket 0) forwards packets on 1 streams:
RX P=1/Q=2 (socket 0) -> TX P=0/Q=2 (socket 0) peer=02:00:00:00:00:00
Logical Core 7 (socket 0) forwards packets on 1 streams:
RX P=0/Q=3 (socket 0) -> TX P=1/Q=3 (socket 0) peer=02:00:00:00:00:01
Logical Core 8 (socket 0) forwards packets on 1 streams:
RX P=1/Q=3 (socket 0) -> TX P=0/Q=3 (socket 0) peer=02:00:00:00:00:00
io packet forwarding packets/burst=32
nb forwarding cores=8 - nb forwarding ports=2
port 0: RX queue number: 4 Tx queue number: 4
Rx offloads=0x0 Tx offloads=0x0
RX queue: 0
RX desc=12288 - RX free threshold=32
RX threshold registers: pthresh=8 hthresh=8 wthresh=0
RX Offloads=0x0
TX queue: 0
TX desc=12288 - TX free threshold=32
TX threshold registers: pthresh=32 hthresh=0 wthresh=0
TX offloads=0x0 - TX RS bit threshold=32
port 1: RX queue number: 4 Tx queue number: 4
Rx offloads=0x0 Tx offloads=0x0
RX queue: 0
RX desc=12288 - RX free threshold=32
RX threshold registers: pthresh=8 hthresh=8 wthresh=0
RX Offloads=0x0
TX queue: 0
TX desc=12288 - TX free threshold=32
TX threshold registers: pthresh=32 hthresh=0 wthresh=0
TX offloads=0x0 - TX RS bit threshold=32
testpmd> stop
Telling cores to stop...
Waiting for lcores to finish...
------- Forward Stats for RX Port= 1/Queue= 2 -> TX Port= 0/Queue= 2 -------
RX-packets: 658591780 TX-packets: 480381142 TX-dropped: 178210638
------- Forward Stats for RX Port= 0/Queue= 3 -> TX Port= 1/Queue= 3 -------
RX-packets: 663573664 TX-packets: 476130290 TX-dropped: 187443374
---------------------- Forward statistics for port 0 ----------------------
RX-packets: 663573664 RX-dropped: 739825632 RX-total: 1403399296
TX-packets: 480381142 TX-dropped: 178210638 TX-total: 658591780
----------------------------------------------------------------------------
---------------------- Forward statistics for port 1 ----------------------
RX-packets: 658591780 RX-dropped: 744014587 RX-total: 1402606367
TX-packets: 476130290 TX-dropped: 187443374 TX-total: 663573664
----------------------------------------------------------------------------
+++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
RX-packets: 1322165444 RX-dropped: 1483840219 RX-total: 2806005663
TX-packets: 956511432 TX-dropped: 365654012 TX-total: 1322165444
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
我使用以下命令 运行 pktgen 和 testpmd，
sudo ./app/x86_64-native-linuxapp-gcc/pktgen -l 0-16 -n 2 -b 0000:af:00.1 -b 0000:b1:00.1 -- P -m "2.0, 4.1
sudo ./x86_64-native-linuxapp-gcc/app/testpmd -c 0xfff -n 4 -- -i --rxq=4 --rxd=12288 --txq=4 --txd=12288 --nb-cores=8
这是我的虚拟机和主机的详细信息，
#Host Computer
CPU Model - Intel(R) Xeon(R) Silver 4214 CPU @ 2.20GHz
Memory - 24 GB (64 GB Total, 40GB allocated for VM)
Numa nodes - 2,32GB memory per each, Cores on Numa 0 are used by processes on host and Numa 1 cores are used by VM.
Cores - 24 (Hyperthreading enabled)
Hugepages - 13 x 1GB pages (All allocated on Numa 0)
Pktgen Version - 19.08.0
DPDK Version - 19.08.2
VF Driver - igb_uio
PF Driver - fm10k
PF Driver Version - 0.26.1
Iommu settings - iommu=pt intel_iommu=on
Kernel Version - 4.4.0-190-generic
#Virtual machine
Memory - 40GB
Numa nodes - 1
Cores - 24 (Hyperthreading enabled)
Hugepages - 25 x 1GB pages
DPDK Version - 19.11.5 (LTS)
VF Driver - igb_uio
Kernel Version - 3.13.0-170-generic
谁能告诉我这个异常丢包的原因是什么？会不会是固件问题？
编辑：这里是htop的截图
主机（运行ning pktgen）
VM 运行 testpmd
cat /proc/cmdline 主机和 VM 的输出，
#Host
BOOT_IMAGE=/vmlinuz-4.4.0-190-generic root=/dev/mapper/paraqum--vg-root ro default_hugepagesz=1G hugepagesz=1G iommu=pt intel_iommu=on
#VM
BOOT_IMAGE=/boot/vmlinuz-3.13.0-170-generic root=UUID=d419ae60-9d4c-4fe5-a4c8-6ca111e762c5 ro default_hugepagesz=1G hugepagesz=1G hugepages=25
有多种因素会影响 NIC PMD 的性能。下面列出了其中一些
- cpu 核心隔离明确地使用户-space 线程成为 CPU 核心时间 的唯一用户
- 内核看门狗计时器回调减少
- 禁用透明大页面（尤其是 1GB）
- 网卡固件
- DPDK 版本
- RX-TX 的矢量代码
- PCIe 通道（直接连接到 CPU 比南桥性能更高）
- CPU时钟频率
- 网卡的DDIO能力
- 流量模式（在 RX 队列或 FLow DIrector 上使用 RSS）
- 用于防止缓存定位的 Resoruce Director。
我强烈推荐@Anuradha 检查 FM10K PMD capacity, BIOS，并使用 smap_affinity, isol_cpu, rcu_callback etc。
注意：我已经能够使用单核和 DPDK 示例 skeleton 使用 X710 NIC 实现 29 Mpps (64B) 数据包。