pktgen 找不到 Mellanox 端口
pktgen cannot find Mellanox port
设置
- Pktgen 版本 21.01.0
- DPDK 版本 20.11
- OS: ubuntu 18.04
- 网卡：Mellanox
driver: mlx5_core
version: 5.1-2.5.8
firmware-version: 16.28.2006 (MT_0000000012)
expansion-rom-version:
bus-info: 0000:03:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
问题
当我在dpdk 20.11和./dpdk-devbind -s运行dpdk-testpmd时，可以找到mlx端口
Network devices using kernel driver
===================================
0000:03:00.0 'MT27800 Family [ConnectX-5] 1017' if=ens1f0 drv=mlx5_core unused=vfio-pci
0000:03:00.1 'MT27800 Family [ConnectX-5] 1017' if=ens1f1 drv=mlx5_core unused=vfio-pci
0000:05:00.0 'I210 Gigabit Network Connection 1533' if=enp5s0 drv=igb unused=vfio-pci *Active*
0000:06:00.0 'I210 Gigabit Network Connection 1533' if=enp6s0 drv=igb unused=vfio-pci
0000:07:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=enp7s0f0 drv=ixgbe unused=vfio-pci
0000:07:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=enp7s0f1 drv=ixgbe unused=vfio-pci
$sudo build/app/dpdk-testpmd -c7 --vdev=net_pcap0,iface=eth0 --vdev=net_pcap1,iface=eth1 -- -i --nb-cores=2 --nb-ports=2 --total-num-mbufs=2048
EAL: Detected 24 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:03:00.0 (socket 0)
mlx5_pci: Size 0xFFFF is not power of 2, will be aligned to 0x10000.
EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:03:00.1 (socket 0)
mlx5_pci: Size 0xFFFF is not power of 2, will be aligned to 0x10000.
open_iface_live(): Couldn't open eth0: eth0: No such device exists (SIOCGIFHWADDR: No such device)
open_single_iface(): Couldn't open interface eth0
vdev_probe(): failed to initialize net_pcap0 device
open_iface_live(): Couldn't open eth1: eth1: No such device exists (SIOCGIFHWADDR: No such device)
open_single_iface(): Couldn't open interface eth1
vdev_probe(): failed to initialize net_pcap1 device
EAL: Bus (vdev) probe failed.
EAL: No legacy callbacks, legacy socket not created
Interactive-mode selected
testpmd: create a new mbuf pool <mb_pool_0>: n=2048, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Configuring Port 0 (socket 0)
Port 0: 98:03:9B:06:AB:34
Configuring Port 1 (socket 0)
Port 1: 98:03:9B:06:AB:35
Checking link statuses...
Done
testpmd> quit
但是当我运行pktgen时，它不起作用。
$sudo ./Builddir/app/pktgen -c 0xff -n 3 -a 0000:03:00.1 -- -p 0x1 -P -m "[1:2].0"
Copyright (c) <2010-2020>, Intel Corporation. All rights reserved. Powered by DPDK
EAL: Detected 24 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: No legacy callbacks, legacy socket not created
*** Copyright (c) <2010-2020>, Intel Corporation. All rights reserved.
*** Pktgen created by: Keith Wiles -- >>> Powered by DPDK <<<
Port: Name IfIndex Alias NUMA PCI
!PANIC!: *** Did not find any ports to use ***
PANIC in pktgen_config_ports():
*** Did not find any ports to use ***
6: [./Builddir/app/pktgen(+0x977a) [0x556f614b477a]]
5: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f1721223bf7]]
4: [./Builddir/app/pktgen(+0x9319) [0x556f614b4319]]
3: [./Builddir/app/pktgen(+0x31fa7) [0x556f614dcfa7]]
2: [/usr/local/lib/x86_64-linux-gnu/librte_eal.so.21(__rte_panic+0xc5) [0x7f172221d285]]
1: [/usr/local/lib/x86_64-linux-gnu/librte_eal.so.21(rte_dump_stack+0x2e) [0x7f172223ef2e]]
Aborted
[更新-1]
我发现系统中存在librte_net_mlx5.so：
/usr/local/lib/x86_64-linux-gnu/librte_net_mlx5.so.21.0
/usr/local/lib/x86_64-linux-gnu/librte_net_mlx5.so.21
/usr/local/lib/x86_64-linux-gnu/librte_net_mlx5.so
/usr/local/lib/x86_64-linux-gnu/dpdk/pmds-21.0/librte_net_mlx5.so.21.0
/usr/local/lib/x86_64-linux-gnu/dpdk/pmds-21.0/librte_net_mlx5.so.21
/usr/local/lib/x86_64-linux-gnu/dpdk/pmds-21.0/librte_net_mlx5.so
/usr/local/lib/x86_64-linux-gnu/librte_net_mlx5.a
/home/dpdk-20.11/build/drivers/librte_net_mlx5.so.21.0
/home/dpdk-20.11/build/drivers/librte_net_mlx5.so.21
/home/dpdk-20.11/build/drivers/librte_net_mlx5.so
/home/dpdk-20.11/build/drivers/librte_net_mlx5.a.p
/home/dpdk-20.11/build/drivers/librte_net_mlx5.so.21.0.p
/home/dpdk-20.11/build/drivers/librte_net_mlx5.a
我尝试将 -d librte_net_mlx5.so 添加为 @Vipin Varghese 的建议，并得到以下输出：
$sudo ./Builddir/app/pktgen -c 0xff -n 3 -a 0000:03:00.1 -d librte_net_mlx5.so -- -p 0x1 -P -m "[1:2].0"
EAL: Detected 24 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:03:00.1 (socket 0)
mlx5_pci: Size 0xFFFF is not power of 2, will be aligned to 0x10000.
EAL: No legacy callbacks, legacy socket not created
*** Copyright (c) <2010-2020>, Intel Corporation. All rights reserved.
*** Pktgen created by: Keith Wiles -- >>> Powered by DPDK <<<
Port: Name IfIndex Alias NUMA PCI
0: mlx5_pci 11 0 15b3:1017/03:00.1
Initialize Port 0 -- TxQ 1, RxQ 1
MBUF: error setting mempool handler
!PANIC!: Cannot create mbuf pool (Default RX 0:0) port 0, queue 0, nb_mbufs 4096, socket_id 0: Invalid argument
PANIC in pktgen_mbuf_pool_create():
Cannot create mbuf pool (Default RX 0:0) port 0, queue 0, nb_mbufs 4096, socket_id 0: Invalid argument
6: [./Builddir/app/pktgen(+0x977a) [0x55fd3145f77a]]
5: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f5479069bf7]]
4: [./Builddir/app/pktgen(+0x9319) [0x55fd3145f319]]
3: [./Builddir/app/pktgen(+0x3198b) [0x55fd3148798b]]
2: [/usr/local/lib/x86_64-linux-gnu/librte_eal.so.21(__rte_panic+0xc5) [0x7f547a063285]]
1: [/usr/local/lib/x86_64-linux-gnu/librte_eal.so.21(rte_dump_stack+0x2e) [0x7f547a084f2e]]
Aborted
显然 pktgen 效用是
- 根据日志，不是使用 Mellanox PMD mlx5 构建的
- 或者 pktgen 没有通过共享库来初始化 MLX5 PMD
由于构建使用的DPDK是DPDK 20.11版本。 pktgen 使用共享库构建的概率很高。将 eal 参数作为 -d librte_net_mlx5.so 传递应该可以解决共享库问题。
不建议静态库路径的原因是因为 testpmd 的日志显示 MLX5 被识别，而 eth0 和 eth1 是不存在的接口并跳过 PCAP PMD
EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:03:00.0 (socket 0)
mlx5_pci: Size 0xFFFF is not power of 2, will be aligned to 0x10000.
EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:03:00.1 (socket 0)
mlx5_pci: Size 0xFFFF is not power of 2, will be aligned to 0x10000.
[EDIT-1] 也要求在评论中提供适当的详细信息，none 可用。
[EDIT-2] 检查 PKTGEN 21.01 是否使用 MXL5 PMD 构建，在控制台检查
pkg-config --cflags --libs --static libdpdk | grep -i mlx5nm [pktgen-application] | grep -i mlx5
。根据更新的日志，pktgen 不是使用位于 /usr/local/lib/x86_64-linux-gnu
中的 DPDK 构建的
设置
- Pktgen 版本 21.01.0
- DPDK 版本 20.11
- OS: ubuntu 18.04
- 网卡：Mellanox
driver: mlx5_core
version: 5.1-2.5.8
firmware-version: 16.28.2006 (MT_0000000012)
expansion-rom-version:
bus-info: 0000:03:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
问题
当我在dpdk 20.11和./dpdk-devbind -s运行dpdk-testpmd时，可以找到mlx端口
Network devices using kernel driver
===================================
0000:03:00.0 'MT27800 Family [ConnectX-5] 1017' if=ens1f0 drv=mlx5_core unused=vfio-pci
0000:03:00.1 'MT27800 Family [ConnectX-5] 1017' if=ens1f1 drv=mlx5_core unused=vfio-pci
0000:05:00.0 'I210 Gigabit Network Connection 1533' if=enp5s0 drv=igb unused=vfio-pci *Active*
0000:06:00.0 'I210 Gigabit Network Connection 1533' if=enp6s0 drv=igb unused=vfio-pci
0000:07:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=enp7s0f0 drv=ixgbe unused=vfio-pci
0000:07:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=enp7s0f1 drv=ixgbe unused=vfio-pci
$sudo build/app/dpdk-testpmd -c7 --vdev=net_pcap0,iface=eth0 --vdev=net_pcap1,iface=eth1 -- -i --nb-cores=2 --nb-ports=2 --total-num-mbufs=2048
EAL: Detected 24 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:03:00.0 (socket 0)
mlx5_pci: Size 0xFFFF is not power of 2, will be aligned to 0x10000.
EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:03:00.1 (socket 0)
mlx5_pci: Size 0xFFFF is not power of 2, will be aligned to 0x10000.
open_iface_live(): Couldn't open eth0: eth0: No such device exists (SIOCGIFHWADDR: No such device)
open_single_iface(): Couldn't open interface eth0
vdev_probe(): failed to initialize net_pcap0 device
open_iface_live(): Couldn't open eth1: eth1: No such device exists (SIOCGIFHWADDR: No such device)
open_single_iface(): Couldn't open interface eth1
vdev_probe(): failed to initialize net_pcap1 device
EAL: Bus (vdev) probe failed.
EAL: No legacy callbacks, legacy socket not created
Interactive-mode selected
testpmd: create a new mbuf pool <mb_pool_0>: n=2048, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Configuring Port 0 (socket 0)
Port 0: 98:03:9B:06:AB:34
Configuring Port 1 (socket 0)
Port 1: 98:03:9B:06:AB:35
Checking link statuses...
Done
testpmd> quit
但是当我运行pktgen时，它不起作用。
$sudo ./Builddir/app/pktgen -c 0xff -n 3 -a 0000:03:00.1 -- -p 0x1 -P -m "[1:2].0"
Copyright (c) <2010-2020>, Intel Corporation. All rights reserved. Powered by DPDK
EAL: Detected 24 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: No legacy callbacks, legacy socket not created
*** Copyright (c) <2010-2020>, Intel Corporation. All rights reserved.
*** Pktgen created by: Keith Wiles -- >>> Powered by DPDK <<<
Port: Name IfIndex Alias NUMA PCI
!PANIC!: *** Did not find any ports to use ***
PANIC in pktgen_config_ports():
*** Did not find any ports to use ***
6: [./Builddir/app/pktgen(+0x977a) [0x556f614b477a]]
5: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f1721223bf7]]
4: [./Builddir/app/pktgen(+0x9319) [0x556f614b4319]]
3: [./Builddir/app/pktgen(+0x31fa7) [0x556f614dcfa7]]
2: [/usr/local/lib/x86_64-linux-gnu/librte_eal.so.21(__rte_panic+0xc5) [0x7f172221d285]]
1: [/usr/local/lib/x86_64-linux-gnu/librte_eal.so.21(rte_dump_stack+0x2e) [0x7f172223ef2e]]
Aborted
[更新-1]
我发现系统中存在librte_net_mlx5.so：
/usr/local/lib/x86_64-linux-gnu/librte_net_mlx5.so.21.0
/usr/local/lib/x86_64-linux-gnu/librte_net_mlx5.so.21
/usr/local/lib/x86_64-linux-gnu/librte_net_mlx5.so
/usr/local/lib/x86_64-linux-gnu/dpdk/pmds-21.0/librte_net_mlx5.so.21.0
/usr/local/lib/x86_64-linux-gnu/dpdk/pmds-21.0/librte_net_mlx5.so.21
/usr/local/lib/x86_64-linux-gnu/dpdk/pmds-21.0/librte_net_mlx5.so
/usr/local/lib/x86_64-linux-gnu/librte_net_mlx5.a
/home/dpdk-20.11/build/drivers/librte_net_mlx5.so.21.0
/home/dpdk-20.11/build/drivers/librte_net_mlx5.so.21
/home/dpdk-20.11/build/drivers/librte_net_mlx5.so
/home/dpdk-20.11/build/drivers/librte_net_mlx5.a.p
/home/dpdk-20.11/build/drivers/librte_net_mlx5.so.21.0.p
/home/dpdk-20.11/build/drivers/librte_net_mlx5.a
我尝试将 -d librte_net_mlx5.so 添加为 @Vipin Varghese 的建议，并得到以下输出：
$sudo ./Builddir/app/pktgen -c 0xff -n 3 -a 0000:03:00.1 -d librte_net_mlx5.so -- -p 0x1 -P -m "[1:2].0"
EAL: Detected 24 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:03:00.1 (socket 0)
mlx5_pci: Size 0xFFFF is not power of 2, will be aligned to 0x10000.
EAL: No legacy callbacks, legacy socket not created
*** Copyright (c) <2010-2020>, Intel Corporation. All rights reserved.
*** Pktgen created by: Keith Wiles -- >>> Powered by DPDK <<<
Port: Name IfIndex Alias NUMA PCI
0: mlx5_pci 11 0 15b3:1017/03:00.1
Initialize Port 0 -- TxQ 1, RxQ 1
MBUF: error setting mempool handler
!PANIC!: Cannot create mbuf pool (Default RX 0:0) port 0, queue 0, nb_mbufs 4096, socket_id 0: Invalid argument
PANIC in pktgen_mbuf_pool_create():
Cannot create mbuf pool (Default RX 0:0) port 0, queue 0, nb_mbufs 4096, socket_id 0: Invalid argument
6: [./Builddir/app/pktgen(+0x977a) [0x55fd3145f77a]]
5: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f5479069bf7]]
4: [./Builddir/app/pktgen(+0x9319) [0x55fd3145f319]]
3: [./Builddir/app/pktgen(+0x3198b) [0x55fd3148798b]]
2: [/usr/local/lib/x86_64-linux-gnu/librte_eal.so.21(__rte_panic+0xc5) [0x7f547a063285]]
1: [/usr/local/lib/x86_64-linux-gnu/librte_eal.so.21(rte_dump_stack+0x2e) [0x7f547a084f2e]]
Aborted
显然 pktgen 效用是
- 根据日志，不是使用 Mellanox PMD mlx5 构建的
- 或者 pktgen 没有通过共享库来初始化 MLX5 PMD
由于构建使用的DPDK是DPDK 20.11版本。 pktgen 使用共享库构建的概率很高。将 eal 参数作为 -d librte_net_mlx5.so 传递应该可以解决共享库问题。
不建议静态库路径的原因是因为 testpmd 的日志显示 MLX5 被识别，而 eth0 和 eth1 是不存在的接口并跳过 PCAP PMD
EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:03:00.0 (socket 0)
mlx5_pci: Size 0xFFFF is not power of 2, will be aligned to 0x10000.
EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:03:00.1 (socket 0)
mlx5_pci: Size 0xFFFF is not power of 2, will be aligned to 0x10000.
[EDIT-1] 也要求在评论中提供适当的详细信息，none 可用。
[EDIT-2] 检查 PKTGEN 21.01 是否使用 MXL5 PMD 构建，在控制台检查
pkg-config --cflags --libs --static libdpdk | grep -i mlx5nm [pktgen-application] | grep -i mlx5
。根据更新的日志，pktgen 不是使用位于 /usr/local/lib/x86_64-linux-gnu