Not able to run pktgen-dpdk (error: Illegal instruction)
Not able to run pktgen-dpdk (error: Illegal instruction)
我已按照以下步骤安装和 运行 pktgen-dpdk。但是我收到“非法指令”错误并且应用程序停止。
系统信息（Centos 8）
$ uname -a
Linux localhost.localdomain 4.18.0-240.22.1.el8_3.x86_64 #1 SMP Thu Apr 8 19:01:30 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
正在安装 DPDK
[local@localhost dpdk-stable-20.11.1]$ cd dpdk-stable-20.11.1/
[local@localhost dpdk-stable-20.11.1]$ meson build
[local@localhost dpdk-stable-20.11.1]$ ninja -C build
[local@localhost dpdk-stable-20.11.1]$ ninja -C build install
[local@localhost dpdk-stable-20.11.1]$ sudo ldconfig
# echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
构建 PKTGEN
[local@localhost dpdk-stable-20.11.1]$ export PKG_CONFIG_PATH=/usr/local/lib64/pkgconfig
[local@localhost pktgen-dpdk]$ cd pktgen-dpdk/
[local@localhost pktgen-dpdk]$ cat VERSION
21.01.2
[local@localhost pktgen-dpdk]$ make
[local@localhost pktgen-dpdk]$ sudo vi /etc/ld.so.conf.d/x86_64-linux-gnu.conf
edited"/usr/local/lib64"
[local@localhost pktgen-dpdk]$ export RTE_SDK=/home/local/dpdk-stable-20.11.1/
[local@localhost pktgen-dpdk]$ export RTE_TARGET=build
# yum install dnf-plugins-core
# yum config-manager --set-enabled powertools
# yum repolist
repo id repo name
appstream CentOS Linux 8 - AppStream
baseos CentOS Linux 8 - BaseOS
extras CentOS Linux 8 - Extras
powertools CentOS Linux 8 - PowerTools
正在编辑 default.cfg 文件以匹配系统设置
[local@localhost pktgen-dpdk]$ lspci | grep Eth
01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
01:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
02:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
04:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
04:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
05:00.0 Ethernet controller: Mellanox Technologies MT28908 Family [ConnectX-6]
05:00.1 Ethernet controller: Mellanox Technologies MT28908 Family [ConnectX-6]
43:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
43:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
[local@localhost pktgen-dpdk]$ vi cfg/default.cfg
'devices': (
#'03:00.0', '05:00.0', '81:00.0', '84:00.0'
'04:00.0', '04:00.1'
),
'allowlist': (
'04:00.0', '04:00.1'
#'05:00.0',
#'84:00.0',
#'03:00.0', '81:00.0'
),
'map': (
'[3:4].0',
'[5:6].1',
#'[16:17].2',
#'[18:19].3'
),
设置端口并将它们附加到 DPDK
[local@localhost pktgen-dpdk]$ ./tools/run.py -s default
>>> sdk '/home/local/dpdk-stable-20.11.1/', target 'build'
<module 'cfg' from 'cfg/default.cfg'>
Setup DPDK to run 'pktgen' application from cfg/default.cfg file
[sudo] password for local:
运行默认配置
[local@localhost pktgen-dpdk]$ ./tools/run.py default
>>> sdk '/home/local/dpdk-stable-20.11.1/', target 'build'
<module 'cfg' from 'cfg/default.cfg'>
Trying ./usr/local/bin/pktgen
sudo -E ./usr/local/bin/pktgen -l 2,3-4,5-6,16-17,18-19 -n 4 --proc-type auto --log-level 7 --file-prefix pg -a 04:00.0 -a 04:00.1 -- -v -T -P -j -m [3:4].0 -m [5:6].1 -f themes/black-yellow.theme
Copyright (c) <2010-2020>, Intel Corporation. All rights reserved. Powered by DPDK
EAL: Detected 24 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Auto-detected process type: PRIMARY
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/pg/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: Probe PCI driver: net_ixgbe (8086:10fb) device: 0000:04:00.0 (socket 0)
EAL: Ignore mapping IO port bar(2)
EAL: Probe PCI driver: net_ixgbe (8086:10fb) device: 0000:04:00.1 (socket 0)
EAL: No legacy callbacks, legacy socket not created
运行使 pktgen 直接创建非法指令错误
[local@localhost pktgen-dpdk]$ sudo -E ./usr/local/bin/pktgen -l 2,3-4,5-6,16-17,18-19 -n 4 --proc-type auto --log-level 7 --file-prefix pg -a 04:00.0 -a 04:00.1 -- -v -T -P -j -m [3:4].0 -m [5:6].1 -f themes/black-yellow.theme
Copyright (c) <2010-2020>, Intel Corporation. All rights reserved. Powered by DPDK
EAL: Detected 24 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Auto-detected process type: PRIMARY
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/pg/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: Probe PCI driver: net_ixgbe (8086:10fb) device: 0000:04:00.0 (socket 0)
EAL: Ignore mapping IO port bar(2)
EAL: Probe PCI driver: net_ixgbe (8086:10fb) device: 0000:04:00.1 (socket 0)
EAL: No legacy callbacks, legacy socket not created
Illegal instruction
我也用 dpdk-stable-19.11.8 尝试过相同的步骤，但它不起作用。
[Vipin 检查下方]
[local@localhost pktgen-dpdk]$ find ./ -name pktgen
./Builddir/app/pktgen
./usr/local/bin/pktgen
rte_memcpy is send illegal instruction 错误
强调文本
/* 获取配置结构的干净副本 */ │
rte_memcpy(&conf, &default_port_conf, sizeof(struct rte_eth_conf));
Thread 1 "pktgen" received signal SIGILL, Illegal instruction.
0x000000000045a51e in _mm256_loadu_si256 (__P=0x6b5cc0 <default_port_conf>) at /usr/lib/gcc/x86_64-redhat-linux/8/include/avxintrin.h:922
(gdb) up
#1 rte_mov32 (src=0x6b5cc0 <default_port_conf> "", dst=0x7fffffffc2f0 "0577777742[=18=]1777") at /usr/local/include/rte_memcpy.h:319
#2 rte_memcpy_generic (n=3064, src=0x6b5cc0 <default_port_conf>, dst=0x7fffffffc2f0) at /usr/local/include/rte_memcpy.h:461
#3 rte_memcpy (n=3080, src=0x6b5cc0 <default_port_conf>, dst=0x7fffffffc2f0) at /usr/local/include/rte_memcpy.h:874
#4 pktgen_config_ports () at ../app/pktgen-port-cfg.c:311
lscpu 输出：
[local@localhost pktgen-dpdk]$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 24
On-line CPU(s) list: 0-23
Thread(s) per core: 2
Core(s) per socket: 6
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 45
Model name: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
Stepping: 7
CPU MHz: 1199.646
CPU max MHz: 2500.0000
CPU min MHz: 1200.0000
BogoMIPS: 4000.03
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 15360K
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid xsaveopt dtherm ida arat pln pts md_clear flush_l1d
Intel Xeon E5-2620 是 Sandy Bridge CPU，它正式支持 AVX 而不是 AVX2。
DPDK 20.11 meson build, ninja -C build 将生成带有 AVX 指令的代码，而不是 AVX2。但是（基于实时调试）PKTGEN 强制编译器添加要插入的 AVX2，从而导致非法指令。
解决方案：在行 22
中编辑 meson.build
来自
if get_option('enable-avx2') and cc.has_argument('-mavx2')
add_project_arguments('-mavx2', language: 'c')
endif
至
if get_option('enable-avx2') and cc.has_argument('-mavx2')
add_project_arguments('-mavx', language: 'c')
endif
以上更改对您有效，但并非对所有情况都有效。
具有此更改的 enable-avx2 标志在支持 AVX2 的平台上仅使用 AVX 指令，而不使用 AVX2 指令。
当 运行 在仅支持 AVX 的平台上时，我将需要修改 meson.build 和 meson_options.txt 以启用 AVX 或 AVX2 指令。当平台支持 AVX2 时，也支持 AVX。
也许默认是 AVX，当 enable-avx2 为真时，则使用 AVX2 标志。
我已按照以下步骤安装和 运行 pktgen-dpdk。但是我收到“非法指令”错误并且应用程序停止。
系统信息（Centos 8）
$ uname -a
Linux localhost.localdomain 4.18.0-240.22.1.el8_3.x86_64 #1 SMP Thu Apr 8 19:01:30 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
正在安装 DPDK
[local@localhost dpdk-stable-20.11.1]$ cd dpdk-stable-20.11.1/
[local@localhost dpdk-stable-20.11.1]$ meson build
[local@localhost dpdk-stable-20.11.1]$ ninja -C build
[local@localhost dpdk-stable-20.11.1]$ ninja -C build install
[local@localhost dpdk-stable-20.11.1]$ sudo ldconfig
# echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
构建 PKTGEN
[local@localhost dpdk-stable-20.11.1]$ export PKG_CONFIG_PATH=/usr/local/lib64/pkgconfig
[local@localhost pktgen-dpdk]$ cd pktgen-dpdk/
[local@localhost pktgen-dpdk]$ cat VERSION
21.01.2
[local@localhost pktgen-dpdk]$ make
[local@localhost pktgen-dpdk]$ sudo vi /etc/ld.so.conf.d/x86_64-linux-gnu.conf
edited"/usr/local/lib64"
[local@localhost pktgen-dpdk]$ export RTE_SDK=/home/local/dpdk-stable-20.11.1/
[local@localhost pktgen-dpdk]$ export RTE_TARGET=build
# yum install dnf-plugins-core
# yum config-manager --set-enabled powertools
# yum repolist
repo id repo name
appstream CentOS Linux 8 - AppStream
baseos CentOS Linux 8 - BaseOS
extras CentOS Linux 8 - Extras
powertools CentOS Linux 8 - PowerTools
正在编辑 default.cfg 文件以匹配系统设置
[local@localhost pktgen-dpdk]$ lspci | grep Eth
01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
01:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
02:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
04:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
04:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
05:00.0 Ethernet controller: Mellanox Technologies MT28908 Family [ConnectX-6]
05:00.1 Ethernet controller: Mellanox Technologies MT28908 Family [ConnectX-6]
43:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
43:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
[local@localhost pktgen-dpdk]$ vi cfg/default.cfg
'devices': (
#'03:00.0', '05:00.0', '81:00.0', '84:00.0'
'04:00.0', '04:00.1'
),
'allowlist': (
'04:00.0', '04:00.1'
#'05:00.0',
#'84:00.0',
#'03:00.0', '81:00.0'
),
'map': (
'[3:4].0',
'[5:6].1',
#'[16:17].2',
#'[18:19].3'
),
设置端口并将它们附加到 DPDK
[local@localhost pktgen-dpdk]$ ./tools/run.py -s default
>>> sdk '/home/local/dpdk-stable-20.11.1/', target 'build'
<module 'cfg' from 'cfg/default.cfg'>
Setup DPDK to run 'pktgen' application from cfg/default.cfg file
[sudo] password for local:
运行默认配置
[local@localhost pktgen-dpdk]$ ./tools/run.py default
>>> sdk '/home/local/dpdk-stable-20.11.1/', target 'build'
<module 'cfg' from 'cfg/default.cfg'>
Trying ./usr/local/bin/pktgen
sudo -E ./usr/local/bin/pktgen -l 2,3-4,5-6,16-17,18-19 -n 4 --proc-type auto --log-level 7 --file-prefix pg -a 04:00.0 -a 04:00.1 -- -v -T -P -j -m [3:4].0 -m [5:6].1 -f themes/black-yellow.theme
Copyright (c) <2010-2020>, Intel Corporation. All rights reserved. Powered by DPDK
EAL: Detected 24 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Auto-detected process type: PRIMARY
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/pg/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: Probe PCI driver: net_ixgbe (8086:10fb) device: 0000:04:00.0 (socket 0)
EAL: Ignore mapping IO port bar(2)
EAL: Probe PCI driver: net_ixgbe (8086:10fb) device: 0000:04:00.1 (socket 0)
EAL: No legacy callbacks, legacy socket not created
运行使 pktgen 直接创建非法指令错误
[local@localhost pktgen-dpdk]$ sudo -E ./usr/local/bin/pktgen -l 2,3-4,5-6,16-17,18-19 -n 4 --proc-type auto --log-level 7 --file-prefix pg -a 04:00.0 -a 04:00.1 -- -v -T -P -j -m [3:4].0 -m [5:6].1 -f themes/black-yellow.theme
Copyright (c) <2010-2020>, Intel Corporation. All rights reserved. Powered by DPDK
EAL: Detected 24 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Auto-detected process type: PRIMARY
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/pg/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: using IOMMU type 1 (Type 1)
EAL: Ignore mapping IO port bar(2)
EAL: Probe PCI driver: net_ixgbe (8086:10fb) device: 0000:04:00.0 (socket 0)
EAL: Ignore mapping IO port bar(2)
EAL: Probe PCI driver: net_ixgbe (8086:10fb) device: 0000:04:00.1 (socket 0)
EAL: No legacy callbacks, legacy socket not created
Illegal instruction
我也用 dpdk-stable-19.11.8 尝试过相同的步骤，但它不起作用。
[Vipin 检查下方]
[local@localhost pktgen-dpdk]$ find ./ -name pktgen
./Builddir/app/pktgen
./usr/local/bin/pktgen
rte_memcpy is send illegal instruction 错误 强调文本 /* 获取配置结构的干净副本 */ │ rte_memcpy(&conf, &default_port_conf, sizeof(struct rte_eth_conf));
Thread 1 "pktgen" received signal SIGILL, Illegal instruction.
0x000000000045a51e in _mm256_loadu_si256 (__P=0x6b5cc0 <default_port_conf>) at /usr/lib/gcc/x86_64-redhat-linux/8/include/avxintrin.h:922
(gdb) up
#1 rte_mov32 (src=0x6b5cc0 <default_port_conf> "", dst=0x7fffffffc2f0 "0577777742[=18=]1777") at /usr/local/include/rte_memcpy.h:319
#2 rte_memcpy_generic (n=3064, src=0x6b5cc0 <default_port_conf>, dst=0x7fffffffc2f0) at /usr/local/include/rte_memcpy.h:461
#3 rte_memcpy (n=3080, src=0x6b5cc0 <default_port_conf>, dst=0x7fffffffc2f0) at /usr/local/include/rte_memcpy.h:874
#4 pktgen_config_ports () at ../app/pktgen-port-cfg.c:311
lscpu 输出：
[local@localhost pktgen-dpdk]$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 24
On-line CPU(s) list: 0-23
Thread(s) per core: 2
Core(s) per socket: 6
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 45
Model name: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
Stepping: 7
CPU MHz: 1199.646
CPU max MHz: 2500.0000
CPU min MHz: 1200.0000
BogoMIPS: 4000.03
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 15360K
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid xsaveopt dtherm ida arat pln pts md_clear flush_l1d
Intel Xeon E5-2620 是 Sandy Bridge CPU，它正式支持 AVX 而不是 AVX2。
DPDK 20.11 meson build, ninja -C build 将生成带有 AVX 指令的代码，而不是 AVX2。但是（基于实时调试）PKTGEN 强制编译器添加要插入的 AVX2，从而导致非法指令。
解决方案：在行 22
meson.build
来自
if get_option('enable-avx2') and cc.has_argument('-mavx2')
add_project_arguments('-mavx2', language: 'c')
endif
至
if get_option('enable-avx2') and cc.has_argument('-mavx2')
add_project_arguments('-mavx', language: 'c')
endif
以上更改对您有效，但并非对所有情况都有效。
具有此更改的 enable-avx2 标志在支持 AVX2 的平台上仅使用 AVX 指令，而不使用 AVX2 指令。
当 运行 在仅支持 AVX 的平台上时，我将需要修改 meson.build 和 meson_options.txt 以启用 AVX 或 AVX2 指令。当平台支持 AVX2 时，也支持 AVX。
也许默认是 AVX，当 enable-avx2 为真时，则使用 AVX2 标志。