I am trying to determine the amount of resources required to forward 20Mp/s using DPDK. I'm using two FM10420 100G Dual NIC adapters to generate and forward traffic. Since I have only one server for testing, I'm generating packets using pktgen
on host computer and forward them with testpmd
on a virtual machine. My setup looks like this,
However, when I run both testpmd and pktgen, I can see there is huge amount of packet drop. Following are the results captured after 60 seconds of generating and forwarding packets.
Pktgen,
*** Copyright (c) <2010-2019>, Intel Corporation. All rights reserved.
*** Pktgen created by: Keith Wiles -- >>> Powered by DPDK <<<
Port: Name IfIndex Alias NUMA PCI
0: net_fm10k 0 0 8086:15a5/3b:00.1
1: net_fm10k 0 0 8086:15a5/3d:00.1
Initialize Port 0 -- TxQ 1, RxQ 1, Src MAC 06:f0:c7:c0:d7:8c
Initialize Port 1 -- TxQ 1, RxQ 1, Src MAC 8a:8c:f3:27:4b:bc
WARNING: Nothing to do on lcore 1: exiting
WARNING: Nothing to do on lcore 3: exiting
WARNING: Nothing to do on lcore 5: exiting
WARNING: Nothing to do on lcore 6: exiting
WARNING: Nothing to do on lcore 7: exiting
WARNING: Nothing to do on lcore 8: exiting
WARNING: Nothing to do on lcore 9: exiting
WARNING: Nothing to do on lcore 12: exiting
WARNING: Nothing to do on lcore 13: exiting
WARNING: Nothing to do on lcore 10: exiting
WARNING: Nothing to do on lcore 11: exiting
WARNING: Nothing to do on lcore 14: exiting
WARNING: Nothing to do on lcore 15: exiting
WARNING: Nothing to do on lcore 16: exiting
RX/TX processing lcore: 2 rx: 1 tx: 1
RX/TX processing lcore: 4 rx: 1 tx: 1
\ Ports 0-1 of 2 <Main Page> Copyright (c) <2010-2019>, Intel Corporation
Flags:Port : -------Single :0 -------Single :1
Link State : <UP-50000-FD> <UP-50000-FD> ---Total Rate---
Pkts/s Max/Rx : 8177732/0 8242150/0 16419882/0
Max/Tx : 23905867/0 23897506/0 47803373/0
MBits/s Rx/Tx : 0/0 0/0 0/0
Broadcast : 0 0
Multicast : 0 0
Sizes 64 : 475784553 480039690
65-127 : 0 0
128-255 : 0 0
256-511 : 0 0
512-1023 : 0 0
1024-1518 : 0 0
Runts/Jumbos : 0/0 0/0
ARP/ICMP Pkts : 0/0 0/0
Errors Rx/Tx : 0/0 0/0
Total Rx Pkts : 475784553 480039690
Tx Pkts : 1402606367 1403399296
Rx MBs : 319727 322586
Tx MBs : 942551 943084
:
Pattern Type : abcd... abcd...
Tx Count/% Rate : Forever /100% Forever /100%
Pkt Size/Tx Burst : 64 / 64 64 / 64
TTL/Port Src/Dest : 4/ 1234/ 5678 4/ 1234/ 5678
Pkt Type:VLAN ID : IPv4 / TCP:0001 IPv4 / TCP:0001
802.1p CoS/DSCP/IPP : 0/ 0/ 0 0/ 0/ 0
VxLAN Flg/Grp/vid : 0000/ 0/ 0 0000/ 0/ 0
IP Destination : 192.168.1.1 192.168.0.1
Source : 192.168.0.1/24 192.168.1.1/24
MAC Destination : 8a:8c:f3:27:4b:bc 06:f0:c7:c0:d7:8c
Source : 06:f0:c7:c0:d7:8c 8a:8c:f3:27:4b:bc
PCI Vendor/Addr : 8086:15a5/3b:00.1 8086:15a5/3d:00.1
-- Pktgen 19.08.0 (DPDK 19.08.2) Powered by DPDK (pid:24868) ----------------
testpmd,
testpmd> start
io packet forwarding - ports=2 - cores=8 - streams=8 - NUMA support enabled, MP allocation mode: native
Logical Core 1 (socket 0) forwards packets on 1 streams:
RX P=0/Q=0 (socket 0) -> TX P=1/Q=0 (socket 0) peer=02:00:00:00:00:01
Logical Core 2 (socket 0) forwards packets on 1 streams:
RX P=1/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00
Logical Core 3 (socket 0) forwards packets on 1 streams:
RX P=0/Q=1 (socket 0) -> TX P=1/Q=1 (socket 0) peer=02:00:00:00:00:01
Logical Core 4 (socket 0) forwards packets on 1 streams:
RX P=1/Q=1 (socket 0) -> TX P=0/Q=1 (socket 0) peer=02:00:00:00:00:00
Logical Core 5 (socket 0) forwards packets on 1 streams:
RX P=0/Q=2 (socket 0) -> TX P=1/Q=2 (socket 0) peer=02:00:00:00:00:01
Logical Core 6 (socket 0) forwards packets on 1 streams:
RX P=1/Q=2 (socket 0) -> TX P=0/Q=2 (socket 0) peer=02:00:00:00:00:00
Logical Core 7 (socket 0) forwards packets on 1 streams:
RX P=0/Q=3 (socket 0) -> TX P=1/Q=3 (socket 0) peer=02:00:00:00:00:01
Logical Core 8 (socket 0) forwards packets on 1 streams:
RX P=1/Q=3 (socket 0) -> TX P=0/Q=3 (socket 0) peer=02:00:00:00:00:00
io packet forwarding packets/burst=32
nb forwarding cores=8 - nb forwarding ports=2
port 0: RX queue number: 4 Tx queue number: 4
Rx offloads=0x0 Tx offloads=0x0
RX queue: 0
RX desc=12288 - RX free threshold=32
RX threshold registers: pthresh=8 hthresh=8 wthresh=0
RX Offloads=0x0
TX queue: 0
TX desc=12288 - TX free threshold=32
TX threshold registers: pthresh=32 hthresh=0 wthresh=0
TX offloads=0x0 - TX RS bit threshold=32
port 1: RX queue number: 4 Tx queue number: 4
Rx offloads=0x0 Tx offloads=0x0
RX queue: 0
RX desc=12288 - RX free threshold=32
RX threshold registers: pthresh=8 hthresh=8 wthresh=0
RX Offloads=0x0
TX queue: 0
TX desc=12288 - TX free threshold=32
TX threshold registers: pthresh=32 hthresh=0 wthresh=0
TX offloads=0x0 - TX RS bit threshold=32
testpmd> stop
Telling cores to stop...
Waiting for lcores to finish...
------- Forward Stats for RX Port= 1/Queue= 2 -> TX Port= 0/Queue= 2 -------
RX-packets: 658591780 TX-packets: 480381142 TX-dropped: 178210638
------- Forward Stats for RX Port= 0/Queue= 3 -> TX Port= 1/Queue= 3 -------
RX-packets: 663573664 TX-packets: 476130290 TX-dropped: 187443374
---------------------- Forward statistics for port 0 ----------------------
RX-packets: 663573664 RX-dropped: 739825632 RX-total: 1403399296
TX-packets: 480381142 TX-dropped: 178210638 TX-total: 658591780
----------------------------------------------------------------------------
---------------------- Forward statistics for port 1 ----------------------
RX-packets: 658591780 RX-dropped: 744014587 RX-total: 1402606367
TX-packets: 476130290 TX-dropped: 187443374 TX-total: 663573664
----------------------------------------------------------------------------
+++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
RX-packets: 1322165444 RX-dropped: 1483840219 RX-total: 2806005663
TX-packets: 956511432 TX-dropped: 365654012 TX-total: 1322165444
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
I use following commands to run pktgen and testpmd,
sudo ./app/x86_64-native-linuxapp-gcc/pktgen -l 0-16 -n 2 -b 0000:af:00.1 -b 0000:b1:00.1 -- P -m "2.0, 4.1
sudo ./x86_64-native-linuxapp-gcc/app/testpmd -c 0xfff -n 4 -- -i --rxq=4 --rxd=12288 --txq=4 --txd=12288 --nb-cores=8
Here are the details of my VM and host machine,
#Host Computer
CPU Model - Intel(R) Xeon(R) Silver 4214 CPU @ 2.20GHz
Memory - 24 GB (64 GB Total, 40GB allocated for VM)
Numa nodes - 2,32GB memory per each, Cores on Numa 0 are used by processes on host and Numa 1 cores are used by VM.
Cores - 24 (Hyperthreading enabled)
Hugepages - 13 x 1GB pages (All allocated on Numa 0)
Pktgen Version - 19.08.0
DPDK Version - 19.08.2
VF Driver - igb_uio
PF Driver - fm10k
PF Driver Version - 0.26.1
Iommu settings - iommu=pt intel_iommu=on
Kernel Version - 4.4.0-190-generic
#Virtual machine
Memory - 40GB
Numa nodes - 1
Cores - 24 (Hyperthreading enabled)
Hugepages - 25 x 1GB pages
DPDK Version - 19.11.5 (LTS)
VF Driver - igb_uio
Kernel Version - 3.13.0-170-generic
Can someone suggest me what could be the reason for this abnormal packet drop? Could it be a firmware problem?
Edit: Here are the screenshots of htop
Host computer (running pktgen)
VM Running testpmd
cat /proc/cmdline output of Host and VM,
#Host
BOOT_IMAGE=/vmlinuz-4.4.0-190-generic root=/dev/mapper/paraqum--vg-root ro default_hugepagesz=1G hugepagesz=1G iommu=pt intel_iommu=on
#VM
BOOT_IMAGE=/boot/vmlinuz-3.13.0-170-generic root=UUID=d419ae60-9d4c-4fe5-a4c8-6ca111e762c5 ro default_hugepagesz=1G hugepagesz=1G hugepages=25
insufficent details
yet. Waiting for your updates.cat /proc/cmdline
, not enabling pktgen range to make use of RX RSS. I will wait for your updates