0

I am trying to determine the amount of resources required to forward 20Mp/s using DPDK. I'm using two FM10420 100G Dual NIC adapters to generate and forward traffic. Since I have only one server for testing, I'm generating packets using pktgen on host computer and forward them with testpmd on a virtual machine. My setup looks like this, test setup

However, when I run both testpmd and pktgen, I can see there is huge amount of packet drop. Following are the results captured after 60 seconds of generating and forwarding packets.

Pktgen,

    *** Copyright (c) <2010-2019>, Intel Corporation. All rights reserved.
*** Pktgen created by: Keith Wiles -- >>> Powered by DPDK <<<

 Port: Name         IfIndex Alias        NUMA  PCI
    0: net_fm10k       0                   0   8086:15a5/3b:00.1
    1: net_fm10k       0                   0   8086:15a5/3d:00.1

Initialize Port 0 -- TxQ 1, RxQ 1, Src MAC 06:f0:c7:c0:d7:8c
Initialize Port 1 -- TxQ 1, RxQ 1, Src MAC 8a:8c:f3:27:4b:bc


WARNING: Nothing to do on lcore 1: exiting
WARNING: Nothing to do on lcore 3: exiting
WARNING: Nothing to do on lcore 5: exiting
WARNING: Nothing to do on lcore 6: exiting
WARNING: Nothing to do on lcore 7: exiting
WARNING: Nothing to do on lcore 8: exiting
WARNING: Nothing to do on lcore 9: exiting
WARNING: Nothing to do on lcore 12: exiting
WARNING: Nothing to do on lcore 13: exiting
WARNING: Nothing to do on lcore 10: exiting
WARNING: Nothing to do on lcore 11: exiting
WARNING: Nothing to do on lcore 14: exiting
WARNING: Nothing to do on lcore 15: exiting
WARNING: Nothing to do on lcore 16: exiting
  RX/TX processing lcore:   2 rx:  1 tx:  1
  RX/TX processing lcore:   4 rx:  1 tx:  1

\ Ports 0-1 of 2   <Main Page>  Copyright (c) <2010-2019>, Intel Corporation
  Flags:Port        : -------Single      :0 -------Single      :1
Link State          :         <UP-50000-FD>         <UP-50000-FD>      ---Total Rate---
Pkts/s Max/Rx       :             8177732/0             8242150/0            16419882/0
       Max/Tx       :            23905867/0            23897506/0            47803373/0
MBits/s Rx/Tx       :                   0/0                   0/0                   0/0
Broadcast           :                     0                     0
Multicast           :                     0                     0
Sizes 64            :             475784553             480039690
      65-127        :                     0                     0
      128-255       :                     0                     0
      256-511       :                     0                     0
      512-1023      :                     0                     0
      1024-1518     :                     0                     0
Runts/Jumbos        :                   0/0                   0/0
ARP/ICMP Pkts       :                   0/0                   0/0
Errors Rx/Tx        :                   0/0                   0/0
Total Rx Pkts       :             475784553             480039690
      Tx Pkts       :            1402606367            1403399296
      Rx MBs        :                319727                322586
      Tx MBs        :                942551                943084
                    :
Pattern Type        :               abcd...               abcd...
Tx Count/% Rate     :         Forever /100%         Forever /100%
Pkt Size/Tx Burst   :             64 /   64             64 /   64
TTL/Port Src/Dest   :         4/ 1234/ 5678         4/ 1234/ 5678
Pkt Type:VLAN ID    :       IPv4 / TCP:0001       IPv4 / TCP:0001
802.1p CoS/DSCP/IPP :             0/  0/  0             0/  0/  0
VxLAN Flg/Grp/vid   :      0000/    0/    0      0000/    0/    0
IP  Destination     :           192.168.1.1           192.168.0.1
    Source          :        192.168.0.1/24        192.168.1.1/24
MAC Destination     :     8a:8c:f3:27:4b:bc     06:f0:c7:c0:d7:8c
    Source          :     06:f0:c7:c0:d7:8c     8a:8c:f3:27:4b:bc
PCI Vendor/Addr     :     8086:15a5/3b:00.1     8086:15a5/3d:00.1

-- Pktgen 19.08.0 (DPDK 19.08.2)  Powered by DPDK  (pid:24868) ----------------

testpmd,

testpmd> start
io packet forwarding - ports=2 - cores=8 - streams=8 - NUMA support enabled, MP allocation mode: native
Logical Core 1 (socket 0) forwards packets on 1 streams:
  RX P=0/Q=0 (socket 0) -> TX P=1/Q=0 (socket 0) peer=02:00:00:00:00:01
Logical Core 2 (socket 0) forwards packets on 1 streams:
  RX P=1/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00
Logical Core 3 (socket 0) forwards packets on 1 streams:
  RX P=0/Q=1 (socket 0) -> TX P=1/Q=1 (socket 0) peer=02:00:00:00:00:01
Logical Core 4 (socket 0) forwards packets on 1 streams:
  RX P=1/Q=1 (socket 0) -> TX P=0/Q=1 (socket 0) peer=02:00:00:00:00:00
Logical Core 5 (socket 0) forwards packets on 1 streams:
  RX P=0/Q=2 (socket 0) -> TX P=1/Q=2 (socket 0) peer=02:00:00:00:00:01
Logical Core 6 (socket 0) forwards packets on 1 streams:
  RX P=1/Q=2 (socket 0) -> TX P=0/Q=2 (socket 0) peer=02:00:00:00:00:00
Logical Core 7 (socket 0) forwards packets on 1 streams:
  RX P=0/Q=3 (socket 0) -> TX P=1/Q=3 (socket 0) peer=02:00:00:00:00:01
Logical Core 8 (socket 0) forwards packets on 1 streams:
  RX P=1/Q=3 (socket 0) -> TX P=0/Q=3 (socket 0) peer=02:00:00:00:00:00

  io packet forwarding packets/burst=32
  nb forwarding cores=8 - nb forwarding ports=2
  port 0: RX queue number: 4 Tx queue number: 4
    Rx offloads=0x0 Tx offloads=0x0
    RX queue: 0
      RX desc=12288 - RX free threshold=32
      RX threshold registers: pthresh=8 hthresh=8  wthresh=0
      RX Offloads=0x0
    TX queue: 0
      TX desc=12288 - TX free threshold=32
      TX threshold registers: pthresh=32 hthresh=0  wthresh=0
      TX offloads=0x0 - TX RS bit threshold=32
  port 1: RX queue number: 4 Tx queue number: 4
    Rx offloads=0x0 Tx offloads=0x0
    RX queue: 0
      RX desc=12288 - RX free threshold=32
      RX threshold registers: pthresh=8 hthresh=8  wthresh=0
      RX Offloads=0x0
    TX queue: 0
      TX desc=12288 - TX free threshold=32
      TX threshold registers: pthresh=32 hthresh=0  wthresh=0
      TX offloads=0x0 - TX RS bit threshold=32
testpmd> stop
Telling cores to stop...
Waiting for lcores to finish...

  ------- Forward Stats for RX Port= 1/Queue= 2 -> TX Port= 0/Queue= 2 -------
  RX-packets: 658591780      TX-packets: 480381142      TX-dropped: 178210638     

  ------- Forward Stats for RX Port= 0/Queue= 3 -> TX Port= 1/Queue= 3 -------
  RX-packets: 663573664      TX-packets: 476130290      TX-dropped: 187443374     

  ---------------------- Forward statistics for port 0  ----------------------
  RX-packets: 663573664      RX-dropped: 739825632     RX-total: 1403399296
  TX-packets: 480381142      TX-dropped: 178210638     TX-total: 658591780
  ----------------------------------------------------------------------------

  ---------------------- Forward statistics for port 1  ----------------------
  RX-packets: 658591780      RX-dropped: 744014587     RX-total: 1402606367
  TX-packets: 476130290      TX-dropped: 187443374     TX-total: 663573664
  ----------------------------------------------------------------------------

  +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
  RX-packets: 1322165444     RX-dropped: 1483840219    RX-total: 2806005663
  TX-packets: 956511432      TX-dropped: 365654012     TX-total: 1322165444
  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

I use following commands to run pktgen and testpmd,

sudo ./app/x86_64-native-linuxapp-gcc/pktgen -l 0-16 -n 2 -b 0000:af:00.1 -b 0000:b1:00.1 -- P -m "2.0, 4.1

sudo ./x86_64-native-linuxapp-gcc/app/testpmd -c 0xfff -n 4 -- -i --rxq=4 --rxd=12288 --txq=4 --txd=12288 --nb-cores=8

Here are the details of my VM and host machine,

#Host Computer
CPU Model - Intel(R) Xeon(R) Silver 4214 CPU @ 2.20GHz
Memory - 24 GB (64 GB Total, 40GB allocated for VM)
Numa nodes - 2,32GB memory per each, Cores on Numa 0 are used by processes on host and Numa 1 cores are used by VM.
Cores - 24 (Hyperthreading enabled) 
Hugepages - 13 x 1GB pages (All allocated on Numa 0)
Pktgen Version -  19.08.0 
DPDK Version - 19.08.2
VF Driver - igb_uio
PF Driver - fm10k
PF Driver Version - 0.26.1 
Iommu settings - iommu=pt intel_iommu=on
Kernel Version - 4.4.0-190-generic

#Virtual machine
Memory - 40GB
Numa nodes - 1
Cores - 24 (Hyperthreading enabled) 
Hugepages - 25 x 1GB pages
DPDK Version - 19.11.5 (LTS)
VF Driver - igb_uio
Kernel Version - 3.13.0-170-generic

Can someone suggest me what could be the reason for this abnormal packet drop? Could it be a firmware problem?

Edit: Here are the screenshots of htop

Host computer (running pktgen)

pktgenhtop

VM Running testpmd

testpmd

cat /proc/cmdline output of Host and VM,

#Host
BOOT_IMAGE=/vmlinuz-4.4.0-190-generic root=/dev/mapper/paraqum--vg-root ro default_hugepagesz=1G hugepagesz=1G iommu=pt intel_iommu=on

#VM
BOOT_IMAGE=/boot/vmlinuz-3.13.0-170-generic root=UUID=d419ae60-9d4c-4fe5-a4c8-6ca111e762c5 ro default_hugepagesz=1G hugepagesz=1G hugepages=25
5
  • there is insufficient information. So please start off in filing up details like Processor, Hyper thread enabled, Huge page size, Kernel boot parameters, htop snap shot etc. I will not mark this ticket as insufficent details yet. Waiting for your updates. Dec 16, 2020 at 12:02
  • @VipinVarghese I have included the htop screenshots and CPU model. Other parameters are already in the details section.
    – Anuradha
    Dec 16, 2020 at 13:28
  • thanks, but you have not shared cat /proc/cmdline, not enabling pktgen range to make use of RX RSS. I will wait for your updates Dec 16, 2020 at 13:31
  • @VipinVarghese thanks for the reply. I have updated with required information.
    – Anuradha
    Dec 16, 2020 at 13:34
  • I have pointed out the cause for performance, Please adjust the various factors on your system to achieve > 20Mpps using single core. Please accept and upvote too, Dec 16, 2020 at 14:07

1 Answer 1

1

There are multiple factors which affect performance for NIC PMD. Some of them are listed below

  1. cpu core isolation to explicitly make user-space threads to sole user of CPU core time
  2. Kernel watchdog timer callback reduction
  3. disable Transparent Huge page (especially with 1GB)
  4. firmware of NIC
  5. DPDK version
  6. vector code for RX-TX
  7. PCIe lane (direct attach to CPU give higher performance than south bridge)
  8. CPU clock frequency
  9. DDIO ability of NIC
  10. Traffic pattern (with RSS on RX-queue or FLow DIrector)
  11. Resoruce Director for preventing cache posioning.

I highly recommend @Anuradha to check FM10K PMD capacity, BIOS, and using smap_affinity, isol_cpu, rcu_callback etc.

Note: I have been able to achieve 29 Mpps (64B) packets using single core and DPDK example skeleton with X710 NIC.

2
  • Thanks for the answer! I'll try to check above parameters and sort out the problem.
    – Anuradha
    Dec 16, 2020 at 14:09
  • thanks the 29Mpps is achieved with 1Gb hugepage, isol, smp_affinity changes. Dec 16, 2020 at 14:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.