How to tune the TX performance? #6
Description
I use this branch (DPDK 20.11) and my CPU is Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz.
For the FPGA design, my config.tcl in corundum is:
# Queue manager configuration
dict set params EVENT_QUEUE_OP_TABLE_SIZE "32"
dict set params TX_QUEUE_OP_TABLE_SIZE "32"
dict set params RX_QUEUE_OP_TABLE_SIZE "32"
dict set params TX_CPL_QUEUE_OP_TABLE_SIZE [dict get $params TX_QUEUE_OP_TABLE_SIZE]
dict set params RX_CPL_QUEUE_OP_TABLE_SIZE [dict get $params RX_QUEUE_OP_TABLE_SIZE]
dict set params EVENT_QUEUE_INDEX_WIDTH "6"
dict set params TX_QUEUE_INDEX_WIDTH "13"
dict set params RX_QUEUE_INDEX_WIDTH "8"
dict set params TX_CPL_QUEUE_INDEX_WIDTH [dict get $params TX_QUEUE_INDEX_WIDTH]
dict set params RX_CPL_QUEUE_INDEX_WIDTH [dict get $params RX_QUEUE_INDEX_WIDTH]
dict set params EVENT_QUEUE_PIPELINE "3"
dict set params TX_QUEUE_PIPELINE [expr 3+([dict get $params TX_QUEUE_INDEX_WIDTH] > 12 ? [dict get $params TX_QUEUE_INDEX_WIDTH]-12 : 0)]
dict set params RX_QUEUE_PIPELINE [expr 3+([dict get $params RX_QUEUE_INDEX_WIDTH] > 12 ? [dict get $params RX_QUEUE_INDEX_WIDTH]-12 : 0)]
dict set params TX_CPL_QUEUE_PIPELINE [dict get $params TX_QUEUE_PIPELINE]
dict set params RX_CPL_QUEUE_PIPELINE [dict get $params RX_QUEUE_PIPELINE]
# TX and RX engine configuration
dict set params TX_DESC_TABLE_SIZE "32"
dict set params RX_DESC_TABLE_SIZE "32"
# Scheduler configuration
dict set params TX_SCHEDULER_OP_TABLE_SIZE [dict get $params TX_DESC_TABLE_SIZE]
dict set params TX_SCHEDULER_PIPELINE [dict get $params TX_QUEUE_PIPELINE]
dict set params TDMA_INDEX_WIDTH "6"
# Interface configuration
dict set params PTP_TS_ENABLE "0"
dict set params TX_CPL_FIFO_DEPTH "32"
dict set params TX_CHECKSUM_ENABLE "1"
dict set params RX_RSS_ENABLE "1"
dict set params RX_HASH_ENABLE "1"
dict set params RX_CHECKSUM_ENABLE "1"
dict set params TX_FIFO_DEPTH "32768"
dict set params RX_FIFO_DEPTH "131072"
dict set params MAX_TX_SIZE "9214"
dict set params MAX_RX_SIZE "9214"
dict set params TX_RAM_SIZE "131072"
dict set params RX_RAM_SIZE "131072"
My U250 NIC is attached with a Mellanox CX6-Dx NIC. I use pktgen on CX6-Dx to generate traffic and use U250 corundum to receive traffic. I assign only one single core for Rx side and the MTU is 1500. The single-core Rx performance is around 93Gbps.
- Ports 0-1 of 2 <Main Page> Copyright(c) <2010-2021>, Intel Corporation
Flags:Port : -------Sngl :0
Link State : <UP-100000-FD> ---Total Rate---
Pkts/s Rx : 7,694,144 7,694,144
Tx : 0 0
MBits/s Rx/Tx : 93,560/0 93,560/0
Pkts/s Rx Max : 7,713,280 7,713,280
Tx Max : 0 0
Broadcast : 0
Multicast : 0
Sizes 64 : 0
65-127 : 0
128-255 : 0
256-511 : 0
512-1023 : 0
1024-1518 : 188,001,803
Runts/Jumbos : 0/0
ARP/ICMP Pkts : 0/0
Errors Rx/Tx : 0/0
Total Rx Pkts : 183,882,923
Tx Pkts : 0
Rx/Tx MBs : 2,236,016/0
Pattern Type : abcd...
Tx Count/% Rate : Forever /100%
Pkt Size/Tx Burst : 64 / 32
TTL/Port Src/Dest : 64/ 1234/ 5678
Pkt Type:VLAN ID : IPv4 / TCP:0001
802.1p CoS/DSCP/IPP : 0/ 0/ 0
VxLAN Flg/Grp/vid : 0000/ 0/ 0
IP Destination : 192.168.1.1
Source : 192.168.0.1/24
MAC Destination : 00:00:00:00:00:00
Source : 67:c6:69:73:51:ff
PCI Vendor/Addr : 1234:1001/d9:00.0
-- Pktgen 21.03.1 (DPDK 20.11.10) Powered by DPDK (pid:623744) --------------
However, on the Tx size, the single-core is quite poor, which is only around 31Gbps
- Ports 0-1 of 2 <Main Page> Copyright(c) <2010-2021>, Intel Corporation
Flags:Port : -------Sngl :0
Link State : <UP-100000-FD> ---Total Rate---
Pkts/s Rx : 0 0
Tx : 2,584,000 2,584,000
MBits/s Rx/Tx : 0/31,421 0/31,421
Pkts/s Rx Max : 0 0
Tx Max : 2,610,336 2,610,336
Broadcast : 0
Multicast : 0
Sizes 64 : 0
65-127 : 0
128-255 : 0
256-511 : 0
512-1023 : 0
1024-1518 : 0
Runts/Jumbos : 0/0
ARP/ICMP Pkts : 0/0
Errors Rx/Tx : 0/0
Total Rx Pkts : 0
Tx Pkts : 32,831,072
Rx/Tx MBs : 0/399,225
Pattern Type : abcd...
Tx Count/% Rate : Forever /100%
Pkt Size/Tx Burst : 1500 / 32
TTL/Port Src/Dest : 64/ 1234/ 5678
Pkt Type:VLAN ID : IPv4 / TCP:0001
802.1p CoS/DSCP/IPP : 0/ 0/ 0
VxLAN Flg/Grp/vid : 0000/ 0/ 0
IP Destination : 192.168.1.1
Source : 192.168.0.1/24
MAC Destination : 00:00:00:00:00:00
Source : 67:c6:69:73:51:ff
PCI Vendor/Addr : 1234:1001/d9:00.0
-- Pktgen 21.03.1 (DPDK 20.11.10) Powered by DPDK (pid:624726) --------------
Is there any way or trick to tune or choose proper hardware design parameter for better Tx performance? Thx!