Problems with processing udp packets at high-rate with embedded ethernet controller ?

[arnaudf]

Hello,

The issue I describe below begins with a simple 1GB/s speed test with iperf3 (using tcp, then udp). With tcp everything is almost ok. With udp it's an other story.
I first try through Internet and notice that there is connection drops. The network topology is : test computer => internet => router => switch (Mikrotik crs-326) => Rockpro64

The iperf3 server is on Rockpro64, and the client is on test computer.

I then try to narrow down :

- I connect directly to the switch and try again : in and out switch ports show near max speed (~1 Gbps), however the iperf3 server logs show 2 or 3 transfers at reduced speed, then show no transfer/throughput after
Topology : test computer => switch => Rockpro64 (ethernet)

- I then connect an usb-c network adapter on Rockpro64 : all is working correctly at around 350MBits throughput (seems to be maximum speed with usb)
Topology : test computer => Rockpro64 (usb adapter)

 

It looks like the Rockpro64 has problems with processing udp packets at high-rate with its embedded ethernet controller.

 

I try to enable ethernet flow control on switch, and check it's enabled on Rockpro64 : no change (pause frames are not even sent).
I increase tx/rx ring buffers to the maximum (1024) : nothing changes as well.
I try to increase kernel net.core.rmem_* buffers to 25M : no changes

 

Is there known issues about this ?
Is there an alternate/improved driver for the nic ?
Have you some advices about extra debug steps ?

 

Regards.

 

 

E:\utils\iperf3\iperf-3.1.3-win64>iperf3 -V -c 192.168.0.64 -u -b 0 --get-server-output
iperf 3.1.3
CYGWIN_NT-10.0 ARNAUD-PC 2.5.1(0.297/5/3) 2016-04-21 22:14 x86_64
Time: Sat, 19 Aug 2023 09:16:46 GMT
Connecting to host 192.168.0.64, port 5201
      Cookie: ARNAUD-PC.1692436606.019644.1e8c9a9b
[  4] local 192.168.0.13 port 64985 connected to 192.168.0.64 port 5201
Starting Test: protocol: UDP, 1 streams, 8192 byte blocks, omitting 0 seconds, 10 second test
[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  4]   0.00-1.00   sec   115 MBytes   962 Mbits/sec  14690
[  4]   1.00-2.00   sec   114 MBytes   958 Mbits/sec  14630
[  4]   2.00-3.00   sec   114 MBytes   953 Mbits/sec  14530
[  4]   3.00-4.00   sec   109 MBytes   918 Mbits/sec  14010
[  4]   4.00-5.00   sec   114 MBytes   956 Mbits/sec  14590
[  4]   5.00-6.00   sec   114 MBytes   958 Mbits/sec  14620
[  4]   6.00-7.00   sec   114 MBytes   958 Mbits/sec  14620
[  4]   7.00-8.00   sec   114 MBytes   958 Mbits/sec  14630
[  4]   8.00-9.00   sec   114 MBytes   958 Mbits/sec  14620
[  4]   9.00-10.00  sec   114 MBytes   958 Mbits/sec  14620
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-10.00  sec  1.11 GBytes   954 Mbits/sec  0.074 ms  521/20469 (2.5%)
[  4] Sent 20469 datagrams
CPU Utilization: local/sender 0.8% (0.0%u/0.8%s), remote/receiver 2.9% (0.2%u/2.7%s)

Server output:
Time: Sat, 19 Aug 2023 09:16:46 GMT
Accepted connection from 192.168.0.13, port 44317
      Cookie: ARNAUD-PC.1692436606.019644.1e8c9a9b
[  5] local 192.168.0.64 port 5201 connected to 192.168.0.13 port 64985
Starting Test: protocol: UDP, 1 streams, 8192 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-1.00   sec   106 MBytes   891 Mbits/sec  0.082 ms  347/13937 (2.5%)
[  5]   1.00-2.00   sec  49.7 MBytes   416 Mbits/sec  0.074 ms  174/6532 (2.7%)
[  5]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec  0.074 ms  0/0 (0%)
[  5]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec  0.074 ms  0/0 (0%)
[  5]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec  0.074 ms  0/0 (0%)
[  5]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec  0.074 ms  0/0 (0%)
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec  0.074 ms  0/0 (0%)
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec  0.074 ms  0/0 (0%)
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec  0.074 ms  0/0 (0%)
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec  0.074 ms  0/0 (0%)
[  5]  10.00-10.26  sec  0.00 Bytes  0.00 bits/sec  0.074 ms  0/0 (0%)
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5] (sender statistics not available)
[  5]   0.00-10.26  sec   156 MBytes   127 Mbits/sec  0.074 ms  521/20469 (2.5%)  receiver


iperf Done

.

Edited August 19, 2023 by arnaudf
iperf detail