iperf3: TCP in schneller als TCP out

[oicfar]

Hi, habe nach langer Zeit mal wieder iperf3 laufen lassen.

Die Messung läuft zwischen zwei NUCs, die am gleichen Switch (TP-Link TL-SG108) hängen.

NUC1:

• OS: Ubuntu 22.04.3 LTS x86_64
• Host: NUC11ATKC4
• Kernel: 6.2.0-39-generic
• CPU: Intel Celeron N5105 (4) @ 2.900GHz
• Memory: 32GB
• Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
  • driver=r8169 driverversion=6.2.0-39-generic duplex=full firmware=rtl8168h-2_0.0.2
• BIOS: ATJSLCPX.0042.2023.0809.1538
• iperf 3.9 (cJSON 1.7.13)

Spoiler: NUC1 eno1

Settings for eno1:
    Supported ports: [ TP     MII ]
    Supported link modes:   10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
    Supported pause frame use: Symmetric Receive-only
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
    Advertised pause frame use: Symmetric Receive-only
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                         100baseT/Half 100baseT/Full
                                         1000baseT/Full
    Link partner advertised pause frame use: Symmetric
    Link partner advertised auto-negotiation: Yes
    Link partner advertised FEC modes: Not reported
    Speed: 1000Mb/s
    Duplex: Full
    Auto-negotiation: on
    master-slave cfg: preferred slave
    master-slave status: slave
    Port: Twisted Pair
    PHYAD: 0
    Transceiver: external
    MDI-X: Unknown
    Supports Wake-on: pumbg
    Wake-on: d
    Link detected: yes

NUC2:

• OS: Ubuntu 22.04.3 LTS x86_64
• Host: NUC8i5BEH
• Kernel: 6.2.0-39-generic
• CPU: Intel i5-8259U (8) @ 3.800GHz
• Memory: 32GB
• Ethernet controller: Intel Corporation Ethernet Connection (6) I219-V (rev 30)
  • driver=e1000e driverversion=6.2.0-39-generic duplex=full firmware=0.4-4
• BIOS: BECFL357.86A.0095.2023.0918.1953
• iperf 3.9 (cJSON 1.7.13)

Spoiler: NUC2 eno1

Settings for eno1:
    Supported ports: [ TP ]
    Supported link modes:   10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
    Supported pause frame use: No
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
    Advertised pause frame use: No
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Speed: 1000Mb/s
    Duplex: Full
    Auto-negotiation: on
    Port: Twisted Pair
    PHYAD: 1
    Transceiver: internal
    MDI-X: off (auto)
    Supports Wake-on: pumbg
    Wake-on: g
        Current message level: 0x00000007 (7)
                               drv probe link
    Link detected: yes

Spoiler: ping NUC1 -> NUC2

24 packets transmitted, 24 received, 0% packet loss, time 23296ms
rtt min/avg/max/mdev = 0.288/0.987/2.206/0.730 ms

Auf beiden starte ich

iperf3 -s

UDP sieht soweit gleich aus.

Spoiler: UDP: NUC1 -> NUC2

[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-10.00  sec  1.25 MBytes  1.05 Mbits/sec  906
[  5]  10.00-20.00  sec  1.25 MBytes  1.05 Mbits/sec  905
[  5]  20.00-30.00  sec  1.25 MBytes  1.05 Mbits/sec  905
[  5]  30.00-40.00  sec  1.25 MBytes  1.05 Mbits/sec  905
[  5]  40.00-50.00  sec  1.25 MBytes  1.05 Mbits/sec  905
[  5]  50.00-60.00  sec  1.25 MBytes  1.05 Mbits/sec  906
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-60.00  sec  7.50 MBytes  1.05 Mbits/sec  0.000 ms  0/5432 (0%)  sender
[  5]   0.00-60.04  sec  7.50 MBytes  1.05 Mbits/sec  0.275 ms  0/5431 (0%)  receiver

Spoiler: UDP: NUC2 -> NUC1

[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-10.00  sec  1.25 MBytes  1.05 Mbits/sec  906
[  5]  10.00-20.00  sec  1.25 MBytes  1.05 Mbits/sec  905
[  5]  20.00-30.00  sec  1.25 MBytes  1.05 Mbits/sec  905
[  5]  30.00-40.00  sec  1.25 MBytes  1.05 Mbits/sec  905
[  5]  40.00-50.00  sec  1.25 MBytes  1.05 Mbits/sec  905
[  5]  50.00-60.00  sec  1.25 MBytes  1.05 Mbits/sec  906
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-60.00  sec  7.50 MBytes  1.05 Mbits/sec  0.000 ms  0/5432 (0%)  sender
[  5]   0.00-60.04  sec  7.50 MBytes  1.05 Mbits/sec  0.233 ms  0/5431 (0%)  receiver

Frage mich, wieso es beim TCP nicht gleich aussieht.

Spoiler: TCP: NUC1 -> NUC2

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-10.00  sec   885 MBytes   742 Mbits/sec    0    607 KBytes
[  5]  10.00-20.00  sec   882 MBytes   740 Mbits/sec    0    607 KBytes
[  5]  20.00-30.00  sec   879 MBytes   737 Mbits/sec    0    607 KBytes
[  5]  30.00-40.00  sec   879 MBytes   737 Mbits/sec    0    930 KBytes
[  5]  40.00-50.00  sec   876 MBytes   735 Mbits/sec    0    930 KBytes
[  5]  50.00-60.00  sec   878 MBytes   736 Mbits/sec    0    930 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-60.00  sec  5.15 GBytes   738 Mbits/sec    0             sender
[  5]   0.00-60.04  sec  5.15 GBytes   737 Mbits/sec                  receiver

Spoiler: TCP: NUC2 -> NUC1

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-10.00  sec  1.09 GBytes   935 Mbits/sec    0    397 KBytes
[  5]  10.00-20.00  sec  1.09 GBytes   935 Mbits/sec    0    574 KBytes
[  5]  20.00-30.00  sec  1.09 GBytes   935 Mbits/sec    0   1.24 MBytes
[  5]  30.00-40.00  sec  1.09 GBytes   934 Mbits/sec    0   1.24 MBytes
[  5]  40.00-50.00  sec  1.09 GBytes   934 Mbits/sec    0   1.24 MBytes
[  5]  50.00-60.00  sec  1.09 GBytes   934 Mbits/sec    0   1.24 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-60.00  sec  6.53 GBytes   935 Mbits/sec    0             sender
[  5]   0.00-60.04  sec  6.53 GBytes   934 Mbits/sec                  receiver

Habe noch 10GB Datei mit scp kopiert.

Spoiler: /proc/sys/net/ipv4/tcp_window_scaling

cat /proc/sys/net/ipv4/tcp_window_scaling
1

Spoiler: NUC1 -> 10GB -> NUC2

10gb.img     100%   10GB 100.3MB/s   01:42

Spoiler: NUC2 -> 10GB -> NUC1

10gb.img   100%   10GB 110.9MB/s   01:32

Das Gleiche beobachte ich von anderem NUC mit Intel Pentium Silver N6005 (4) @ 3.300GHz der an anderen Switch hängt.

Ich versuche rauszubekommen/verstehen wieso raus es nur ~200Mbits weniger läuft.

Wenn

• Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)

der Grund sein sollte, wieso es rein schnell geht.

 

[oicfar]

OK, es wird wohl am RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller liegen. Die NUCs, die den Ethernet Controller haben, performen hier nicht so gut.