When two or more connections compete, the proxy suffers from starvation problems

[jiridanek]

Let's run two iperf3 servers on ports 5201 and 5202

iperf3 -s & iperf3 -s --port 5202

Now, lets duplicate the listener creation function to proxy both

    create_server_socket_handler("5101", "127.0.0.1", "5201");
    create_server_socket_handler("5102", "127.0.0.1", "5202");

And lets run two iperf3 clients at the same time

$ iperf3 -c 127.0.0.1 -p 5101 -Z & iperf3 -c 127.0.0.1 -p 5102 -Z
[1] 1577298
Connecting to host 127.0.0.1, port 5102
Connecting to host 127.0.0.1, port 5101
[  5] local 127.0.0.1 port 36344 connected to 127.0.0.1 port 5102
[  5] local 127.0.0.1 port 50832 connected to 127.0.0.1 port 5101
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  3.45 GBytes  29.6 Gbits/sec    0   2.06 MBytes       
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  2.50 MBytes  21.0 Mbits/sec    0    320 KBytes       
[  5]   1.00-2.00   sec  3.58 GBytes  30.8 Gbits/sec    0   2.06 MBytes       
[  5]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec    0    320 KBytes       
[  5]   2.00-3.00   sec  3.55 GBytes  30.5 Gbits/sec    0   2.06 MBytes       
[  5]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec    0    320 KBytes       
[  5]   3.00-4.00   sec  2.61 GBytes  22.4 Gbits/sec    0   2.06 MBytes       
[  5]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec    0    320 KBytes       
[  5]   4.00-5.00   sec  2.06 GBytes  17.7 Gbits/sec    0   2.06 MBytes       
[  5]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec    0    320 KBytes       
[  5]   5.00-6.00   sec  2.07 GBytes  17.8 Gbits/sec    0   2.06 MBytes       
[  5]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec    0    320 KBytes       
[  5]   6.00-7.00   sec  2.00 GBytes  17.1 Gbits/sec    0   2.06 MBytes       
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec    0    320 KBytes       
[  5]   7.00-8.00   sec  1.76 GBytes  15.2 Gbits/sec    1   2.06 MBytes       
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0    320 KBytes       
[  5]   8.00-9.00   sec   932 MBytes  7.82 Gbits/sec    0   2.06 MBytes       
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    0    320 KBytes       
[  5]   9.00-10.00  sec   784 MBytes  6.57 Gbits/sec    0   2.06 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  22.8 GBytes  19.5 Gbits/sec    1             sender
[  5]   0.00-10.55  sec  14.0 GBytes  11.4 Gbits/sec                  receiver

iperf Done.
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0    320 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.50 MBytes  2.10 Mbits/sec    0             sender
[  5]   0.00-10.55  sec  2.50 MBytes  1.99 Mbits/sec                  receiver

This shows the EPOLL problem with starvation, which has to be resolved (at least, so the article linked below suggests) with either EPOLLEXCLUSIVE or EPOLLONESHOT.

https://idea.popcount.org/2017-02-20-epoll-is-fundamentally-broken-12/

(@gpjt I don't expect you to fix any of these issues; I only want to leave a note for future users to be aware of these problems with EPOLL. Your series of articles is extremely helpful for beginners like me and I am of course grateful.)