The connection can not close correctly for EloqSQL

[yi-xmu]

What happened?
After sysbench run, the connections can not close correctly.
#34

Steps to Reproduce

sysbench oltp_read_write --mysql_storage_engine=eloq --tables=1 --table_size=10000 --mysql-user=tpcc_test --mysql-password=tpcc123 --mysql-host=127.0.0.1 --mysql-port=3317 --mysql-db=sbtest --mysql-ignore-errors=1213,1020,1205,199 --auto_inc=true --rand-type=uniform --threads=10 --report-interval=10 --time=10 --verbosity=5 --mysql-debug=on run > sysbench_$(date +%Y%m%d_%H%M%S).log 2>&1

Environment

• OS:
• Version:
• Any other relevant details:

[yi-xmu]

sysbench log:

 mysql_real_connect(0x7ffff0014810, "127.0.0.1", "tpcc_test", "tpcc123", "sbtest", 3317, "(null)", CLIENT_MULTI_STATEMENTS)
mysql_real_connect(0x7fffe0000ea0, "127.0.0.1", "tpcc_test", "tpcc123", "sbtest", 3317, "(null)", CLIENT_MULTI_STATEMENTS)
mysql_real_connect(0x7fffe4000ea0, "127.0.0.1", "tpcc_test", "tpcc123", "sbtest", 3317, "(null)", CLIENT_MULTI_STATEMENTS)
mysql_real_connect(0x7fffd8000ea0, "127.0.0.1", "tpcc_test", "tpcc123", "sbtest", 3317, "(null)", CLIENT_MULTI_STATEMENTS)
mysql_real_connect(0x7fffe8000ea0, "127.0.0.1", "tpcc_test", "tpcc123", "sbtest", 3317, "(null)", CLIENT_MULTI_STATEMENTS)
mysql_real_connect(0x7fffc8000ea0, "127.0.0.1", "tpcc_test", "tpcc123", "sbtest", 3317, "(null)", CLIENT_MULTI_STATEMENTS)
mysql_real_connect(0x7fffcc000ea0, "127.0.0.1", "tpcc_test", "tpcc123", "sbtest", 3317, "(null)", CLIENT_MULTI_STATEMENTS)
mysql_real_connect(0x7fffc4000ea0, "127.0.0.1", "tpcc_test", "tpcc123", "sbtest", 3317, "(null)", CLIENT_MULTI_STATEMENTS)
mysql_real_connect(0x7fffc0000ea0, "127.0.0.1", "tpcc_test", "tpcc123", "sbtest", 3317, "(null)", CLIENT_MULTI_STATEMENTS)
mysql_real_connect(0x7fffd4000ea0, "127.0.0.1", "tpcc_test", "tpcc123", "sbtest", 3317, "(null)", CLIENT_MULTI_STATEMENTS)
mysql_close(0x7fffe8000ea0)
mysql_close(0x7fffcc000ea0)
mysql_close(0x7fffe0000ea0)
mysql_close(0x7fffe4000ea0)
mysql_close(0x7fffc0000ea0)
mysql_close(0x7fffc8000ea0)
mysql_close(0x7fffd4000ea0)
mysql_close(0x7ffff0014810)
mysql_close(0x7fffd8000ea0)
mysql_close(0x7fffc4000ea0)

[yi-xmu]

Run sysbench with 1 thread:

sysbench oltp_read_write --mysql_storage_engine=eloq --tables=1 --table_size=10000 --mysql-user=tpcc_test --mysql-password=tpcc123 --mysql-host=127.0.0.1 --mysql-port=3317 --mysql-db=sbtest --mysql-ignore-errors=1213,1020,1205,199 --auto_inc=true --rand-type=uniform --threads=1 --report-interval=10 --time=10 --verbosity=5 --mysql-debug=on run

The sysbench run with 11 prepare statement:

mysql_stmt_prepare(0x7ffff005d5d0, "BEGIN", 5) = 0x7ffff005d5d0
mysql_stmt_prepare(0x7ffff005da90, "COMMIT", 6) = 0x7ffff005da90
mysql_stmt_prepare(0x7ffff005df70, "SELECT c FROM sbtest1 WHERE id=?", 32) = 0x7ffff005df70
mysql_stmt_prepare(0x7ffff005ef30, "SELECT c FROM sbtest1 WHERE id BETWEEN ? AND ?", 46) = 0x7ffff005ef30
mysql_stmt_prepare(0x7ffff005ff90, "SELECT SUM(k) FROM sbtest1 WHERE id BETWEEN ? AND ?", 51) = 0x7ffff005ff90
mysql_stmt_prepare(0x7ffff0060d00, "SELECT c FROM sbtest1 WHERE id BETWEEN ? AND ? ORDER BY c", 57) = 0x7ffff0060d00
mysql_stmt_prepare(0x7ffff0061c20, "SELECT DISTINCT c FROM sbtest1 WHERE id BETWEEN ? AND ? ORDER BY c", 66) = 0x7ffff0061c20
mysql_stmt_prepare(0x7ffff0062a00, "UPDATE sbtest1 SET k=k+1 WHERE id=?", 35) = 0x7ffff0062a00
mysql_stmt_prepare(0x7ffff0063730, "UPDATE sbtest1 SET c=? WHERE id=?", 33) = 0x7ffff0063730
mysql_stmt_prepare(0x7ffff0064610, "DELETE FROM sbtest1 WHERE id=?", 30) = 0x7ffff0064610
mysql_stmt_prepare(0x7ffff0065390, "INSERT INTO sbtest1 (id, k, c, pad) VALUES (?, ?, ?, ?)", 55) = 0x7ffff0065390
...

mysql_stmt_close(0x7ffff0061c20) = 0
 mysql_stmt_close(0x7ffff005ff90) = 0
 mysql_stmt_close(0x7ffff0063730) = 0
 mysql_stmt_close(0x7ffff0065390) = 0
 mysql_stmt_close(0x7ffff005df70) = 0
 mysql_stmt_close(0x7ffff005ef30) = 0
 mysql_stmt_close(0x7ffff0062a00) = 0
 mysql_stmt_close(0x7ffff0060d00) = 0
 mysql_stmt_close(0x7ffff0064610) = 0
 mysql_stmt_close(0x7ffff005d5d0) = 0
 mysql_stmt_close(0x7ffff005da90) = 0

 mysql_close(0x7ffff0014810)

 mysql_thread_end()

But, the server only handle 2 stmt close command:

...
 info: Command on TCP/IP = 23 (Execute) for sd: 69

 info: Command on TCP/IP = 25 (Close stmt) for sd: 69

 info: Command on TCP/IP = 25 (Close stmt) for sd: 69

[yi-xmu]

The listener of the thread group will be blocked on the epoll_wait() after the thread group worker handled one Close stmt command. That is to say, the server does not get the next event.

[yi-xmu]

1. The first close stmt command message:

2. The last close stmt command message:

3. The Quit command message:

[yi-xmu]

The last read info from socket:

my_real_read: There are 64 bytes to read

>vio_read
vio_read enter: sd: 94  buf: 0x629000370218  size: 4
read_data: Memory: 0x629000370218  Bytes: (4)
05 00 00 00
vio_read exit: 4
<vio_read

>vio_read
vio_read enter: sd: 94  buf: 0x629000370218  size: 5
read_data: Memory: 0x629000370218  Bytes: (5)
19 02 00 00 00
vio_read exit: 5
<vio_read

Parse the command:

>fetch_command
info: Command on TCP/IP = 25 (Close stmt) for sd: 94
<fetch_command

The last return of the epoll wait:

>io_poll_wait
<io_poll_wait

lockfree_listener: epeer closed write direction. thread group 0, connection 0x60b0001965f0, sd 94

>lockfree_queue_put_bulk
lockfree_queue_put_bulk: thread group 0, io event count 1
<lockfree_queue_put_bulk

That is to say, the event is EPOLLRDHUP.

[yi-xmu]

The problem is that there are more than 9 bytes of data in the socket buffer, but only 9 bytes of data are read out.

The character of the EPOLLET mode in epoll is edge-triggered. This means that events are delivered to the application only when there is a change in the state of the monitored file descriptor.

Once an event is triggered (e.g., data becomes available for reading), epoll_wait will report it only once. Subsequent calls to epoll_wait will not report the same event again unless the state of the file descriptor changes back to non-ready and then becomes ready again.

[yi-xmu]

The logic in the SqlCmdCoroutine:

The vio->has_data() return false in the mode of non-buff-read.

[yi-xmu]

Modify the vio read mode:

[yi-xmu]

The read info with buffe read mode:

>my_real_read
my_real_read: There are 64 bytes to read
>vio_read_buff
enter: sd: 93  buf: 0x629001432218  size:4
>vio_read
vio_read enter: sd: 93  buf: 0x62900142d218  size: 16384
read_data: Memory: 0x62900142d218  Bytes: (104)
05 00 00 00 19 01 00 00 00 05 00 00 00 19 02 00 00 00 05 00 00 00 19 03 00 00
00 05 00 00 00 19 04 00 00 00 05 00 00 00 19 05 00 00 00 05 00 00 00 19 06 00
00 00 05 00 00 00 19 07 00 00 00 05 00 00 00 19 08 00 00 00 05 00 00 00 19 09
00 00 00 05 00 00 00 19 0A 00 00 00 05 00 00 00 19 0B 00 00 00 01 00 00 00 01
vio_read exit: 104
<vio_read
<vio_read_buff
my_real_read: after vio read,  i 0, length 4, remain 0, net->buff 0x629001432218, net->where_b 0, pos 0x62900143221c
packet_header: Memory: 0x629001432218  Bytes: (4)
05 00 00 00
my_real_read: after packet header i 0, remain 5, net->buff 0x629001432218, net->where_b 0, pos 0x629001432218

>vio_read_buff
enter: sd: 93  buf: 0x629001432218  size:5
<vio_read_buff

my_real_read: after vio read,  i 1, length 5, remain 0, net->buff 0x629001432218, net->where_b 0, pos 0x62900143221d
my_real_read: end. len 5, net->where_b 0, net->buff 0x629001432218
<my_real_read

>fetch_command
info: Command on TCP/IP = 25 (Close stmt) for sd: 93
<fetch_command

Next

>vio_buff_has_data
enter: sd: 93, read_pos: 0x62900142d221, read_end: 0x62900142d280
<vio_buff_has_data


>my_real_read
my_real_read: net compress 0, remain 4, net->buff 0x629001432218, net->where_b 0
>vio_read_buff
enter: sd: 93  buf: 0x629001432218  size:4
<vio_read_buff
my_real_read: after vio read,  i 0, length 4, remain 0, net->buff 0x629001432218, net->where_b 0, pos 0x62900143221c
packet_header: Memory: 0x629001432218  Bytes: (4)
05 00 00 00
my_real_read: after packet header i 0, remain 5, net->buff 0x629001432218, net->where_b 0, pos 0x629001432218


>vio_read_buff
enter: sd: 93  buf: 0x629001432218  size:5
<vio_read_buff

my_real_read: after vio read,  i 1, length 5, remain 0, net->buff 0x629001432218, net->where_b 0, pos 0x62900143221d
my_real_read: end. len 5, net->where_b 0, net->buff 0x629001432218
<my_real_read

>fetch_command
info: Command on TCP/IP = 25 (Close stmt) for sd: 93
<fetch_command

...

The last:

>my_real_read
>vio_read_buff
enter: sd: 93  buf: 0x629001432218  size:4
<vio_read_buff
my_real_read: after vio read,  i 0, length 4, remain 0, net->buff 0x629001432218, net->where_b 0, pos 0x62900143221c
packet_header: Memory: 0x629001432218  Bytes: (4)
01 00 00 00
my_real_read: after packet header i 0, remain 1, net->buff 0x629001432218, net->where_b 0, pos 0x629001432218

>vio_read_buff
enter: sd: 93  buf: 0x629001432218  size:1
<vio_read_buff
my_real_read: after vio read,  i 1, length 1, remain 0, net->buff 0x629001432218, net->where_b 0, pos 0x629001432219
my_real_read: end. len 1, net->where_b 0, net->buff 0x629001432218
<my_real_read

>fetch_command
info: Command on TCP/IP = 1 (Quit) for sd: 93
<fetch_command