iscsi-scst crash when use iser link

[jumpapex]

When test iscsi-scst with iser link(use user mode device), it will crash when initiator do writing.

scst version: 3.8 release
initiator and target side are all RockyLinux 8.10

initiator:
e1000 nic with IP: 192.168.1.57
Mellanox ConnectX-3 Pro with IP: 192.168.2.57
target:
e1000 nic with IP: 192.168.1.58
Mellanox ConnectX-3 Pro with IP: 192.168.2.58

use the following command to setup iser target:
insmod scst/src/scst.ko
insmod scst/src/dev_handlers/scst_user.ko
insmod iscsi-scst/kernel/iscsi-scst.ko
insmod iscsi-scst/kernel/isert-scst/isert-scst.ko
new shell run "./iscsi-scstd -f"
new shell run "./fileio_tgt disk1 /root/vdisk.5g"
echo "add_target iqn.2020-22.com.sh-tangram:647c6666e7f:tgt" > /sys/kernel/scst_tgt/targets/iscsi/mgmt
echo "create default" > /sys/kernel/scst_tgt/targets/iscsi/iqn.2020-22.com.sh-tangram:647c6666e7f:tgt/ini_groups/mgmt
echo "add iqn.1994-05.com.redhat:647c6666e7f" > /sys/kernel/scst_tgt/targets/iscsi/iqn.2020-22.com.sh-tangram:647c6666e7f:tgt/ini_groups/default/initiators/mgmt
echo "add disk1 0" > /sys/kernel/scst_tgt/targets/iscsi/iqn.2020-22.com.sh-tangram:647c6666e7f:tgt/ini_groups/default/luns/mgmt
echo 1 > /sys/kernel/scst_tgt/targets/iscsi/iqn.2020-22.com.sh-tangram:647c6666e7f:tgt/enabled
echo 1 > /sys/kernel/scst_tgt/targets/iscsi/enabled

initiator side:
iscsiadm -m discovery -I iser -t sendtargets -p 192.168.2.58
iscsiadm -m node --login -p 192.168.2.58
lsscsi -g
[0:0:0:0] disk VMware Virtual disk 2.0 /dev/sda /dev/sg0
[3:0:0:0] cd/dvd NECVMWar VMware SATA CD00 1.00 /dev/sr0 /dev/sg1
[33:0:0:0] disk SCST_USR disk1 380 /dev/sdb /dev/sg2
do writing
dd if=/dev/zero of=/dev/sdb bs=262144 count=16384
16384+0 records in
16384+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 40.2643 s, 107 MB/s

then target side crash, some messages from kernel dump vmcore-dmesg.txt

If it is needed, I could do more test or collect more information.

[ 704.492978] [4752]: iscsi-scst: iscsi_tgt_params_set:269:Target parameters set for session 10a00003d0200: QueuedCommands 32, Response timeout 90, Nop-In interval 30, Nop-In timeout 30
[ 704.496185] [4980]: scst_set_pending_UA:12984:Setting pending UA cmd 0000000032fe77e9 (tgt_dev 000000009eb57cdb, dev disk1, initiator iqn.1994-05.com.redhat:647c6666e7f)
[ 778.751747] BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
[ 778.751760] PGD 0 P4D 0
[ 778.751766] Oops: 0000 [#1] SMP PTI
[ 778.751773] CPU: 7 PID: 555 Comm: kworker/7:3 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.el8_10.x86_64 #1
[ 778.751779] Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.21100432.B64.2301110304 01/11/2023
[ 778.751783] Workqueue: isert_cq_000000002fcbc7 isert_cq_comp_work_cb [isert_scst]
[ 778.751803] RIP: 0010:dma_direct_map_sg+0x48/0x290
[ 778.751814] Code: 65 48 8b 04 25 28 00 00 00 48 89 44 24 28 31 c0 85 d2 0f 8e cb 00 00 00 49 89 ff 4d 89 c6 49 89 f5 31 db 49 c7 c4 ff ff ff ff <49> 8b 75 00 41 8b 45 08 41 8b 55 0c 48 83 e6 fc 48 2b 35 11 0f 23
[ 778.751819] RSP: 0018:ffffc074c21ebd00 EFLAGS: 00010246
[ 778.751824] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000002
[ 778.751828] RDX: 0000000000000080 RSI: 0000000000000000 RDI: ffffa0ea424c30b8
[ 778.751832] RBP: 0000000000000080 R08: 0000000000000000 R09: 0000000000000001
[ 778.751835] R10: ffffa0ea84782580 R11: 0000000000000000 R12: ffffffffffffffff
[ 778.751838] R13: 0000000000000000 R14: 0000000000000000 R15: ffffa0ea424c30b8
[ 778.751842] FS: 0000000000000000(0000) GS:ffffa0ed6fdc0000(0000) knlGS:0000000000000000
[ 778.751846] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 778.751849] CR2: 0000000000000000 CR3: 0000000417610005 CR4: 00000000003706e0
[ 778.751902] Call Trace:
[ 778.751910] ? __die_body+0x1a/0x60
[ 778.751923] ? no_context+0x1ba/0x3f0
[ 778.751930] ? __bad_area_nosemaphore+0x16c/0x1c0
[ 778.751935] ? do_page_fault+0x37/0x12d
[ 778.751940] ? page_fault+0x1e/0x30
[ 778.751951] ? dma_direct_map_sg+0x48/0x290
[ 778.751957] dma_map_sg_attrs+0x30/0x60
[ 778.751963] isert_prepare_rdma+0x8d/0x230 [isert_scst]
[ 778.751976] isert_request_data_out+0x1a/0x40 [isert_scst]
[ 778.751987] isert_pdu_rx+0xae/0xd0 [isert_scst]
[ 778.751996] isert_recv_completion_handler+0x374/0x400 [isert_scst]
[ 778.752007] isert_poll_cq+0x2d8/0x4e0 [isert_scst]
[ 778.752017] isert_cq_comp_work_cb+0x28/0xa0 [isert_scst]
[ 778.752026] process_one_work+0x1d3/0x390
[ 778.752037] ? process_one_work+0x390/0x390
[ 778.752043] worker_thread+0x30/0x390
[ 778.752050] ? process_one_work+0x390/0x390
[ 778.752057] kthread+0x134/0x150

[jumpapex]

further investigation shows that:
in isert_prepare_rdma():
when crash, op=ISER_WR_RDMA_READ isert_pdu->iscsi.sg = NULL, while isert_pdu->iscsi.sg_cnt != 0 and isert_pdu->iscsi.bufflen = 0
normal: op=ISER_WR_RDMA_WRITE

more further :)
dd command:
dd if=/dev/zero of=/dev/sdb bs=1048576 count=1 seek=1 oflag=direct

when crash occurs:
scst_user: dev_user_alloc_sg:628:Buf not alloced (ucmd 0000000025767b79, h 211, buff_cached, 1, sg_cnt 128, ubuff 0, sgv 0000000093778ae9
scst_user: dev_user_alloc_space:697:Preparing ALLOC_MEM for user space (ucmd=0000000025767b79, h=211, alloc_len 524288)
dma_map_sg_attrs will crash because "isert_pdu->iscsi.sg = NULL, while isert_pdu->iscsi.sg_cnt != 0"

at the mean time , fileio_tgt print:
do_alloc_mem:Alloc mem (cmd 211, sess_h ffff8eac4eee2d00, cdb_len 10, alloc_len 524288, queue_type 1, data_direction 1)

It seem that the issue is kernel mode does not wait user mode allocate memory, but continue to do dma_map_sg_attrs .

more :)

int isert_pdu_rx(struct iscsi_cmnd *cmnd)
{
    ...
    	cmnd_rx_start(cmnd);

	if (unlikely(!cmnd->scst_cmd)) {
		cmnd_rx_end(cmnd);
		goto out;
	}
    ...
}

should check return code of cmnd_rx_start here? if the return code = 1(means memory not allocated), should do more loop as process_read_io() do in nthread.c

BTW, set --prealloc_buffers=1 --prealloc_buffer_size=1024 for fileio_tgt does not resolve the issue.

Thanks!