Skip to content

qa/iscsi_test: test also user:rbd (tcmu-runner) backstore #699

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tserong merged 2 commits into from
Jun 26, 2023
Merged

qa/iscsi_test: test also user:rbd (tcmu-runner) backstore #699

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
edeee51
qa/iscsi_test: test also user:rbd (tcmu-runner) backstore
trociny Jun 19, 2023
ca92fa0
qa/iscsi_test: test user:rbd backstore only on sles
trociny Jun 24, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
121 changes: 76 additions & 45 deletions qa/common/common.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1418,7 +1418,7 @@ function core_dump_test {

function _iscsi_test {
local client gw_name gw_ip gwcli name node rbd_pool rbd_image target used
local username password
local supported_backstores backstore i username password

set -e
set -x
Expand All @@ -1439,7 +1439,6 @@ function _iscsi_test {
client=$(sed -ne 's/^InitiatorName=//p' /etc/iscsi/initiatorname.iscsi)
target=iqn.2023-01.com.suse.iscsi-gw:iscsi-igw
rbd_pool=rbd
rbd_image=iscsi_disk_1
node=$(ceph orch ls --service-type iscsi --format json-pretty |
jq -r '.[].placement.hosts[0]')
gw_name=$(getent hosts ${node} | awk '{print $2}')
Expand All @@ -1450,62 +1449,94 @@ function _iscsi_test {
jq -r '.[] | select(.name | startswith("iscsi.")) | .name')
test -n "${name}"

ssh ${node} rbd showmapped | grep "${rbd_pool} *${rbd_image}" && false
rbd info ${rbd_pool}/${rbd_image} && false

gwcli="ssh ${node} cephadm enter --name "${name}" gwcli"

${gwcli} /iscsi-targets create ${target}
${gwcli} /iscsi-targets/${target}/gateways create ${gw_name} ${gw_ip} skipchecks=true
${gwcli} /disks create pool=${rbd_pool} image=${rbd_image} size=1G
${gwcli} /iscsi-targets/${target}/hosts create ${client}
${gwcli} /iscsi-targets/${target}/hosts/${client} auth username=${username} password=${password}
${gwcli} /iscsi-targets/${target}/hosts/${client} disk add ${rbd_pool}/${rbd_image}
if [ ${ID} = sles ]; then
supported_backstores='rbd user:rbd'
else
supported_backstores='rbd'
fi

ssh ${node} rbd showmapped | grep "${rbd_pool} *${rbd_image}"
for backstore in ${supported_backstores}; do
rbd_image=iscsi_disk_${backstore%:*}
ssh ${node} rbd showmapped | grep "${rbd_pool} *${rbd_image}" && false
rbd info ${rbd_pool}/${rbd_image} && false
gwcli="ssh ${node} cephadm enter --name "${name}" gwcli"

${gwcli} /iscsi-targets create ${target}
${gwcli} /iscsi-targets/${target}/gateways create ${gw_name} ${gw_ip} skipchecks=true
${gwcli} /disks create pool=${rbd_pool} image=${rbd_image} size=1G backstore=${backstore}
${gwcli} /iscsi-targets/${target}/hosts create ${client}
${gwcli} /iscsi-targets/${target}/hosts/${client} auth username=${username} password=${password}
${gwcli} /iscsi-targets/${target}/hosts/${client} disk add ${rbd_pool}/${rbd_image}

rbd info ${rbd_pool}/${rbd_image}

case ${backstore} in
rbd)
ssh ${node} rbd showmapped | grep "${rbd_pool} *${rbd_image}"
;;
user:rbd)
ssh ${node} rbd showmapped | grep "${rbd_pool} *${rbd_image}" && false
Copy link
Member

@tserong tserong Jun 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why && false here? I realise that construct is also present in a couple other places in _iscsi_test(), but looking at it I feel like that should cause those lines to just always fail and abort the script. What am I missing?

Copy link
Contributor Author

@trociny trociny Jun 22, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the delayed response, my sesdev github notifications seem to be filtered-out some how, will need to check the filters.

Yes, the idea to check that it is not present in rbd showmapped output. We do this several times, e.g. at the beginning of the test to check the state clean and at the end of the test. But in this particular case (backstore=user:rbd) we check that is not in the list, because rbd map is used only for 'rbd' (kernel) backstore. And this check is to test that the 'backstore' setting is not ignored and is actually applied, and the default 'rbd' backstore is not used. It is not strictly necessary, because it would fail on the next check anyway (where we test that is is indeed tcmu), on the other hand the cause would not be so clear. Anyway, it is not mandatory and may safely remove it if you think it is confusing or unnecessary.

Copy link
Contributor Author

@trociny trociny Jun 22, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, actually now I understand what you mean. It looks like it should fail for both case (i.e. if the image is present in output or not). Hm, interesting why the test did not fail for me. I will need to recheck this

Copy link
Contributor Author

@trociny trociny Jun 22, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just checked, and it works as it was intended, i.e. if there is no the image in the list (grep fails) it does not abort and proceeds. And if the image is in the list (grep succeeds) then it aborts due to && false. Although if you run it from the command line and check $? it is 1 in both cases. I suppose it is a peculiarity of how set -e works that I am not aware of. I just saw this idiom in ceph tests many times and used it here not thinking much about it, so for me it did not look confusing, but if you think it is we may change it to something more clear.

Copy link
Contributor Author

@trociny trociny Jun 22, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually the man bash explains this, and I remember I read it in the past (probably several times) but forgot after some time had passed.

              -e      Exit immediately if a pipeline (which may consist of a single simple command), a list, or a compound command (see SHELL GRAMMAR above), exits with a non-zero  sta-
                      tus.   The shell does not exit if the command that fails is part of the command list immediately following a while or until keyword, part of the test following the
                      if or elif reserved words, part of any command executed in a && or || list except the command following the final && or ||, any command in a pipeline but the last,
                      or if the command's return value is being inverted with !.  If a compound command other than a subshell returns a non-zero status because a command failed while -e
                      was being ignored, the shell does not exit.  A trap on ERR, if set, is executed before the shell exits.  This option applies to the shell environment and each sub-
                      shell environment separately (see COMMAND EXECUTION ENVIRONMENT above), and may cause subshells to exit before executing all the commands in the subshell.

Copy link
Member

@tserong tserong Jun 23, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just looking at http://see.prv.suse.net:8080/blue/organizations/jenkins/sesdev-integration/detail/PR-699/1/pipeline, that CI test seems to have failed because:

    master: + ssh master rbd showmapped
    master: 0   rbd              iscsi_disk_user  -     /dev/rbd0

So with backstore=user:rbd the volume is (incorrectly) mapped in this CI test. Note that the CI does a single node deployment on openSUSE 15.3, in case that makes any difference. I'll try to reproduce locally myself and poke around a bit...

Copy link
Contributor Author

@trociny trociny Jun 23, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, interesting. I will try to reproduce it with one node scenario, although I can't imaging how it could be related.

Copy link
Contributor Author

@trociny trociny Jun 23, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, it is failing for openSUSE. I think it is more important here. Right now I have no idea how iscsi support is built for openSUSE, if it has necessary bits. I will investigate. As the last resort we may enable this test for SES only.

Copy link
Member

@tserong tserong Jun 23, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hrm... openSUSE 15.3 is EOL since the end of December 2022. Under the circumstances, I wonder if we should maybe switch the CI to deploying ses7p on SLE, rather than pacific on openSUSE? I've opened #702 to see if that might work.

Copy link
Contributor Author

@trociny trociny Jun 23, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried sesdev pacific deploy and reproduced the issue. The backstore=user:rbd fails because of this tcmu-runner error:

2023-06-23 10:46:30.521 7 [ERROR] add_device:456: could not find handler for uio0

And it is because tcmu-runner is built without rbd support here:

[ceph: root@node1 /]# ls /usr/lib64/tcmu-runner/
handler_qcow.so

There is no handler_rbd.so here.
I will modify the test to test backstore=user:rbd only if it is running on ses deployment.

ssh ${node} cat '/sys/devices/tcm_user/uio/uio*/name' |
grep "${rbd_pool}/${rbd_image}"
;;
*)
false
;;
esac

iscsiadm -m discovery -t st -p ${node}
test $(iscsiadm -m discovery -t st -p ${node} | awk '{print $2; exit}') = ${target}
iscsiadm -m discovery -t st -p ${node}
test $(iscsiadm -m discovery -t st -p ${node} | awk '{print $2; exit}') = ${target}

! test -e /dev/sda
! test -e /dev/sda

iscsiadm -m node -T ${target} -l
iscsiadm -m node -T ${target} -l

for i in `seq 10`; do
test -e /sys/block/sda/device/model && break
sleep 1
done
test -e /dev/sda
test -e /sys/block/sda/device/model
test $(cat /sys/block/sda/device/model) = RBD
for i in `seq 10`; do
test -e /sys/block/sda/device/model && break
sleep 1
done
test -e /dev/sda
test -e /sys/block/sda/device/model
case ${backstore} in
rbd)
grep RBD /sys/block/sda/device/model
;;
user:rbd)
grep TCMU /sys/block/sda/device/model
;;
*)
false
;;
esac

rbd du ${rbd_pool}/${rbd_image}
used=$(rbd du ${rbd_pool}/${rbd_image} --format json |
jq -r ".images[0].used_size")
test ${used} -eq 0
rbd du ${rbd_pool}/${rbd_image}
used=$(rbd du ${rbd_pool}/${rbd_image} --format json |
jq -r ".images[0].used_size")
test ${used} -eq 0

dd if=/dev/urandom of=/tmp/test.dat bs=4M count=10
dd if=/tmp/test.dat of=/dev/sda bs=4M count=10
dd if=/dev/urandom of=/tmp/test.dat bs=4M count=10
dd if=/tmp/test.dat of=/dev/sda bs=4M count=10

test "$(dd if=/tmp/test.dat bs=4M count=10 | md5sum)" = \
"$(dd if=/dev/sda bs=4M count=10 | md5sum)"
test "$(dd if=/tmp/test.dat bs=4M count=10 | md5sum)" = \
"$(dd if=/dev/sda bs=4M count=10 | md5sum)"

rbd du ${rbd_pool}/${rbd_image}
used=$(rbd du ${rbd_pool}/${rbd_image} --format json |
jq -r ".images[0].used_size")
test ${used} -eq 41943040
rbd du ${rbd_pool}/${rbd_image}
used=$(rbd du ${rbd_pool}/${rbd_image} --format json |
jq -r ".images[0].used_size")
test ${used} -eq 41943040

iscsiadm -m node -T ${target} -u
iscsiadm -m node -T ${target} -u

${gwcli} /iscsi-targets/${target}/hosts delete ${client}
${gwcli} /iscsi-targets/${target}/disks delete ${rbd_pool}/${rbd_image}
${gwcli} /iscsi-targets/${target}/gateways delete ${gw_name} confirm=true
${gwcli} /iscsi-targets delete ${target}
${gwcli} /disks detach ${rbd_pool}/${rbd_image}
${gwcli} /iscsi-targets/${target}/hosts delete ${client}
${gwcli} /iscsi-targets/${target}/disks delete ${rbd_pool}/${rbd_image}
${gwcli} /iscsi-targets/${target}/gateways delete ${gw_name} confirm=true
${gwcli} /iscsi-targets delete ${target}
${gwcli} /disks detach ${rbd_pool}/${rbd_image}

ssh ${node} rbd showmapped | grep "${rbd_pool} *${rbd_image}" && false
ssh ${node} rbd showmapped | grep "${rbd_pool} *${rbd_image}" && false

rbd rm ${rbd_pool}/${rbd_image}
rbd rm ${rbd_pool}/${rbd_image}
done
}

function iscsi_test {
Expand Down