soft-roce can't connect successfully with hw-ROCE

[janes09]

I test that it is ok that hw-roce connect hw-roce, also it is ok that soft-roce connect soft-roce. but hw-roce connect failure.

i trace the message: try to find the message difference,  i found that hw-roce's RC Acknowledge.s BTH header: MigReq is TRUE, but soft-roce's value is FALSE.

Because i can't check whether this is the root cause, so i raise the issue here.

[monis410]

Which hardware do you use for HardRoCE?
And which test?

[janes09]

ConnectX-3_Pro, use rsocket feature in librdmacm.

[monis410]

I ran some tests of Soft vs. Hard (ConnectX-3) successfully and without any issue
Can you please try other utils?

1. ibv_rc-pingpong
2. ucmatose
3. rping

ALso, please send full tcpdump (or even better, ibdump) of a test that fails

[janes09]

Base Transport Header
Opcode: 11
0... .... = Solicited Event: False
.1.. .... = MigReq: True //here is different between soft-roce and hw-roce. Hw-roce is 1, soft-roce is 0.
.... 0000 = Header Version: 0

[janes09]

i use ROCE V2, not ROCE V1

[monis410]

SoftRoCE supports only RoCEv2

[janes09]

Does soft-ROCE ROCE V1? i install soft-ROCE, the default is ROCE V2. Now i don't know how to set soft-ROCE to support ROCE V1, can you tell me the method for setting?

[janes09]

ok. but you said： “I ran some tests of Soft vs. Hard (ConnectX-3)”， the ConnectX-3 don't support ROCE V2, only ConnectX-3_pro support ROCE V2

[janes09]

I use rping to test, it is fine. But i don't know why rsocket is not ok.

[janes09]

soft-roce don't support qperf test, also don't have OFED package, it isn't very convenient to use.

[monis410]

To all your questions

1. I wrote ConnectX-3 but meant to say ConnectX-3 Pro. SoftRoCE supports only V1 so compatability with is only with RoCEv2 capable hardware.
2. I will try with rsocket and let you know
3. Not sure what you mean by soft-roce don't support qperf test. Did you run a test that failed? Which one?
4. SoftRoCE is planned for OFED but I can't say when exactly

[janes09]

qperf tool will check some configuration: in file: configure.in, it do:

"AC_CHECK_LIB(ibverbs, ibv_open_device, RDMA=1)
AC_CHECK_LIB(ibverbs, ibv_open_xrc_domain, HAS_XRC=1)"

finally after install qperf, don't have RDMA test function, i analyze it, and find the RDMA is compiler macro in qperf source code, it should be that don't open the RDMA macro when install qperf. but HW-ROCE is fine to install qperf RDMA test function.

[Housin9]

Please can we install soft roce over ethernet adapter card?