Issues with rsockets and softroce?

[davew723]

I'm wondering if I'm doing something wrong here? Although the same approach is working with my mellanox cards. Example Issue:

Using AWS I setup two EC2 instances. Verify that security group allows me to ssh between the two instances. Them I install...

sudo yum groupinstall -y "Development Tools"
sudo yum install -y cmake git
sudo yum install -y rdma-core rdma-core-devel

... Next I setup my SoftRoCE interface.

sudo rdma link add rxe0 type rxe netdev eth0

Run ./rdma_server on the first EC2 instance ...

[ec2-user@ip-10-0-3-213 build]$ ./rdma_server
rdma_server: start
rdma_server: end 0

And rdma_client on the second...

[dev2:/home/ec2-user/git/rsockets/build]./rdma_client 10.0.3.213
rdma_client: start
rdma_client: end 0

SUCCESS. Although rdma_client does NOT use https://github.com/linux-rdma/rdma-core/blob/master/librdmacm/rsocket.*

Next test rcopy in the same fashion.

[ec2-user@ip-10-0-3-213 build]$ ./rcopy
waiting for connection...

[dev2:/home/ec2-user/git/rsockets/build]./rcopy /etc/hosts 10.0.3.213:/tmp/hosts
rconnect failed
: Invalid argument

FAIL Invalid argument. I tried a couple of the other rsockets examples like riostream and it fails similarly but on rpoll.

These same tests work fine with a mellanox adapter running RoCE.

Anyone else run into this?