The META service is always bound to the IP address of TCP,causing the meta service to be unable to work

[MingWangSong]

Hello, about the problem that IB network card cannot be recognized when deploying 3fs, I did not get the answer from the official, please tell me how you deal with it. I am currently in a similar environment to this problem, but I manually deployed it locally by myself. From the log file, I can see that after the meta service configured RDMA protocol, it still chose ordinary network card instead of IB network card.

I hope you get your guidance, thank you very much!

[MingWangSong]

This issue88 is very helpful to me, but in the end I don't understand how to adapt the meta service to recognize the ib network card, please give me some advice.

[diabloneo]

@MingWangSong Do you mean you need to know how to config the network options in the services' config when deploying 3FS cluster manually?

[MingWangSong]

Yes, I am looking forward to your help very much. I am currently deploying in my cluster environment according to the 3fs official document, but my server node contains an Ethernet card and an IB card. Now I have a problem when configuring meta network. Instead, select the Ethernet card.

I would like to ask how your open source project finally implements the deployment of monitor service and mgmtd service in IB environment

Thank you very much for your help. I think the official reply is too official and of no practical use

[diabloneo]

The key part is the mgmtd service's main configuration. You need to set it's first service group's network_type = 'IPoIB'. Other services' configurations are same as official documents suggested.

# mgmtd_main.toml

[[server.base.groups]]
check_connections_interval = '1min'
connection_expiration_time = '1day'
network_type = '{{ .MgmtdProtocol }}'   <-- set here to IPoIB
services = [ 'Mgmtd' ]
use_independent_thread_pool = false

[MingWangSong]

Yes, the mgmtd service is configured in this way. Thank you for reminding me of this key point. I have learned about it through your other comments before.
However, what bothers me is that after the mgmtd service is configured in this way, the RDMA network card still cannot be identified after the subsequent deployment of meta service by referring to the official configuration suggestions. My log file is as follows, thank you for your guidance:

[MingWangSong]

In the screenshot above, 32.8.32.1 is Ethernet and 89.72.32.1 is IB

[diabloneo]

Did you set the mgmtd_server_addresses option to something like 'IPoIB://xxxx' in your storage and fuse configuration files?