Support setups with multiple YB instances running on a single machine or on non-default port

[eivanov89]

Hi,

tpccbenchmark has an argument "--nodes". YB nodes should be listed without ports, which assumes they work on different machines. Also it looks like it's impossible to change the default port even for a single instance. It could be very beneficial for our TPC-C measurements, if you support nodes specification with per-node ports. Because we run TPC-C on a servers with 128 cores, and unfortunately one tserver doesn't scale vertically well enough (as expected).

[FranckPachot]

one tserver doesn't scale vertically well enough (as expected)

Can you explain why? Maybe not with the default settings that set 8 tablets per table per servers (to fit 8 or 16 vCPUs and many tables), but everything in a tserver should scale as it can hold many tablets. With 128 cores, you may increase yb_num_shards_per_tserver to, maybe 64 or 128

[eivanov89]

one tserver doesn't scale vertically well enough (as expected)

Can you explain why? Maybe not with the default settings that set 8 tablets per table per servers (to fit 8 or 16 vCPUs and many tables), but everything in a tserver should scale as it can hold many tablets. With 128 cores, you may increase yb_num_shards_per_tserver to, maybe 64 or 128

Mainly, because that was the verdict of YugabyteDB team after our intensive discussion of results in the slack and later by email. Also, YB docs state, that the system has been tested on the hardware up to 64 cores (and IIRC up to 200 GiB RAM). I don't have a link or quote, but I heard that with 128 cores even advanced allocators like TCMalloc and Jemaloc might work worse. Besides that, with a one huge process running on all cores I would expect high inefficiency in CPU cache usage and messed NUMA. In my experience with YDB, running multiple instances within a taskset (taskset is main thing here) improves average performance by 20%.

Unfortunately, I don't remember if yb_num_shards_per_tserver has been tried. If you have access to the old slack messages, you could check it. Anyway, I'm ready to give it a try as well.

[FranckPachot]

Okay, yes, there may be multiple things to configure differently than the default to run on a 128 vCPU machine, as the goal of YugabyteDB is to scale out rather than scale up. So you want to run multiple T-Servers on one physical host, but tpccbenchmark connects to different hosts, not to other ports on the same host.

I would run all that in Docker or Kubernetes and problem solved. It is cloud-native 😀

But everything that containers do, you can do it on Linux. You can add IP aliases (something like ip addr add) and route from them to the ports where you run the yb-tserver (maybe something like iptables -t nat -A PREROUTING).

[eivanov89]

I would run all that in Docker or Kubernetes and problem solved. It is cloud-native 😀

Indeed :) That's our Plan B for the next performance round. But I have a much warmer feelings about the bare metal anyway :)