Micro-benchmark Results for version 2.0

[Christoph-Denzel]

Hello, I just wanted to note that this is a non-issue.
I just did not know where else to put it, since there is no discussion tab.

In the paper from 2014, there was a reported maximum throughput of roughly 80,000 requests for a 0/0 micro-benchmark.
I did some performance tests on more powerful machines and a fast network, but in general, the performance was worse than reported.
I did achieve roughly 40 to 50k requests per second.
The difference, I think, is latency, where even for a single client, it takes 4ms to receive an answer, even after some warm-up time.
This is a lot higher than the reported 1ms from back then.
The round-trip time for the network is 0.2 - 0.3ms using ICMP pings.

As far as I understand, the biggest difference is the use of TLS for replica communication and signatures for client communication.
Has something else changed since then that could have impacted performance?

Did you make any new measurements with version 2.0, and if so, could you report them?

Thanks in advance :)

[rvassantlal]

Hello,

Can you let me know how many clients you experimented with?

The last measurements that I have are for f=1 and requests 1024/0. I got throughput around 16.5k req/s, which is consistent with the paper.

I will run some tests for 0/0 and get back to you.

[Christoph-Denzel]

Thanks,

these are my results for 0/0 and 4kb/4kb.

I couldn't get more clients to work, since the system detected replay leader attacks or threw other errors and got stuck by repeating this message.
-- Last regency: 0, next regency: 1
-- Re-transmitting STOP message to install regency 1

I already increased the system.totalordermulticast.timeout to 60000.
Fair batching is used, if that makes a difference.

[rvassantlal]

Hi,

What value did you use for system.optimizations.readonly_requests? That parameter is due to the integration of the work https://arxiv.org/pdf/2107.11144. I forgot to mention that, besides the TLS, this feature is also new in version 2.0.

With that parameter set to false, I reached at least ~80000 req/s with 2000 clients in 0/0 benchmark. With 2500 clients, the throughput reached ~85000 req/s. With 1000 clients, the system could only reach ~74000 req/s.

Regarding the 4k/4k benchmark, I reached ~4400 req/s with 1500 clients.

However, with system.optimizations.readonly_requests set to true, I got similar results that you observed for 0/0, and could measure for 4k/0 (instead of 4kb/4kb) and got ~4300 req/s. With the parameter true, the servers have to exchange responses among them, increasing network usage.

In the recent commit to master, can you also check bftsmart.benchmark.BenchmarkSetup::createSystemConf? That method contains benchmark parameters that I use.

[Christoph-Denzel]

The read-only optimisation is turned off.

My config was a bit different, but I changed it to the values that you used.
However, the results I got were the same as before.

I don't use the ThroughputLatencyBenchmarkStrategy,
only the "normal" classes and execute these commands on the different servers.

This is for 0/0
./smartrun.sh bftsmart.benchmark.ThroughputLatencyServer 0 0
./smartrun.sh bftsmart.benchmark.ThroughputLatencyClient 0 1000 100000 0 true false 0

This is for 4kb/4kb
./smartrun.sh bftsmart.benchmark.ThroughputLatencyServer 0 4000
./smartrun.sh bftsmart.benchmark.ThroughputLatencyClient 0 1000 100000 4000 true false 0

This is an example output
-- M:(clients[#]|requests[#]|delta[ns]|throughput[ops/s], max[ops/s])>(1000|94717|2004028720|47263.29470966863|47263.29470966863)

Could the set up be wrong?

It is just very interesting that my throughput for 4kb is so much higher but for 0 it is lower.

[rvassantlal]

In my tests, I also use ThroughputLatencyClient; however, the clients are distributed among 10 machines.

Do you only use a single machine for clients? It is possible that a single machine cannot handle 1000 clients sending requests. This might increase the latency and decrease the throughput because clients are not sending enough requests.

Indeed, it is impressive that bftsmark can process almost 20k req/s with 4kb requests.

[Christoph-Denzel]

I tried using 1 with 1000 clients.
But then I thought the same, that maybe a single machine cannot handle it.
However, using 2 with 500 each and after that 2 with 1000 each,
the results stayed roughly the same for all cases.

So sadly I don't think that is the reason.

But it is still weird that the 4kb is that much better.

Not sure what else to tweak,
but anyways thank you so much for your help :)

[rvassantlal]

I am intrigued. The only way I could justify better results for 4kb is that a single machine can handle some number of clients. After that, it does not help.

However, you tried using multiple machines for clients.

I checked the round-trip time, and it is similar to yours.

Wait a minute. You said you are starting clients with 100k requests, meaning each client sends 100k requests. Since you get throughput of 50k req/s, each client sends its 100k requests in a few seconds and terminates its execution before other clients are initialized. Therefore, you might not have all 1k clients sending requests together, meaning you only have a few clients per second sending requests. In my tests, I made clients send 10_000_000 requests. This allows all clients to initialize and send requests together.

Can you please check the throughput by increasing the number of requests?

[Christoph-Denzel]

50k requests for 1000 clients means 50 requests for a client each second.
Thus, I would think it is enough, because that still takes quite some time to reach 100k per client.
Also the LatencyClient uses a countdown latch, so they should start roughly at the same time I suppose?

But I also tried with more requests and it didn't make a difference :(

[rvassantlal]

-- M:(clients[#]|requests[#]|delta[ns]|throughput[ops/s], max[ops/s])>(1000|94717|2004028720|47263.29470966863|47263.29470966863)

You are right about the number of requests; the log message you observed shows that there are 1k clients sending requests.

The use of the latch does not affect when clients start sending requests.

For now, I don't have more ideas.

[Christoph-Denzel]

I found an interesting behaviour for the 4KB/4KB version.
Where after some time for a larger number of clients a spike in throughput occurs.
Do you experienced something similar or have any idea why that could be the case?

[rvassantlal]

It could be related to the JVM (garbage collection) or the OS; it's hard to say.

In my measurements, I also have variation in throughput. You can see here for 250 clients:

Here is a graph of the average throughput for different numbers of clients:

Can you profile the machine and network during the execution?

[Christoph-Denzel]

The server measurement starts when the first server is started and the client when the servers are ready, hence the delay between them.

I took the median of 30 measurements and this seems to be a regular pattern.
And since it occurs later with an increasing number of clients it doesn't seem to be random.
But if there is nothing in BFT-SMaRt that could cause this, then I guess it has to be because of the OS or JVM.