Olio workload: got a SocketTimeoutException exception

[sguazt]

Hello,

I'm running RAIN agains Olio (Ruby on Rails version).

In the "rain.config.olio.json" I've changed the target hostname to "192.168.122.219", which is the IP address of the host where Olio is running.

After executing the command:
java -Xmx1g -Xms256m -cp "rain.jar:workloads/olio.jar" radlab.rain.Benchmark config/rain.config.olio.json

I obtain the following result:

...
[SCOREBOARD TRACK: cloudstone-001] Operations initiated : 413
[SCOREBOARD TRACK: cloudstone-001] Operations successfully completed : 6
[SCOREBOARD TRACK: cloudstone-001] Average operation response time (s): 1.6710
[SCOREBOARD TRACK: cloudstone-001] Operations late : 68
[SCOREBOARD TRACK: cloudstone-001] Operations failed : 339
...
SCOREBOARD TRACK: cloudstone-001] Error Summary Results : 1 types of error(s)
[SCOREBOARD TRACK: cloudstone-001] Read timed out (class java.net.SocketTimeoutException): 339
[SCOREBOARD TRACK: cloudstone-001] Total failures : 339

Note the number of failed operations and the error summary.

I've tried to connect from the RAIN host to the Olio host with telnet is it seems OK:
$ telnet 192.168.122.219 80
Trying 192.168.122.219...
Connected to 192.168.122.219.
Escape character is '^]'.

So I don't think it is a network problem.

What could it be?

Thank you very much for the support.

Best,

-- Marco Guazzone

[rean]

Hello Marco,

A socket timeout means that the server did not respond in time (default
is 10 seconds) to a request from a load generator thread.

You can:
a) increase the max timeout from 10 seconds to something larger
b) increase the resources available on the Olio webserver and/or
database (cpu and memory) if possible
c) run netperf between client and server machines to see what the max
bandwidth is and calibrate your workload to suit.

thanks,
Rean

On 06/18/2012 08:49 AM, sguazt wrote:

Hello,

I'm running RAIN agains Olio (Ruby on Rails version).

In the "rain.config.olio.json" I've changed the target hostname to "192.168.122.219", which is the IP address of the host where Olio is running.

After executing the command:
java -Xmx1g -Xms256m -cp "rain.jar:workloads/olio.jar" radlab.rain.Benchmark config/rain.config.olio.json

I obtain the following result:

...
[SCOREBOARD TRACK: cloudstone-001] Operations initiated : 413
[SCOREBOARD TRACK: cloudstone-001] Operations successfully completed : 6
[SCOREBOARD TRACK: cloudstone-001] Average operation response time (s): 1.6710
[SCOREBOARD TRACK: cloudstone-001] Operations late : 68
[SCOREBOARD TRACK: cloudstone-001] Operations failed : 339
...
SCOREBOARD TRACK: cloudstone-001] Error Summary Results : 1 types of error(s)
[SCOREBOARD TRACK: cloudstone-001] Read timed out (class java.net.SocketTimeoutException): 339
[SCOREBOARD TRACK: cloudstone-001] Total failures : 339

Note the number of failed operations and the error summary.

I've tried to connect from the RAIN host to the Olio host with telnet is it seems OK:
$ telnet 192.168.122.219 80
Trying 192.168.122.219...
Connected to 192.168.122.219.
Escape character is '^]'.

So I don't think it is a network problem.

What could it be?

Thank you very much for the support.

Best,

-- Marco Guazzone

---

Reply to this email directly or view it on GitHub:
#6

[sguazt]

Thank you, Rean.

I've tried to investigate.
Unfortunately I didn't find any value related to timeouts, except for the "keepalive_timeout" parameter of the NGINX server (the Web Server running on the Olio host) which was set to 65.
Anyway, I've tried to lower the workload, by decreasing the number of users in the JSON profile file:
"loadProfile": [
{
"interval": 20,
"users": X,
"mix": "default"
},
{
"interval": 15,
"users": Y,
"mix": "default"
},
{
"interval": 25,
"users": Z,
"mix": "default"
}

Interestingly, even if I set X=1, Y=1, Z=1, I occasionally obtain errors.
In particular, I've tried to perform 10 runs of RAIN with this profile and the result is that:

• For 5 runs, I got no error
• For the remaining 5 runs, I got the following type of error:
  Could not find success message in result body! (class java.lang.Exception)

Here below, a more detailed report of each run.

-- 1st Run:
[SCOREBOARD TRACK: cloudstone-001] Operations initiated : 11
[SCOREBOARD TRACK: cloudstone-001] Operations successfully completed : 9
[SCOREBOARD TRACK: cloudstone-001] Average operation response time (s): 0.2897
[SCOREBOARD TRACK: cloudstone-001] Operations late : 0
[SCOREBOARD TRACK: cloudstone-001] Operations failed : 2
...
[SCOREBOARD TRACK: cloudstone-001] Error Summary Results : 1 types of error(s)
[SCOREBOARD TRACK: cloudstone-001] Could not find success message in result body! (class java.lang.Exception): 2
[SCOREBOARD TRACK: cloudstone-001] Total failures : 2

-- 2nd Run:
[SCOREBOARD TRACK: cloudstone-001] Operations initiated : 11
[SCOREBOARD TRACK: cloudstone-001] Operations successfully completed : 10
[SCOREBOARD TRACK: cloudstone-001] Average operation response time (s): 0.3713
[SCOREBOARD TRACK: cloudstone-001] Operations late : 1
[SCOREBOARD TRACK: cloudstone-001] Operations failed : 0
...
[SCOREBOARD TRACK: cloudstone-001] Error Summary Results : 0 types of error(s)
[SCOREBOARD TRACK: cloudstone-001] Total failures : 0

-- 3rd Run:
[SCOREBOARD TRACK: cloudstone-001] Operations initiated : 8
[SCOREBOARD TRACK: cloudstone-001] Operations successfully completed : 8
[SCOREBOARD TRACK: cloudstone-001] Average operation response time (s): 0.3499
[SCOREBOARD TRACK: cloudstone-001] Operations late : 0
[SCOREBOARD TRACK: cloudstone-001] Operations failed : 0
...
[SCOREBOARD TRACK: cloudstone-001] Error Summary Results : 0 types of error(s)
[SCOREBOARD TRACK: cloudstone-001] Total failures : 0

-- 4th Run:
[SCOREBOARD TRACK: cloudstone-001] Operations initiated : 12
[SCOREBOARD TRACK: cloudstone-001] Operations successfully completed : 11
[SCOREBOARD TRACK: cloudstone-001] Average operation response time (s): 0.3310
[SCOREBOARD TRACK: cloudstone-001] Operations late : 0
[SCOREBOARD TRACK: cloudstone-001] Operations failed : 1
...
[SCOREBOARD TRACK: cloudstone-001] Error Summary Results : 1 types of error(s)
[SCOREBOARD TRACK: cloudstone-001] Could not find success message in result body! (class java.lang.Exception): 1
[SCOREBOARD TRACK: cloudstone-001] Total failures : 1

-- 5th Run:
[SCOREBOARD TRACK: cloudstone-001] Operations initiated : 17
[SCOREBOARD TRACK: cloudstone-001] Operations successfully completed : 14
[SCOREBOARD TRACK: cloudstone-001] Average operation response time (s): 0.3284
[SCOREBOARD TRACK: cloudstone-001] Operations late : 1
[SCOREBOARD TRACK: cloudstone-001] Operations failed : 2
...
[SCOREBOARD TRACK: cloudstone-001] Error Summary Results : 1 types of error(s)
[SCOREBOARD TRACK: cloudstone-001] Could not find success message in result body! (class java.lang.Exception): 2
[SCOREBOARD TRACK: cloudstone-001] Total failures : 2

-- 6th Run:
[SCOREBOARD TRACK: cloudstone-001] Operations initiated : 14
[SCOREBOARD TRACK: cloudstone-001] Operations successfully completed : 13
[SCOREBOARD TRACK: cloudstone-001] Average operation response time (s): 0.3383
[SCOREBOARD TRACK: cloudstone-001] Operations late : 0
[SCOREBOARD TRACK: cloudstone-001] Operations failed : 1
...
[SCOREBOARD TRACK: cloudstone-001] Error Summary Results : 1 types of error(s)
[SCOREBOARD TRACK: cloudstone-001] Could not find success message in result body! (class java.lang.Exception): 1
[SCOREBOARD TRACK: cloudstone-001] Total failures : 1

-- 7th Run:
[SCOREBOARD TRACK: cloudstone-001] Operations initiated : 10
[SCOREBOARD TRACK: cloudstone-001] Operations successfully completed : 10
[SCOREBOARD TRACK: cloudstone-001] Average operation response time (s): 0.3160
[SCOREBOARD TRACK: cloudstone-001] Operations late : 0
[SCOREBOARD TRACK: cloudstone-001] Operations failed : 0
...
[SCOREBOARD TRACK: cloudstone-001] Error Summary Results : 0 types of error(s)
[SCOREBOARD TRACK: cloudstone-001] Total failures : 0

-- 8th Run:
[SCOREBOARD TRACK: cloudstone-001] Operations initiated : 9
[SCOREBOARD TRACK: cloudstone-001] Operations successfully completed : 8
[SCOREBOARD TRACK: cloudstone-001] Average operation response time (s): 0.3476
[SCOREBOARD TRACK: cloudstone-001] Operations late : 0
[SCOREBOARD TRACK: cloudstone-001] Operations failed : 1
...
[SCOREBOARD TRACK: cloudstone-001] Error Summary Results : 1 types of error(s)
[SCOREBOARD TRACK: cloudstone-001] Could not find success message in result body! (class java.lang.Exception): 1
[SCOREBOARD TRACK: cloudstone-001] Total failures : 1

-- 9th Run:
[SCOREBOARD TRACK: cloudstone-001] Operations initiated : 11
[SCOREBOARD TRACK: cloudstone-001] Operations successfully completed : 11
[SCOREBOARD TRACK: cloudstone-001] Average operation response time (s): 0.3101
[SCOREBOARD TRACK: cloudstone-001] Operations late : 0
[SCOREBOARD TRACK: cloudstone-001] Operations failed : 0
...
[SCOREBOARD TRACK: cloudstone-001] Error Summary Results : 0 types of error(s)
[SCOREBOARD TRACK: cloudstone-001] Total failures : 0

-- 10th Run:
[SCOREBOARD TRACK: cloudstone-001] Operations initiated : 7
[SCOREBOARD TRACK: cloudstone-001] Operations successfully completed : 7
[SCOREBOARD TRACK: cloudstone-001] Average operation response time (s): 0.3401
[SCOREBOARD TRACK: cloudstone-001] Operations late : 0
[SCOREBOARD TRACK: cloudstone-001] Operations failed : 0
...
[SCOREBOARD TRACK: cloudstone-001] Error Summary Results : 0 types of error(s)

[SCOREBOARD TRACK: cloudstone-001] Total failures : 0

Very strange.
Where am wrong?

Thank you very much.

Best,

-- Marco