A12EC2.log

CPEN 431 EVALUATION test Client
Time: Fri Apr 14 04:16:09 UTC 2023
Build Date: Fri Apr 14 04:15:54 UTC 2023
Java process ID: 8089
Assignment ID: 11
Client Version: 1

[Info] A11 client running in SUBMIT mode.

[Info] Starting A11 Tests

[Info] Correct submission information.
[Info] Checking if the test client is running on AWS
[Info] Detected instance type: c5a.16xlarge

[Info] Done building nodes' list.

[Info] The deployment has 40 server nodes.

[Info] Checking SSH connectivity to deployed servers ...
[Info] SSH check passed!

[Info] Checking server deployment.
Deployment information:
c5a.16xlarge 
 Ubuntu 20.04.6 LTS 
 qdisc netem 8001: root refcnt 2 limit 1000 delay 5.0ms loss 0.25% 
 qdisc netem 8002: root refcnt 65 limit 1000 delay 5.0ms loss 0.25% 
 net-tools 1.60+git20180626.aebd88e-1ubuntu1

[Test isAlive] for 127.0.0.1:43100
Sending GETPID to all servers... 
[Info] Server node 127.0.0.143100 has process ID 8059.
[ ... Completed in 0.008 seconds ]
[OK]
Sending GetMembershipCount to all servers... 
[Info] Server 127.0.0.1:43100 replied with membership count = 1 (expected count = 1)
[ ... Completed in 0.003 seconds ]
[Ok] All servers returned the expected membership count (1)
Sending WIPE-OUT to all servers...
[ ... Completed in 0.012 seconds ]
[OK]
Waiting for 5 seconds for the cache to be cleared!

[ TEST Basic PUT/GET (PUT -> GET). Value Length = 8 Bytes ]
TEST_PASSED

[ TEST Basic PUT/GET (PUT -> GET). Value Length = 32 Bytes ]
TEST_PASSED

[ TEST Basic PUT/GET (PUT -> GET). Value Length = 2048 Bytes ]
TEST_PASSED

[ TEST Basic PUT/GET (PUT -> GET). Value Length = 10000 Bytes ]
TEST_PASSED


[ TEST Basic KVS with message version (PUT Version 1 -> GET -> PUT Version 2 -> GET). Value Length = 8 Bytes ]
TEST_PASSED


[ TEST Basic KVS with message version (PUT Version 1 -> GET -> PUT Version 2 -> GET). Value Length = 32 Bytes ]
TEST_PASSED


[ TEST Basic KVS with message version (PUT Version 1 -> GET -> PUT Version 2 -> GET). Value Length = 2048 Bytes ]
TEST_PASSED


[ TEST Basic KVS with message version (PUT Version 1 -> GET -> PUT Version 2 -> GET). Value Length = 10000 Bytes ]
TEST_PASSED

[ TEST Basic REMOVE (PUT -> GET -> REM -> GET). Value Length = 32 Bytes. ]
TEST_PASSED

[ TEST Double remove (PUT -> REM -> REM). Value Length = 32 Bytes. ]
TEST_PASSED

[ TEST Invalid Command. ]
TEST_PASSED

[ TEST At-Most-Once Client Policy (PUT -> REM -> REM). Value length =  32 Bytes ]
TEST_PASSED

[ TEST At-Most-Once Client Policy Empty REMs (EMPTY -> REM -> REM) ]
TEST_PASSED

[ TEST At-Most-Once Client Policy Error Return Code (REM -> PUT -> REM) ]
TEST_PASSED

[ TEST At-Most-Once Client Policy (PUT -> REM -> many PUT -> REM). Value length =  32 Bytes ]
Pausing for 10 seconds to wait for the cache to be cleared!
TEST_PASSED

[ TEST At-Most-Once Client Policy For PUTs (PUT1 -> PUT2 -> PUT1 -> GET2) ]
TEST_PASSED

[TEST Invalid Key ]
TEST_PASSED

[TEST Invalid Value ]
TEST_PASSED

[ TEST Single Front-End Performance (1 clients, 60.0 seconds): PUT -> GET (closed loop) ]
Test time limit reached: 60.0s.
Test completed after 316168 closed loops.
[Test isAlive] for 127.0.0.1:43100
[ ... Completed in 0.001 seconds ]
Local Single node: Test Status: TEST_PASSED
Clients: 1
Total Requests (without replies): 632336
Total Requests (including replies): 632336
Successful Responses: 632336
Timeout Responses: 0
Total Failed Responses: 0
% Success: 100.0
% Timeout: 0.0
% Failed: 0.0
Response Time (Successful requests):
[max] 11ms, [min] 0ms
[avg] 0.08126692138356822 ms, [stdev] 0.2860644114180748ms
Throughput: 10538.757687371877 requests per second
Goodput: 10538.757687371877 requests per second
Retry Rate (Successful requests): 1.0
Sending WIPE-OUT to all servers...
[ ... Completed in 0.023 seconds ]
[OK]
Waiting for 5 seconds for the cache to be cleared!
[ ... TEST Completed in 65.536 seconds ]

[ TEST Single Front-End Performance (16 clients, 60.0 seconds): PUT -> GET (closed loop) ]
Test time limit reached: 60.0s.
Test completed after 46942 closed loops.
Test time limit reached: 60.0s.
Test completed after 46904 closed loops.
Test time limit reached: 60.0s.
Test time limit reached: 60.0s.
Test time limit reached: 60.0s.
Test time limit reached: 60.0s.
Test completed after 46911 closed loops.
Test completed after 46929 closed loops.
Test time limit reached: 60.0s.
Test completed after 46928 closed loops.
Test time limit reached: 60.0s.
Test completed after 46940 closed loops.
Test time limit reached: 60.0s.
Test completed after 46923 closed loops.
Test time limit reached: 60.0s.
Test completed after 46951 closed loops.
Test time limit reached: 60.0s.
Test completed after 46950 closed loops.
Test completed after 46914 closed loops.
Test time limit reached: 60.0s.
Test time limit reached: 60.0s.
Test time limit reached: 60.0s.
Test completed after 46915 closed loops.
Test time limit reached: 60.0s.
Test time limit reached: 60.0s.
Test completed after 46953 closed loops.
Test completed after 46931 closed loops.
Test completed after 46945 closed loops.
Test completed after 46933 closed loops.
Test completed after 46883 closed loops.
[Test isAlive] for 127.0.0.1:43100
[ ... Completed in 0.001 seconds ]
Local Single node: Test Status: TEST_PASSED
Clients: 16
Total Requests (without replies): 1501695
Total Requests (including replies): 1501695
Successful Responses: 1501695
Timeout Responses: 0
Total Failed Responses: 0
% Success: 100.0
% Timeout: 0.0
% Failed: 0.0
Response Time (Successful requests):
[max] 15ms, [min] 0ms
[avg] 0.6250703371856469 ms, [stdev] 0.6909067665227075ms
Throughput: 25027.41575280824 requests per second
Goodput: 25027.41575280824 requests per second
Retry Rate (Successful requests): 1.0
Sending WIPE-OUT to all servers...
[ ... Completed in 0.019 seconds ]
[OK]
Waiting for 5 seconds for the cache to be cleared!
[ ... TEST Completed in 65.764 seconds ]

[ TEST Single Front-End Performance (64 clients, 60.0 seconds): PUT -> GET (closed loop) ]
Test time limit reached: 60.0s.
Test time limit reached: 60.0s.
Test time limit reached: 60.0s.
Test completed after 11677 closed loops.
Test time limit reached: 60.0s.
Test completed after 11679 closed loops.
Test time limit reached: 60.0s.
Test completed after 11682 closed loops.
Test time limit reached: 60.0s.
Test time limit reached: 60.0s.
Test time limit reached: 60.0s.
Test time limit reached: 60.0s.
Test time limit reached: 60.0s.
Test completed after 11690 closed loops.
Test time limit reached: 60.0s.
Test time limit reached: 60.0s.
Test time limit reached: 60.0s.
Test completed after 11684 closed loops.
Test time limit reached: 60.0s.
Test time limit reached: 60.0s.
Test completed after 11678 closed loops.
Test time limit reached: 60.0s.
Test completed after 11680 closed loops.
Test time limit reached: 60.0s.
Test time limit reached: 60.0s.
Test completed after 11676 closed loops.
Test time limit reached: 60.0s.
Test completed after 11678 closed loops.
Test time limit reached: 60.0s.
Test completed after 11677 closed loops.
Test time limit reached: 60.0s.
Test completed after 11677 closed loops.
Test time limit reached: 60.0s.
Test completed after 11674 closed loops.
Test time limit reached: 60.0s.
Test time limit reached: 60.0s.
Test completed after 11678 closed loops.
Test time limit reached: 60.0s.
Test completed after 11679 closed loops.
Test time limit reached: 60.0s.
Test completed after 11673 closed loops.
Test time limit reached: 60.0s.
Test time limit reached: 60.0s.
Test time limit reached: 60.0s.
Test completed after 11678 closed loops.
Test time limit reached: 60.0s.
Test time limit reached: 60.0s.
Test completed after 11678 closed loops.
Test time limit reached: 60.0s.
Test time limit reached: 60.0s.
Test completed after 11684 closed loops.
Test time limit reached: 60.0s.
Test completed after 11678 closed loops.
Test time limit reached: 60.0s.
Test completed after 11684 closed loops.
Test time limit reached: 60.0s.
Test time limit reached: 60.0s.
Test completed after 11678 closed loops.
Test time limit reached: 60.0s.
Test completed after 11677 closed loops.
Test time limit reached: 60.0s.
Test completed after 11679 closed loops.
Test time limit reached: 60.0s.
Test completed after 11678 closed loops.
Test time limit reached: 60.0s.
Test completed after 11681 closed loops.
Test time limit reached: 60.0s.
Test completed after 11682 closed loops.
Test time limit reached: 60.0s.
Test completed after 11678 closed loops.
Test time limit reached: 60.0s.
Test completed after 11681 closed loops.
Test time limit reached: 60.0s.
Test completed after 11675 closed loops.
Test time limit reached: 60.0s.
Test completed after 11681 closed loops.
Test time limit reached: 60.0s.
Test completed after 11682 closed loops.
Test time limit reached: 60.0s.
Test completed after 11680 closed loops.
Test time limit reached: 60.0s.
Test completed after 11678 closed loops.
Test time limit reached: 60.0s.
Test completed after 11686 closed loops.
Test time limit reached: 60.0s.
Test completed after 11678 closed loops.
Test completed after 11678 closed loops.
Test time limit reached: 60.0s.
Test completed after 11684 closed loops.
Test time limit reached: 60.0s.
Test time limit reached: 60.0s.
Test time limit reached: 60.0s.
Test completed after 11671 closed loops.
Test completed after 11679 closed loops.
Test completed after 11684 closed loops.
Test completed after 11682 closed loops.
Test completed after 11679 closed loops.
Test completed after 11683 closed loops.
Test completed after 11676 closed loops.
Test completed after 11675 closed loops.
Test completed after 11680 closed loops.
Test time limit reached: 60.0s.
Test time limit reached: 60.0s.
Test completed after 11686 closed loops.
Test completed after 11679 closed loops.
Test completed after 11682 closed loops.
Test time limit reached: 60.0s.
Test time limit reached: 60.0s.
Test completed after 11683 closed loops.
Test completed after 11673 closed loops.
Test completed after 11687 closed loops.
Test time limit reached: 60.0s.
Test completed after 11680 closed loops.
Test completed after 11684 closed loops.
Test time limit reached: 60.0s.
Test completed after 11678 closed loops.
Test time limit reached: 60.0s.
Test time limit reached: 60.0s.
Test completed after 11677 closed loops.
Test completed after 11684 closed loops.
Test completed after 11674 closed loops.
Test time limit reached: 60.0s.
Test completed after 11676 closed loops.
Test completed after 11682 closed loops.
Test completed after 11676 closed loops.
Test completed after 11682 closed loops.
Test completed after 11687 closed loops.
Test completed after 11682 closed loops.
[Test isAlive] for 127.0.0.1:43100
[ ... Completed in 0.001 seconds ]
Local Single node: Test Status: TEST_PASSED
Clients: 64
Total Requests (without replies): 1494969
Total Requests (including replies): 1494969
Successful Responses: 1494969
Timeout Responses: 0
Total Failed Responses: 0
% Success: 100.0
% Timeout: 0.0
% Failed: 0.0
Response Time (Successful requests):
[max] 14ms, [min] 0ms
[avg] 2.547884270509957 ms, [stdev] 0.8855694620435495ms
Throughput: 24913.658634136587 requests per second
Goodput: 24913.658634136587 requests per second
Retry Rate (Successful requests): 1.0
Sending WIPE-OUT to all servers...
[ ... Completed in 0.021 seconds ]
[OK]
Waiting for 5 seconds for the cache to be cleared!
[ ... TEST Completed in 65.769 seconds ]

[ TEST Single Client Capacity (Value Size = 500 bytes, Limit = 146800 PUTs, ~69.99969482421875 MiB) : PUT -> GET (closed loop) ]
Sending WIPE-OUT to all servers...
[ ... Completed in 0.007 seconds ]
[OK]
Waiting for 5 seconds for the cache to be cleared!
ERROR_OUT_OF_SPACE
Received error code after 85084 closed loops (~40.57121276855469 MiB).
PUTs sent = 85084(~40.57121276855469 MiB) , Limit = 146800
[Test isAlive]
Test Passed. Server is ok.
TEST_PASSED
[Info] Shutting down local server(s)!
[Test isAlive] for 127.0.0.1:43100
[ ... Completed in 10.005 seconds ]
[Info] Local tests completed in 257.65 seconds
[Info] Finished tests for the local deployment!
Sending GET PID to all servers (v2) ... 
>>> Server node 3.99.144.144:12347 has process ID 5110.
>>> Server node 3.99.144.144:12348 has process ID 5114.
>>> Server node 3.99.144.144:12349 has process ID 5119.
>>> Server node 3.99.144.144:12350 has process ID 5124.
>>> Server node 3.99.144.144:12351 has process ID 5129.
>>> Server node 3.99.144.144:12352 has process ID 5134.
>>> Server node 3.99.144.144:12353 has process ID 5139.
>>> Server node 3.99.144.144:12354 has process ID 5144.
>>> Server node 3.99.144.144:12355 has process ID 5149.
>>> Server node 3.99.144.144:12356 has process ID 5157.
>>> Server node 3.99.144.144:12357 has process ID 5162.
>>> Server node 3.99.144.144:12358 has process ID 5169.
>>> Server node 3.99.144.144:12359 has process ID 5174.
>>> Server node 3.99.144.144:12360 has process ID 5180.
>>> Server node 3.99.144.144:12361 has process ID 5194.
>>> Server node 3.99.144.144:12362 has process ID 5208.
>>> Server node 3.99.144.144:12363 has process ID 5221.
>>> Server node 3.99.144.144:12364 has process ID 5230.
>>> Server node 3.99.144.144:12365 has process ID 5242.
>>> Server node 3.99.144.144:12366 has process ID 5258.
>>> Server node 3.99.144.144:12367 has process ID 5270.
>>> Server node 3.99.144.144:12368 has process ID 5303.
>>> Server node 3.99.144.144:12369 has process ID 5316.
>>> Server node 3.99.144.144:12370 has process ID 5341.
>>> Server node 3.99.144.144:12371 has process ID 5355.
>>> Server node 3.99.144.144:12372 has process ID 5381.
>>> Server node 3.99.144.144:12373 has process ID 5400.
>>> Server node 3.99.144.144:12374 has process ID 5417.
>>> Server node 3.99.144.144:12375 has process ID 5448.
>>> Server node 3.99.144.144:12376 has process ID 5462.
>>> Server node 3.99.144.144:12377 has process ID 5491.
>>> Server node 3.99.144.144:12378 has process ID 5503.
>>> Server node 3.99.144.144:12379 has process ID 5523.
>>> Server node 3.99.144.144:12380 has process ID 5542.
>>> Server node 3.99.144.144:12381 has process ID 5562.
>>> Server node 3.99.144.144:12382 has process ID 5597.
>>> Server node 3.99.144.144:12383 has process ID 5607.
>>> Server node 3.99.144.144:12384 has process ID 5644.
>>> Server node 3.99.144.144:12385 has process ID 5662.
>>> Server node 3.99.144.144:12386 has process ID 5690.
[ ... Completed in 71.327 seconds ]
[OK]
Testing memory usage 
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12347 has memory limit of 512M.
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12348 has memory limit of 512M.
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12349 has memory limit of 512M.
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12350 has memory limit of 512M.
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12351 has memory limit of 512M.
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12352 has memory limit of 512M.
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12353 has memory limit of 512M.
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12354 has memory limit of 512M.
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12355 has memory limit of 512M.
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12356 has memory limit of 512M.
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12357 has memory limit of 512M.
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12358 has memory limit of 512M.
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12359 has memory limit of 512M.
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12360 has memory limit of 512M.
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12361 has memory limit of 512M.
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12362 has memory limit of 512M.
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12363 has memory limit of 512M.
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12364 has memory limit of 512M.
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12365 has memory limit of 512M.
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12366 has memory limit of 512M.
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12367 has memory limit of 512M.
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12368 has memory limit of 512M.
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12369 has memory limit of 512M.
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12370 has memory limit of 512M.
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12371 has memory limit of 512M.
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12372 has memory limit of 512M.
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12373 has memory limit of 512M.
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12374 has memory limit of 512M.
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12375 has memory limit of 512M.
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12376 has memory limit of 512M.
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12377 has memory limit of 512M.
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12378 has memory limit of 512M.
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12379 has memory limit of 512M.
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12380 has memory limit of 512M.
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12381 has memory limit of 512M.
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12382 has memory limit of 512M.
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12383 has memory limit of 512M.
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12384 has memory limit of 512M.
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12385 has memory limit of 512M.
Command information:
c5a.16xlarge 
 Xmx512m
>>> Server node 3.99.144.144:12386 has memory limit of 512M.
[ ... Completed in 61.639 seconds ]
[OK]

Sending GetMembershipCount to all servers... 
[Info] Server 3.99.144.144:12347 replied with membership count = 40 (expected count = 40)
[Info] Server 3.99.144.144:12348 replied with membership count = 40 (expected count = 40)
[Info] Server 3.99.144.144:12350 replied with membership count = 40 (expected count = 40)
[Info] Server 3.99.144.144:12349 replied with membership count = 40 (expected count = 40)
[Info] Server 3.99.144.144:12352 replied with membership count = 40 (expected count = 40)
[Info] Server 3.99.144.144:12351 replied with membership count = 40 (expected count = 40)
[Info] Server 3.99.144.144:12353 replied with membership count = 40 (expected count = 40)
[Info] Server 3.99.144.144:12355 replied with membership count = 40 (expected count = 40)
[Info] Server 3.99.144.144:12354 replied with membership count = 40 (expected count = 40)
[Info] Server 3.99.144.144:12357 replied with membership count = 40 (expected count = 40)
[Info] Server 3.99.144.144:12356 replied with membership count = 40 (expected count = 40)
[Info] Server 3.99.144.144:12359 replied with membership count = 40 (expected count = 40)
[Info] Server 3.99.144.144:12358 replied with membership count = 40 (expected count = 40)
[Info] Server 3.99.144.144:12360 replied with membership count = 40 (expected count = 40)
[Info] Server 3.99.144.144:12362 replied with membership count = 40 (expected count = 40)
[Info] Server 3.99.144.144:12361 replied with membership count = 40 (expected count = 40)
[Info] Server 3.99.144.144:12363 replied with membership count = 40 (expected count = 40)
[Info] Server 3.99.144.144:12364 replied with membership count = 40 (expected count = 40)
[Info] Server 3.99.144.144:12366 replied with membership count = 40 (expected count = 40)
[Info] Server 3.99.144.144:12365 replied with membership count = 40 (expected count = 40)
[Info] Server 3.99.144.144:12368 replied with membership count = 40 (expected count = 40)
[Info] Server 3.99.144.144:12367 replied with membership count = 40 (expected count = 40)
[Info] Server 3.99.144.144:12369 replied with membership count = 40 (expected count = 40)
[Info] Server 3.99.144.144:12371 replied with membership count = 40 (expected count = 40)
[Info] Server 3.99.144.144:12372 replied with membership count = 40 (expected count = 40)
[Info] Server 3.99.144.144:12373 replied with membership count = 40 (expected count = 40)
[Info] Server 3.99.144.144:12374 replied with membership count = 40 (expected count = 40)
[Info] Server 3.99.144.144:12375 replied with membership count = 40 (expected count = 40)
[Info] Server 3.99.144.144:12376 replied with membership count = 40 (expected count = 40)
[Info] Server 3.99.144.144:12378 replied with membership count = 40 (expected count = 40)
[Info] Server 3.99.144.144:12377 replied with membership count = 40 (expected count = 40)
[Info] Server 3.99.144.144:12379 replied with membership count = 40 (expected count = 40)
[Info] Server 3.99.144.144:12380 replied with membership count = 40 (expected count = 40)
[Info] Server 3.99.144.144:12381 replied with membership count = 40 (expected count = 40)
[Info] Server 3.99.144.144:12382 replied with membership count = 40 (expected count = 40)
[Info] Server 3.99.144.144:12383 replied with membership count = 40 (expected count = 40)
[Info] Server 3.99.144.144:12384 replied with membership count = 40 (expected count = 40)
[Info] Server 3.99.144.144:12385 replied with membership count = 40 (expected count = 40)
[Info] Server 3.99.144.144:12386 replied with membership count = 40 (expected count = 40)
[Info] Server 3.99.144.144:12370 replied with membership count = 40 (expected count = 40)
[ ... Completed in 0.05 seconds ]
[Ok] All servers returned the expected membership count (40)
Sending WIPE-OUT to all servers...
[ ... Completed in 0.083 seconds ]
[OK]
Waiting for 5 seconds for the cache to be cleared!
[Test isAlive] for 3.99.144.144:12347

[ TEST Basic PUT/GET (PUT -> GET). Value Length = 8 Bytes ]
TEST_PASSED

[ TEST Basic PUT/GET (PUT -> GET). Value Length = 32 Bytes ]
TEST_PASSED

[ TEST Basic PUT/GET (PUT -> GET). Value Length = 512 Bytes ]
TEST_PASSED

[ TEST Basic PUT/GET (PUT -> GET). Value Length = 1024 Bytes ]
TEST_PASSED
Sending WIPE-OUT to all servers...
[ ... Completed in 0.019 seconds ]
[OK]
Waiting for 5 seconds for the cache to be cleared!


[ TEST Basic KVS with message version (PUT Version 1 -> GET -> PUT Version 2 -> GET). Value Length = 8 Bytes ]
TEST_PASSED


[ TEST Basic KVS with message version (PUT Version 1 -> GET -> PUT Version 2 -> GET). Value Length = 32 Bytes ]
TEST_PASSED


[ TEST Basic KVS with message version (PUT Version 1 -> GET -> PUT Version 2 -> GET). Value Length = 512 Bytes ]
TEST_PASSED


[ TEST Basic KVS with message version (PUT Version 1 -> GET -> PUT Version 2 -> GET). Value Length = 1024 Bytes ]
TEST_PASSED
Sending WIPE-OUT to all servers...
[ ... Completed in 0.02 seconds ]
[OK]
Waiting for 5 seconds for the cache to be cleared!

[ TEST Basic REMOVE (PUT -> GET -> REM -> GET). Value Length = 32 Bytes. ]
TEST_PASSED
Sending WIPE-OUT to all servers...
[ ... Completed in 0.023 seconds ]
[OK]
Waiting for 5 seconds for the cache to be cleared!

[ TEST Double remove (PUT -> REM -> REM). Value Length = 32 Bytes. ]
TEST_PASSED
Sending WIPE-OUT to all servers...
[ ... Completed in 0.051 seconds ]
[OK]
Waiting for 5 seconds for the cache to be cleared!

[ TEST Invalid Command. ]
TEST_PASSED
Sending WIPE-OUT to all servers...
[ ... Completed in 0.019 seconds ]
[OK]
Waiting for 5 seconds for the cache to be cleared!

[ TEST At-Most-Once Client Policy (PUT -> REM -> REM). Value length =  32 Bytes ]
TEST_PASSED
Sending WIPE-OUT to all servers...
[ ... Completed in 0.02 seconds ]
[OK]
Waiting for 5 seconds for the cache to be cleared!

[ TEST At-Most-Once Client Policy Empty REMs (EMPTY -> REM -> REM) ]
TEST_PASSED
Sending WIPE-OUT to all servers...
[ ... Completed in 0.018 seconds ]
[OK]
Waiting for 5 seconds for the cache to be cleared!

[ TEST At-Most-Once Client Policy (PUT -> REM -> many PUT -> REM). Value length =  32 Bytes ]
Pausing for 10 seconds to wait for the cache to be cleared!
TEST_PASSED

[ TEST At-Most-Once Client Policy Error Return Code (REM -> PUT -> REM) ]
TEST_PASSED
Sending WIPE-OUT to all servers...
[ ... Completed in 0.02 seconds ]
[OK]
Waiting for 5 seconds for the cache to be cleared!

[ TEST At-Most-Once Client Policy For PUTs (PUT1 -> PUT2 -> PUT1 -> GET2) ]
TEST_PASSED
Sending WIPE-OUT to all servers...
[ ... Completed in 0.048 seconds ]
[OK]
Waiting for 5 seconds for the cache to be cleared!

[TEST Invalid Key ]
TEST_PASSED
Sending WIPE-OUT to all servers...
[ ... Completed in 0.022 seconds ]
[OK]
Waiting for 5 seconds for the cache to be cleared!

[TEST Invalid Value ]
TEST_PASSED
Sending WIPE-OUT to all servers...
[ ... Completed in 0.02 seconds ]
[OK]
Waiting for 5 seconds for the cache to be cleared!

[ TEST Multi Client Crash: PUT -> GET (closed loop) + REM -> WIPEOUT (closed loop) ]
[Test isAlive]
Test Passed. Server is ok.
TEST_PASSED
Sending WIPE-OUT to all servers...
[ ... Completed in 0.019 seconds ]
[OK]
Waiting for 5 seconds for the cache to be cleared!
[ ... TEST Completed in 78.789 seconds ]

[ TEST Random Front-End Basic (PUT -> GET -> REM -> REM -> GET). Value Length = 8 Bytes ]
TEST_PASSED

[ TEST Random Front-End Basic (PUT -> GET -> REM -> REM -> GET). Value Length = 32 Bytes ]
TEST_PASSED

[ TEST Random Front-End Basic (PUT -> GET -> REM -> REM -> GET). Value Length = 512 Bytes ]
TEST_PASSED

[ TEST Random Front-End Basic (PUT -> GET -> REM -> REM -> GET). Value Length = 1024 Bytes ]
TEST_PASSED
[ ... TEST Completed in 0.28 seconds ]
Sending WIPE-OUT to all servers...
[ ... Completed in 0.02 seconds ]
[OK]
Waiting for 5 seconds for the cache to be cleared!
[Info] Pausing for 6 seconds to wait for the cache to be cleared!

[ TEST Random Front-End Performance (1 clients, 60.0 seconds): PUT -> GET (closed loop) ]
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12349
[Test isAlive] for 3.99.144.144:12350
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12353
[Test isAlive] for 3.99.144.144:12354
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12357
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12364
[Test isAlive] for 3.99.144.144:12365
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12367
[Test isAlive] for 3.99.144.144:12368
[Test isAlive] for 3.99.144.144:12369
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12371
[Test isAlive] for 3.99.144.144:12372
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12377
[Test isAlive] for 3.99.144.144:12378
[Test isAlive] for 3.99.144.144:12379
[Test isAlive] for 3.99.144.144:12380
[Test isAlive] for 3.99.144.144:12381
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12382
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12385
[Test isAlive] for 3.99.144.144:12386
[ ... Completed in 0.048 seconds ]
TEST_PASSED
Stage 1 Random Front-End: Test Status: TEST_PASSED
Clients: 1
Total Requests (without replies): 3599
Total Requests (including replies): 3643
Successful Responses: 3599
Timeout Responses: 0
Total Failed Responses: 0
% Success: 100.0
% Timeout: 0.0
% Failed: 0.0
Response Time (Successful requests):
[max] 63ms, [min] 5ms
[avg] 16.626285079188662 ms, [stdev] 7.255476769973824ms
Throughput: 59.96434462420233 requests per second
Goodput: 59.96434462420233 requests per second
Retry Rate (Successful requests): 1.0122256182272853
Sending WIPE-OUT to all servers...
[ ... Completed in 0.049 seconds ]
[OK]
Waiting for 5 seconds for the cache to be cleared!
[ ... TEST Completed in 65.127 seconds ]
[Info] Pausing for 6 seconds to wait for the cache to be cleared!

[ TEST Random Front-End Performance (16 clients, 60.0 seconds): PUT -> GET (closed loop) ]
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12349
[Test isAlive] for 3.99.144.144:12350
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12353
[Test isAlive] for 3.99.144.144:12354
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12357
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12364
[Test isAlive] for 3.99.144.144:12365
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12367
[Test isAlive] for 3.99.144.144:12368
[Test isAlive] for 3.99.144.144:12369
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12371
[Test isAlive] for 3.99.144.144:12372
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12377
[Test isAlive] for 3.99.144.144:12378
[Test isAlive] for 3.99.144.144:12379
[Test isAlive] for 3.99.144.144:12380
[Test isAlive] for 3.99.144.144:12381
[Test isAlive] for 3.99.144.144:12382
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12385
[Test isAlive] for 3.99.144.144:12386
[ ... Completed in 0.01 seconds ]
TEST_PASSED
Stage 1 Random Front-End: Test Status: TEST_PASSED
Clients: 16
Total Requests (without replies): 59876
Total Requests (including replies): 60620
Successful Responses: 59876
Timeout Responses: 0
Total Failed Responses: 0
% Success: 100.0
% Timeout: 0.0
% Failed: 0.0
Response Time (Successful requests):
[max] 141ms, [min] 5ms
[avg] 16.004509319259803 ms, [stdev] 7.134648766053801ms
Throughput: 997.3681580437752 requests per second
Goodput: 997.3681580437752 requests per second
Retry Rate (Successful requests): 1.0124256797381255
Sending WIPE-OUT to all servers...
[ ... Completed in 0.026 seconds ]
[OK]
Waiting for 5 seconds for the cache to be cleared!
[ ... TEST Completed in 65.199 seconds ]
[Info] Pausing for 6 seconds to wait for the cache to be cleared!

[ TEST Random Front-End Performance (256 clients, 60.0 seconds): PUT -> GET (closed loop) ]
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12349
[Test isAlive] for 3.99.144.144:12350
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12353
[Test isAlive] for 3.99.144.144:12354
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12357
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12364
[Test isAlive] for 3.99.144.144:12365
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12367
[Test isAlive] for 3.99.144.144:12368
[Test isAlive] for 3.99.144.144:12369
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12371
[Test isAlive] for 3.99.144.144:12372
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12377
[Test isAlive] for 3.99.144.144:12378
[Test isAlive] for 3.99.144.144:12379
[Test isAlive] for 3.99.144.144:12380
[Test isAlive] for 3.99.144.144:12381
[Test isAlive] for 3.99.144.144:12382
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12385
[Test isAlive] for 3.99.144.144:12386
[ ... Completed in 0.01 seconds ]
TEST_PASSED
[ CPU load (loadavg, last one minute) 0.43 ]
Stage 1 Random Front-End: Test Status: TEST_PASSED
Clients: 256
Total Requests (without replies): 964620
Total Requests (including replies): 976791
Successful Responses: 964620
Timeout Responses: 0
Total Failed Responses: 0
% Success: 100.0
% Timeout: 0.0
% Failed: 0.0
Response Time (Successful requests):
[max] 301ms, [min] 5ms
[avg] 15.903286268167776 ms, [stdev] 7.216297607076377ms
Throughput: 16061.206480294377 requests per second
Goodput: 16061.206480294377 requests per second
Retry Rate (Successful requests): 1.0126174037444797
Sending WIPE-OUT to all servers...
[ ... Completed in 0.046 seconds ]
[OK]
Waiting for 5 seconds for the cache to be cleared!
[ ... TEST Completed in 65.47 seconds ]
[Info] Pausing for 6 seconds to wait for the cache to be cleared!

[ TEST Random Front-End Performance (1024 clients, 60.0 seconds): PUT -> GET (closed loop) ]
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12349
[Test isAlive] for 3.99.144.144:12350
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12353
[Test isAlive] for 3.99.144.144:12354
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12357
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12364
[Test isAlive] for 3.99.144.144:12365
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12367
[Test isAlive] for 3.99.144.144:12368
[Test isAlive] for 3.99.144.144:12369
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12371
[Test isAlive] for 3.99.144.144:12372
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12377
[Test isAlive] for 3.99.144.144:12378
[Test isAlive] for 3.99.144.144:12379
[Test isAlive] for 3.99.144.144:12380
[Test isAlive] for 3.99.144.144:12381
[Test isAlive] for 3.99.144.144:12382
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12385
[Test isAlive] for 3.99.144.144:12386
[ ... Completed in 0.048 seconds ]
TEST_PASSED
[ CPU load (loadavg, last one minute) 4.35 ]
Stage 1 Random Front-End: Test Status: TEST_PASSED
Clients: 1024
Total Requests (without replies): 2392904
Total Requests (including replies): 2759840
Successful Responses: 2390694
Timeout Responses: 2210
Total Failed Responses: 0
% Success: 99.90764359957608
% Timeout: 0.09235640042392006
% Failed: 0.0
Response Time (Successful requests):
[max] 585ms, [min] 5ms
[avg] 24.735254281811056 ms, [stdev] 41.726413359644404ms
Throughput: 39341.44416677627 requests per second
Goodput: 39305.10982506905 requests per second
Retry Rate (Successful requests): 1.1533433852757988
Sending WIPE-OUT to all servers...
[ ... Completed in 0.044 seconds ]
[OK]
Waiting for 5 seconds for the cache to be cleared!
[ ... TEST Completed in 67.295 seconds ]
[Info] Pausing for 6 seconds to wait for the cache to be cleared!

[ TEST Random Front-End Performance (2048 clients, 60.0 seconds): PUT -> GET (closed loop) ]
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12349
[Test isAlive] for 3.99.144.144:12350
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12353
[Test isAlive] for 3.99.144.144:12354
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12357
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12364
[Test isAlive] for 3.99.144.144:12365
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12367
[Test isAlive] for 3.99.144.144:12368
[Test isAlive] for 3.99.144.144:12369
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12371
[Test isAlive] for 3.99.144.144:12372
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12377
[Test isAlive] for 3.99.144.144:12378
[Test isAlive] for 3.99.144.144:12379
[Test isAlive] for 3.99.144.144:12380
[Test isAlive] for 3.99.144.144:12381
[Test isAlive] for 3.99.144.144:12382
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12385
[Test isAlive] for 3.99.144.144:12386
[ ... Completed in 0.012 seconds ]
TEST_PASSED
[ CPU load (loadavg, last one minute) 8.44 ]
Stage 1 Random Front-End: Test Status: TEST_PASSED
Clients: 2048
Total Requests (without replies): 2047263
Total Requests (including replies): 2937533
Successful Responses: 2015301
Timeout Responses: 31962
Total Failed Responses: 0
% Success: 98.43879364790943
% Timeout: 1.5612063520905717
% Failed: 0.0
Response Time (Successful requests):
[max] 589ms, [min] 5ms
[avg] 45.038097534809935 ms, [stdev] 90.28374099317577ms
Throughput: 33626.184649245275 requests per second
Goodput: 33101.210518535554 requests per second
Retry Rate (Successful requests): 1.4348586380938844
Sending WIPE-OUT to all servers...
[ ... Completed in 0.037 seconds ]
[OK]
Waiting for 5 seconds for the cache to be cleared!
[ ... TEST Completed in 66.711 seconds ]
[Info] Pausing for 6 seconds to wait for the cache to be cleared!

[ TEST Concurrent Store/Load v2; Stage 0; (Clients = 64, Limit = 40000 PUTs) ]
Issuing PUTS ... 
[DONE in 13.801s]
[Info] Concurrent Store/Load v2; Stage 0; PUT Phase; Of 40000 stored keys: 40000 SUCCESS, 0 TIMEOUT, 0 KEY-NOT-FOUND, 0 ERROR-UID, 0 ERROR-INVALID-GET (value or version), 0 ERROR-OTHER.
[Info] Concurrent Store/Load v2; Stage 0; PUT Phase; Expected Key Loss: 0.0%. Actual Key Loss: 0.0%
[Info] Concurrent Store/Load v2; Stage 0; PUT Phase; Total Value Size (MiB): 19.120856285095215 MiB.
[Info] Concurrent Store/Load v2; Stage 0; PUT Phase; SUCCESS Response Count: 40000.0.
[Info] Concurrent Store/Load v2; Stage 0; PUT Phase; % SUCCESS: 100.0%.
[Info] Concurrent Store/Load v2; Stage 0; PUT Phase; % FAILED (Not SUCCESS): 0.0%.
[Info] Concurrent Store/Load v2; Stage 0; PUT Phase; % TIMEOUT: 0.0%.
[Info] Concurrent Store/Load v2; Stage 0; PUT Phase; % KEY-NOT-FOUND: 0.0%.
[Info] Concurrent Store/Load v2; Stage 0; PUT Phase; % ERROR-UID: 0.0%.
[Info] Concurrent Store/Load v2; Stage 0; PUT Phase; % ERROR-INVALID-GET (value or version): 0.0%.
[Info] Concurrent Store/Load v2; Stage 0; PUT Phase; % ERROR-OTHER: 0.0%.
[Info] PUTs % SUCCESS Ok: 100.0% Success.(90.0% minimum required).
Confirming keys were stored correctly.
Issuing GETs ... 
[DONE in 6.758s]
[Info] Concurrent Store/Load v2; Stage 0; GET Phase; Of 40000 stored keys: 40000 SUCCESS, 0 TIMEOUT, 0 KEY-NOT-FOUND, 0 ERROR-UID, 0 ERROR-INVALID-GET (value or version), 0 ERROR-OTHER.
[Info] Concurrent Store/Load v2; Stage 0; GET Phase; Expected Key Loss: 0.0%. Actual Key Loss: 0.0%
[Info] Concurrent Store/Load v2; Stage 0; GET Phase; Total Value Size (MiB): 19.120856285095215 MiB.
[Info] Concurrent Store/Load v2; Stage 0; GET Phase; SUCCESS Response Count: 40000.0.
[Info] Concurrent Store/Load v2; Stage 0; GET Phase; % SUCCESS: 100.0%.
[Info] Concurrent Store/Load v2; Stage 0; GET Phase; % FAILED (Not SUCCESS): 0.0%.
[Info] Concurrent Store/Load v2; Stage 0; GET Phase; % TIMEOUT: 0.0%.
[Info] Concurrent Store/Load v2; Stage 0; GET Phase; % KEY-NOT-FOUND: 0.0%.
[Info] Concurrent Store/Load v2; Stage 0; GET Phase; % ERROR-UID: 0.0%.
[Info] Concurrent Store/Load v2; Stage 0; GET Phase; % ERROR-INVALID-GET (value or version): 0.0%.
[Info] Concurrent Store/Load v2; Stage 0; GET Phase; % ERROR-OTHER: 0.0%.
[Info] GETs % SUCCESS Ok: 100.0% Success.(90.0% minimum required).
Checking that 40 nodes are alive
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12349
[Test isAlive] for 3.99.144.144:12350
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12353
[Test isAlive] for 3.99.144.144:12354
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12357
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12364
[Test isAlive] for 3.99.144.144:12365
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12367
[Test isAlive] for 3.99.144.144:12368
[Test isAlive] for 3.99.144.144:12369
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12371
[Test isAlive] for 3.99.144.144:12372
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12377
[Test isAlive] for 3.99.144.144:12378
[Test isAlive] for 3.99.144.144:12379
[Test isAlive] for 3.99.144.144:12380
[Test isAlive] for 3.99.144.144:12381
[Test isAlive] for 3.99.144.144:12382
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12385
[Test isAlive] for 3.99.144.144:12386
[ ... Completed in 0.012 seconds ]
TEST_PASSED
[ ... TEST Completed in 20.584 seconds ]

[Info] Preparing for Stage 1 of the Suspend Test
[Info] To suspend 24 node(s) in Stage 1.
[Info] Will suspend 3.99.144.144:12366
[Info] Will suspend 3.99.144.144:12355
[Info] Will suspend 3.99.144.144:12383
[Info] Will suspend 3.99.144.144:12384
[Info] Will keep alive 3.99.144.144:12367
[Info] Will keep alive 3.99.144.144:12354
[Info] Will keep alive 3.99.144.144:12348
[Info] Will keep alive 3.99.144.144:12374
[Info] Will keep alive 3.99.144.144:12359
[Info] Will keep alive 3.99.144.144:12352
[Info] Will keep alive 3.99.144.144:12353
[Info] Will keep alive 3.99.144.144:12356
[Info] Will keep alive 3.99.144.144:12347
[Info] Will keep alive 3.99.144.144:12375
[Info] Will keep alive 3.99.144.144:12377
[Info] Will keep alive 3.99.144.144:12379
[Info] Will keep alive 3.99.144.144:12386
[Info] Will keep alive 3.99.144.144:12380
[Info] Will keep alive 3.99.144.144:12357
[Info] Will keep alive 3.99.144.144:12371
[Info] Will keep alive 3.99.144.144:12361
[Info] Will keep alive 3.99.144.144:12365
[Info] Will keep alive 3.99.144.144:12350
[Info] Will keep alive 3.99.144.144:12382
[Info] Will keep alive 3.99.144.144:12360
[Info] Will keep alive 3.99.144.144:12364
[Info] Will keep alive 3.99.144.144:12376
[Info] Will keep alive 3.99.144.144:12349
[Info] Will keep alive 3.99.144.144:12358
[Info] Will keep alive 3.99.144.144:12372
[Info] Will keep alive 3.99.144.144:12351
[Info] Will keep alive 3.99.144.144:12378
[Info] Will keep alive 3.99.144.144:12368
[Info] Will keep alive 3.99.144.144:12373
[Info] Will keep alive 3.99.144.144:12362
[Info] Will keep alive 3.99.144.144:12363
[Info] Will keep alive 3.99.144.144:12370
[Info] Will keep alive 3.99.144.144:12369
[Info] Will keep alive 3.99.144.144:12381
[Info] Will keep alive 3.99.144.144:12385
Waiting 10 seconds before suspending nodes
[Info] Suspending 4 servers.
Server 3.99.144.144:12366 suspended
Server 3.99.144.144:12355 suspended
Server 3.99.144.144:12383 suspended
Server 3.99.144.144:12384 suspended
[ ... Completed in 5.406 seconds ]
Waiting 30 seconds after suspending nodes
[Info] Stage 1 Scenario: Alive Nodes: 36. Dead Nodes: 4
[Info] Will suspend 3.99.144.144:12367
[Info] Will suspend 3.99.144.144:12354
[Info] Will suspend 3.99.144.144:12348
[Info] Will suspend 3.99.144.144:12374
[Info] Will keep alive 3.99.144.144:12359
[Info] Will keep alive 3.99.144.144:12352
[Info] Will keep alive 3.99.144.144:12353
[Info] Will keep alive 3.99.144.144:12356
[Info] Will keep alive 3.99.144.144:12347
[Info] Will keep alive 3.99.144.144:12375
[Info] Will keep alive 3.99.144.144:12377
[Info] Will keep alive 3.99.144.144:12379
[Info] Will keep alive 3.99.144.144:12386
[Info] Will keep alive 3.99.144.144:12380
[Info] Will keep alive 3.99.144.144:12357
[Info] Will keep alive 3.99.144.144:12371
[Info] Will keep alive 3.99.144.144:12361
[Info] Will keep alive 3.99.144.144:12365
[Info] Will keep alive 3.99.144.144:12350
[Info] Will keep alive 3.99.144.144:12382
[Info] Will keep alive 3.99.144.144:12360
[Info] Will keep alive 3.99.144.144:12364
[Info] Will keep alive 3.99.144.144:12376
[Info] Will keep alive 3.99.144.144:12349
[Info] Will keep alive 3.99.144.144:12358
[Info] Will keep alive 3.99.144.144:12372
[Info] Will keep alive 3.99.144.144:12351
[Info] Will keep alive 3.99.144.144:12378
[Info] Will keep alive 3.99.144.144:12368
[Info] Will keep alive 3.99.144.144:12373
[Info] Will keep alive 3.99.144.144:12362
[Info] Will keep alive 3.99.144.144:12363
[Info] Will keep alive 3.99.144.144:12370
[Info] Will keep alive 3.99.144.144:12369
[Info] Will keep alive 3.99.144.144:12381
[Info] Will keep alive 3.99.144.144:12385
Waiting 10 seconds before suspending nodes
[Info] Suspending 4 servers.
Server 3.99.144.144:12367 suspended
Server 3.99.144.144:12354 suspended
Server 3.99.144.144:12374 suspended
Server 3.99.144.144:12348 suspended
[ ... Completed in 5.424 seconds ]
Waiting 30 seconds after suspending nodes
[Info] Stage 1 Scenario: Alive Nodes: 32. Dead Nodes: 8
[Info] Will suspend 3.99.144.144:12359
[Info] Will suspend 3.99.144.144:12352
[Info] Will suspend 3.99.144.144:12353
[Info] Will suspend 3.99.144.144:12356
[Info] Will keep alive 3.99.144.144:12347
[Info] Will keep alive 3.99.144.144:12375
[Info] Will keep alive 3.99.144.144:12377
[Info] Will keep alive 3.99.144.144:12379
[Info] Will keep alive 3.99.144.144:12386
[Info] Will keep alive 3.99.144.144:12380
[Info] Will keep alive 3.99.144.144:12357
[Info] Will keep alive 3.99.144.144:12371
[Info] Will keep alive 3.99.144.144:12361
[Info] Will keep alive 3.99.144.144:12365
[Info] Will keep alive 3.99.144.144:12350
[Info] Will keep alive 3.99.144.144:12382
[Info] Will keep alive 3.99.144.144:12360
[Info] Will keep alive 3.99.144.144:12364
[Info] Will keep alive 3.99.144.144:12376
[Info] Will keep alive 3.99.144.144:12349
[Info] Will keep alive 3.99.144.144:12358
[Info] Will keep alive 3.99.144.144:12372
[Info] Will keep alive 3.99.144.144:12351
[Info] Will keep alive 3.99.144.144:12378
[Info] Will keep alive 3.99.144.144:12368
[Info] Will keep alive 3.99.144.144:12373
[Info] Will keep alive 3.99.144.144:12362
[Info] Will keep alive 3.99.144.144:12363
[Info] Will keep alive 3.99.144.144:12370
[Info] Will keep alive 3.99.144.144:12369
[Info] Will keep alive 3.99.144.144:12381
[Info] Will keep alive 3.99.144.144:12385
Waiting 10 seconds before suspending nodes
[Info] Suspending 4 servers.
Server 3.99.144.144:12352 suspended
Server 3.99.144.144:12359 suspended
Server 3.99.144.144:12356 suspended
Server 3.99.144.144:12353 suspended
[ ... Completed in 5.412 seconds ]
Waiting 30 seconds after suspending nodes
[Info] Stage 1 Scenario: Alive Nodes: 28. Dead Nodes: 12
[Info] Will suspend 3.99.144.144:12347
[Info] Will suspend 3.99.144.144:12375
[Info] Will suspend 3.99.144.144:12377
[Info] Will suspend 3.99.144.144:12379
[Info] Will keep alive 3.99.144.144:12386
[Info] Will keep alive 3.99.144.144:12380
[Info] Will keep alive 3.99.144.144:12357
[Info] Will keep alive 3.99.144.144:12371
[Info] Will keep alive 3.99.144.144:12361
[Info] Will keep alive 3.99.144.144:12365
[Info] Will keep alive 3.99.144.144:12350
[Info] Will keep alive 3.99.144.144:12382
[Info] Will keep alive 3.99.144.144:12360
[Info] Will keep alive 3.99.144.144:12364
[Info] Will keep alive 3.99.144.144:12376
[Info] Will keep alive 3.99.144.144:12349
[Info] Will keep alive 3.99.144.144:12358
[Info] Will keep alive 3.99.144.144:12372
[Info] Will keep alive 3.99.144.144:12351
[Info] Will keep alive 3.99.144.144:12378
[Info] Will keep alive 3.99.144.144:12368
[Info] Will keep alive 3.99.144.144:12373
[Info] Will keep alive 3.99.144.144:12362
[Info] Will keep alive 3.99.144.144:12363
[Info] Will keep alive 3.99.144.144:12370
[Info] Will keep alive 3.99.144.144:12369
[Info] Will keep alive 3.99.144.144:12381
[Info] Will keep alive 3.99.144.144:12385
Waiting 10 seconds before suspending nodes
[Info] Suspending 4 servers.
Server 3.99.144.144:12377 suspended
Server 3.99.144.144:12347 suspended
Server 3.99.144.144:12375 suspended
Server 3.99.144.144:12379 suspended
[ ... Completed in 5.431 seconds ]
Waiting 30 seconds after suspending nodes
[Info] Stage 1 Scenario: Alive Nodes: 24. Dead Nodes: 16
[Info] Will suspend 3.99.144.144:12386
[Info] Will suspend 3.99.144.144:12380
[Info] Will suspend 3.99.144.144:12357
[Info] Will suspend 3.99.144.144:12371
[Info] Will keep alive 3.99.144.144:12361
[Info] Will keep alive 3.99.144.144:12365
[Info] Will keep alive 3.99.144.144:12350
[Info] Will keep alive 3.99.144.144:12382
[Info] Will keep alive 3.99.144.144:12360
[Info] Will keep alive 3.99.144.144:12364
[Info] Will keep alive 3.99.144.144:12376
[Info] Will keep alive 3.99.144.144:12349
[Info] Will keep alive 3.99.144.144:12358
[Info] Will keep alive 3.99.144.144:12372
[Info] Will keep alive 3.99.144.144:12351
[Info] Will keep alive 3.99.144.144:12378
[Info] Will keep alive 3.99.144.144:12368
[Info] Will keep alive 3.99.144.144:12373
[Info] Will keep alive 3.99.144.144:12362
[Info] Will keep alive 3.99.144.144:12363
[Info] Will keep alive 3.99.144.144:12370
[Info] Will keep alive 3.99.144.144:12369
[Info] Will keep alive 3.99.144.144:12381
[Info] Will keep alive 3.99.144.144:12385
Waiting 10 seconds before suspending nodes
[Info] Suspending 4 servers.
Server 3.99.144.144:12357 suspended
Server 3.99.144.144:12386 suspended
Server 3.99.144.144:12380 suspended
Server 3.99.144.144:12371 suspended
[ ... Completed in 5.394 seconds ]
Waiting 30 seconds after suspending nodes
[Info] Stage 1 Scenario: Alive Nodes: 20. Dead Nodes: 20
[Info] Will suspend 3.99.144.144:12361
[Info] Will suspend 3.99.144.144:12365
[Info] Will suspend 3.99.144.144:12350
[Info] Will suspend 3.99.144.144:12382
[Info] Will keep alive 3.99.144.144:12360
[Info] Will keep alive 3.99.144.144:12364
[Info] Will keep alive 3.99.144.144:12376
[Info] Will keep alive 3.99.144.144:12349
[Info] Will keep alive 3.99.144.144:12358
[Info] Will keep alive 3.99.144.144:12372
[Info] Will keep alive 3.99.144.144:12351
[Info] Will keep alive 3.99.144.144:12378
[Info] Will keep alive 3.99.144.144:12368
[Info] Will keep alive 3.99.144.144:12373
[Info] Will keep alive 3.99.144.144:12362
[Info] Will keep alive 3.99.144.144:12363
[Info] Will keep alive 3.99.144.144:12370
[Info] Will keep alive 3.99.144.144:12369
[Info] Will keep alive 3.99.144.144:12381
[Info] Will keep alive 3.99.144.144:12385
Waiting 10 seconds before suspending nodes
[Info] Suspending 4 servers.
Server 3.99.144.144:12350 suspended
Server 3.99.144.144:12365 suspended
Server 3.99.144.144:12382 suspended
Server 3.99.144.144:12361 suspended
[ ... Completed in 5.444 seconds ]
Waiting 30 seconds after suspending nodes
[Info] Stage 1 Scenario: Alive Nodes: 16. Dead Nodes: 24


[ TEST Suspend Verify v2; Stage 1; Key Recovery; ]
Checking that 24 nodes are suspended
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12367
[Test isAlive] for 3.99.144.144:12354
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12353
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12377
[Test isAlive] for 3.99.144.144:12379
[Test isAlive] for 3.99.144.144:12386
[Test isAlive] for 3.99.144.144:12380
[Test isAlive] for 3.99.144.144:12357
[Test isAlive] for 3.99.144.144:12371
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12365
[Test isAlive] for 3.99.144.144:12350
[Test isAlive] for 3.99.144.144:12382
[ ... Completed in 10.005 seconds ]
Checking that 16 nodes are alive
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12364
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12349
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12372
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12378
[Test isAlive] for 3.99.144.144:12368
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12369
[Test isAlive] for 3.99.144.144:12381
[Test isAlive] for 3.99.144.144:12385
[ ... Completed in 0.008 seconds ]
Sending GetMembershipCount to all servers... 
[Info] Server 3.99.144.144:12360 replied with membership count = 16 (expected count = 16)
[Info] Server 3.99.144.144:12364 replied with membership count = 16 (expected count = 16)
[Info] Server 3.99.144.144:12376 replied with membership count = 16 (expected count = 16)
[Info] Server 3.99.144.144:12349 replied with membership count = 16 (expected count = 16)
[Info] Server 3.99.144.144:12372 replied with membership count = 16 (expected count = 16)
[Info] Server 3.99.144.144:12358 replied with membership count = 16 (expected count = 16)
[Info] Server 3.99.144.144:12351 replied with membership count = 16 (expected count = 16)
[Info] Server 3.99.144.144:12378 replied with membership count = 16 (expected count = 16)
[Info] Server 3.99.144.144:12368 replied with membership count = 16 (expected count = 16)
[Info] Server 3.99.144.144:12373 replied with membership count = 16 (expected count = 16)
[Info] Server 3.99.144.144:12363 replied with membership count = 16 (expected count = 16)
[Info] Server 3.99.144.144:12370 replied with membership count = 16 (expected count = 16)
[Info] Server 3.99.144.144:12362 replied with membership count = 16 (expected count = 16)
[Info] Server 3.99.144.144:12369 replied with membership count = 16 (expected count = 16)
[Info] Server 3.99.144.144:12381 replied with membership count = 16 (expected count = 16)
[Info] Server 3.99.144.144:12385 replied with membership count = 16 (expected count = 16)
[ ... Completed in 0.007 seconds ]
[Ok] All servers returned the expected membership count (16)
Attempting retrieval of keys... 
Issuing GETs ... 
[DONE in 6.593s]
[Info] Suspend Verify v2; Stage 1; Key Recovery; Of 40000 stored keys: 39816 SUCCESS, 0 TIMEOUT, 184 KEY-NOT-FOUND, 0 ERROR-UID, 0 ERROR-INVALID-GET (value or version), 0 ERROR-OTHER.
[Info] Suspend Verify v2; Stage 1; Key Recovery; Expected Key Loss: 60.0%. Actual Key Loss: 0.46%
[Info] Suspend Verify v2; Stage 1; Key Recovery; Total Value Size (MiB): 19.03245449066162 MiB.
[Info] Suspend Verify v2; Stage 1; Key Recovery; SUCCESS Response Count: 39816.0.
[Info] Suspend Verify v2; Stage 1; Key Recovery; % SUCCESS: 99.54%.
[Info] Suspend Verify v2; Stage 1; Key Recovery; % FAILED (Not SUCCESS): 0.46%.
[Info] Suspend Verify v2; Stage 1; Key Recovery; % TIMEOUT: 0.0%.
[Info] Suspend Verify v2; Stage 1; Key Recovery; % KEY-NOT-FOUND: 0.46%.
[Info] Suspend Verify v2; Stage 1; Key Recovery; % ERROR-UID: 0.0%.
[Info] Suspend Verify v2; Stage 1; Key Recovery; % ERROR-INVALID-GET (value or version): 0.0%.
[Info] Suspend Verify v2; Stage 1; Key Recovery; % ERROR-OTHER: 0.0%.
[Info] Minimum GET SUCCESS % Ok: 99.54% Success.(40.0% SUCCESS expected, 20.0% minimum required).
TEST_PASSED
[ ... TEST Completed in 16.615 seconds ]

[Info] Preparing for Stage 2a of the Suspend/Resume Test
[Info] Stage 2a Scenario: Alive Nodes: 16. Dead Nodes: 24

[ TEST Concurrent Store/Load v2; Stage 2a; (Clients = 64, Limit = 20000 PUTs) ]
Issuing PUTS ... 
[DONE in 7.011s]
[Info] Concurrent Store/Load v2; Stage 2a; PUT Phase; Of 20000 stored keys: 20000 SUCCESS, 0 TIMEOUT, 0 KEY-NOT-FOUND, 0 ERROR-UID, 0 ERROR-INVALID-GET (value or version), 0 ERROR-OTHER.
[Info] Concurrent Store/Load v2; Stage 2a; PUT Phase; Expected Key Loss: 0.0%. Actual Key Loss: 0.0%
[Info] Concurrent Store/Load v2; Stage 2a; PUT Phase; Total Value Size (MiB): 9.514667510986328 MiB.
[Info] Concurrent Store/Load v2; Stage 2a; PUT Phase; SUCCESS Response Count: 20000.0.
[Info] Concurrent Store/Load v2; Stage 2a; PUT Phase; % SUCCESS: 100.0%.
[Info] Concurrent Store/Load v2; Stage 2a; PUT Phase; % FAILED (Not SUCCESS): 0.0%.
[Info] Concurrent Store/Load v2; Stage 2a; PUT Phase; % TIMEOUT: 0.0%.
[Info] Concurrent Store/Load v2; Stage 2a; PUT Phase; % KEY-NOT-FOUND: 0.0%.
[Info] Concurrent Store/Load v2; Stage 2a; PUT Phase; % ERROR-UID: 0.0%.
[Info] Concurrent Store/Load v2; Stage 2a; PUT Phase; % ERROR-INVALID-GET (value or version): 0.0%.
[Info] Concurrent Store/Load v2; Stage 2a; PUT Phase; % ERROR-OTHER: 0.0%.
[Info] PUTs % SUCCESS Ok: 100.0% Success.(90.0% minimum required).
Confirming keys were stored correctly.
Issuing GETs ... 
[DONE in 3.309s]
[Info] Concurrent Store/Load v2; Stage 2a; GET Phase; Of 20000 stored keys: 20000 SUCCESS, 0 TIMEOUT, 0 KEY-NOT-FOUND, 0 ERROR-UID, 0 ERROR-INVALID-GET (value or version), 0 ERROR-OTHER.
[Info] Concurrent Store/Load v2; Stage 2a; GET Phase; Expected Key Loss: 0.0%. Actual Key Loss: 0.0%
[Info] Concurrent Store/Load v2; Stage 2a; GET Phase; Total Value Size (MiB): 9.514667510986328 MiB.
[Info] Concurrent Store/Load v2; Stage 2a; GET Phase; SUCCESS Response Count: 20000.0.
[Info] Concurrent Store/Load v2; Stage 2a; GET Phase; % SUCCESS: 100.0%.
[Info] Concurrent Store/Load v2; Stage 2a; GET Phase; % FAILED (Not SUCCESS): 0.0%.
[Info] Concurrent Store/Load v2; Stage 2a; GET Phase; % TIMEOUT: 0.0%.
[Info] Concurrent Store/Load v2; Stage 2a; GET Phase; % KEY-NOT-FOUND: 0.0%.
[Info] Concurrent Store/Load v2; Stage 2a; GET Phase; % ERROR-UID: 0.0%.
[Info] Concurrent Store/Load v2; Stage 2a; GET Phase; % ERROR-INVALID-GET (value or version): 0.0%.
[Info] Concurrent Store/Load v2; Stage 2a; GET Phase; % ERROR-OTHER: 0.0%.
[Info] GETs % SUCCESS Ok: 100.0% Success.(90.0% minimum required).
Checking that 16 nodes are alive
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12364
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12349
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12372
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12378
[Test isAlive] for 3.99.144.144:12368
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12369
[Test isAlive] for 3.99.144.144:12381
[Test isAlive] for 3.99.144.144:12385
[ ... Completed in 0.008 seconds ]
TEST_PASSED
[ ... TEST Completed in 10.329 seconds ]

[Info] Preparing for Stage 2b of the Suspend/Resume Test
[Info] To resume 18 node(s) in Stage 2a.
[Info] Will resume 3.99.144.144:12361
[Info] Will resume 3.99.144.144:12374
[Info] Will resume 3.99.144.144:12366
[Info] Will resume 3.99.144.144:12347
[Info] Will resume 3.99.144.144:12352
[Info] Will resume 3.99.144.144:12356
[Info] Will resume 3.99.144.144:12359
[Info] Will resume 3.99.144.144:12348
[Info] Will resume 3.99.144.144:12375
[Info] Will resume 3.99.144.144:12350
[Info] Will resume 3.99.144.144:12353
[Info] Will resume 3.99.144.144:12371
[Info] Will resume 3.99.144.144:12383
[Info] Will resume 3.99.144.144:12355
[Info] Will resume 3.99.144.144:12386
[Info] Will resume 3.99.144.144:12384
[Info] Will resume 3.99.144.144:12379
[Info] Will resume 3.99.144.144:12377
[Info] Will keep suspended 3.99.144.144:12382
[Info] Will keep suspended 3.99.144.144:12380
[Info] Will keep suspended 3.99.144.144:12365
[Info] Will keep suspended 3.99.144.144:12367
[Info] Will keep suspended 3.99.144.144:12357
[Info] Will keep suspended 3.99.144.144:12354
Waiting 10 seconds before resuming nodes
[Info] Resuming 18 servers.
Server 3.99.144.144:12361 resumed
Server 3.99.144.144:12374 resumed
Server 3.99.144.144:12348 resumed
Server 3.99.144.144:12366 resumed
Server 3.99.144.144:12375 resumed
Server 3.99.144.144:12356 resumed
Server 3.99.144.144:12347 resumed
Server 3.99.144.144:12355 resumed
Server 3.99.144.144:12350 resumed
Server 3.99.144.144:12377 resumed
Server 3.99.144.144:12352 resumed
Server 3.99.144.144:12383 resumed
Server 3.99.144.144:12386 resumed
Server 3.99.144.144:12371 resumed
Server 3.99.144.144:12359 resumed
Server 3.99.144.144:12353 resumed
Server 3.99.144.144:12384 resumed
Server 3.99.144.144:12379 resumed
[ ... Completed in 24.243 seconds ]
Waiting 60 seconds after resuming nodes
[Info] Stage 2b Scenario: Alive Nodes: 34. Dead Nodes: 6


[ TEST Suspend Verify v2; Stage 2b; Key Recovery; ]
Checking that 6 nodes are suspended
[Test isAlive] for 3.99.144.144:12382
[Test isAlive] for 3.99.144.144:12380
[Test isAlive] for 3.99.144.144:12365
[Test isAlive] for 3.99.144.144:12367
[Test isAlive] for 3.99.144.144:12357
[Test isAlive] for 3.99.144.144:12354
[ ... Completed in 10.004 seconds ]
Checking that 34 nodes are alive
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12350
[Test isAlive] for 3.99.144.144:12353
[Test isAlive] for 3.99.144.144:12371
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12386
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12379
[Test isAlive] for 3.99.144.144:12377
[Test isAlive] for 3.99.144.144:12385
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12349
[Test isAlive] for 3.99.144.144:12378
[Test isAlive] for 3.99.144.144:12364
[Test isAlive] for 3.99.144.144:12369
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12368
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12372
[Test isAlive] for 3.99.144.144:12381
[ ... Completed in 0.009 seconds ]
Sending GetMembershipCount to all servers... 
[Info] Server 3.99.144.144:12361 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12374 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12366 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12352 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12347 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12356 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12359 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12348 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12375 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12350 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12353 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12383 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12371 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12355 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12386 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12384 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12379 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12377 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12385 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12363 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12349 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12378 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12364 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12369 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12360 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12373 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12358 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12370 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12368 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12351 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12376 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12362 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12372 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12381 replied with membership count = 34 (expected count = 34)
[ ... Completed in 0.009 seconds ]
[Ok] All servers returned the expected membership count (34)
Attempting retrieval of keys... 
Issuing GETs ... 
[DONE in 6.729s]
[Info] Suspend Verify v2; Stage 2b; Key Recovery; Of 40000 stored keys: 40000 SUCCESS, 0 TIMEOUT, 0 KEY-NOT-FOUND, 0 ERROR-UID, 0 ERROR-INVALID-GET (value or version), 0 ERROR-OTHER.
[Info] Suspend Verify v2; Stage 2b; Key Recovery; Expected Key Loss: 15.0%. Actual Key Loss: 0.0%
[Info] Suspend Verify v2; Stage 2b; Key Recovery; Total Value Size (MiB): 19.120856285095215 MiB.
[Info] Suspend Verify v2; Stage 2b; Key Recovery; SUCCESS Response Count: 40000.0.
[Info] Suspend Verify v2; Stage 2b; Key Recovery; % SUCCESS: 100.0%.
[Info] Suspend Verify v2; Stage 2b; Key Recovery; % FAILED (Not SUCCESS): 0.0%.
[Info] Suspend Verify v2; Stage 2b; Key Recovery; % TIMEOUT: 0.0%.
[Info] Suspend Verify v2; Stage 2b; Key Recovery; % KEY-NOT-FOUND: 0.0%.
[Info] Suspend Verify v2; Stage 2b; Key Recovery; % ERROR-UID: 0.0%.
[Info] Suspend Verify v2; Stage 2b; Key Recovery; % ERROR-INVALID-GET (value or version): 0.0%.
[Info] Suspend Verify v2; Stage 2b; Key Recovery; % ERROR-OTHER: 0.0%.
[Info] Minimum GET SUCCESS % Ok: 100.0% Success.(85.0% SUCCESS expected, 42.5% minimum required).
TEST_PASSED
[ ... TEST Completed in 16.752 seconds ]

[Info] Preparing for Stage 2c of the Suspend/Resume Test
[Info] Stage 2c Scenario: Alive Nodes: 34. Dead Nodes: 6


[ TEST Suspend Verify v2; Stage 2c; Key Recovery; ]
Checking that 6 nodes are suspended
[Test isAlive] for 3.99.144.144:12382
[Test isAlive] for 3.99.144.144:12380
[Test isAlive] for 3.99.144.144:12365
[Test isAlive] for 3.99.144.144:12367
[Test isAlive] for 3.99.144.144:12357
[Test isAlive] for 3.99.144.144:12354
[ ... Completed in 10.003 seconds ]
Checking that 34 nodes are alive
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12350
[Test isAlive] for 3.99.144.144:12353
[Test isAlive] for 3.99.144.144:12371
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12386
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12379
[Test isAlive] for 3.99.144.144:12377
[Test isAlive] for 3.99.144.144:12385
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12349
[Test isAlive] for 3.99.144.144:12378
[Test isAlive] for 3.99.144.144:12364
[Test isAlive] for 3.99.144.144:12369
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12368
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12372
[Test isAlive] for 3.99.144.144:12381
[ ... Completed in 0.009 seconds ]
Sending GetMembershipCount to all servers... 
[Info] Server 3.99.144.144:12361 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12374 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12366 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12347 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12352 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12356 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12359 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12348 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12375 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12350 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12353 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12371 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12383 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12355 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12386 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12384 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12379 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12385 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12377 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12363 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12349 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12378 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12364 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12360 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12369 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12373 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12358 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12368 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12370 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12376 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12351 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12362 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12372 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12381 replied with membership count = 34 (expected count = 34)
[ ... Completed in 0.009 seconds ]
[Ok] All servers returned the expected membership count (34)
Attempting retrieval of keys... 
Issuing GETs ... 
[DONE in 3.374s]
[Info] Suspend Verify v2; Stage 2c; Key Recovery; Of 20000 stored keys: 19997 SUCCESS, 0 TIMEOUT, 3 KEY-NOT-FOUND, 0 ERROR-UID, 0 ERROR-INVALID-GET (value or version), 0 ERROR-OTHER.
[Info] Suspend Verify v2; Stage 2c; Key Recovery; Expected Key Loss: 0.0%. Actual Key Loss: 0.015%
[Info] Suspend Verify v2; Stage 2c; Key Recovery; Total Value Size (MiB): 9.51309871673584 MiB.
[Info] Suspend Verify v2; Stage 2c; Key Recovery; SUCCESS Response Count: 19997.0.
[Info] Suspend Verify v2; Stage 2c; Key Recovery; % SUCCESS: 99.985%.
[Info] Suspend Verify v2; Stage 2c; Key Recovery; % FAILED (Not SUCCESS): 0.015%.
[Info] Suspend Verify v2; Stage 2c; Key Recovery; % TIMEOUT: 0.0%.
[Info] Suspend Verify v2; Stage 2c; Key Recovery; % KEY-NOT-FOUND: 0.015%.
[Info] Suspend Verify v2; Stage 2c; Key Recovery; % ERROR-UID: 0.0%.
[Info] Suspend Verify v2; Stage 2c; Key Recovery; % ERROR-INVALID-GET (value or version): 0.0%.
[Info] Suspend Verify v2; Stage 2c; Key Recovery; % ERROR-OTHER: 0.0%.
[Info] Minimum GET SUCCESS % Ok: 99.985% Success.(100.0% SUCCESS expected, 50.0% minimum required).
TEST_PASSED
[ ... TEST Completed in 13.395 seconds ]
Sending WIPE-OUT to all servers...
[ ... Completed in 0.027 seconds ]
[OK]
Waiting for 5 seconds for the cache to be cleared!
[ ... TEST Completed in 5.027 seconds ]
Pausing for 6 seconds to wait for the cache to be cleared!

Sending GetMembershipCount to all servers... 
[Info] Server 3.99.144.144:12361 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12374 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12366 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12347 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12352 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12356 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12348 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12375 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12359 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12350 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12353 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12371 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12383 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12355 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12386 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12384 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12379 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12377 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12385 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12363 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12349 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12378 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12364 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12369 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12360 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12373 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12358 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12370 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12368 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12351 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12376 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12362 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12372 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12381 replied with membership count = 34 (expected count = 34)
[ ... Completed in 0.01 seconds ]
[Ok] All servers returned the expected membership count (34)
[ ... TEST Completed in 0.01 seconds ]

[ TEST Random Front-End Basic (PUT -> GET -> REM -> REM -> GET). Value Length = 8 Bytes ]
TEST_PASSED

[ TEST Random Front-End Basic (PUT -> GET -> REM -> REM -> GET). Value Length = 32 Bytes ]
TEST_PASSED

[ TEST Random Front-End Basic (PUT -> GET -> REM -> REM -> GET). Value Length = 512 Bytes ]
TEST_PASSED

[ TEST Random Front-End Basic (PUT -> GET -> REM -> REM -> GET). Value Length = 1024 Bytes ]
TEST_PASSED
[ ... TEST Completed in 0.257 seconds ]
Sending WIPE-OUT to all servers...
[ ... Completed in 0.018 seconds ]
[OK]
Waiting for 5 seconds for the cache to be cleared!

[ TEST Random Front-End Performance (1 clients, 60.0 seconds): PUT -> GET (closed loop) ]
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12350
[Test isAlive] for 3.99.144.144:12353
[Test isAlive] for 3.99.144.144:12371
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12386
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12379
[Test isAlive] for 3.99.144.144:12377
[Test isAlive] for 3.99.144.144:12385
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12349
[Test isAlive] for 3.99.144.144:12378
[Test isAlive] for 3.99.144.144:12364
[Test isAlive] for 3.99.144.144:12369
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12368
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12372
[Test isAlive] for 3.99.144.144:12381
[ ... Completed in 0.01 seconds ]
TEST_PASSED
Stage 2 Random Front-End: Test Status: TEST_PASSED
Clients: 1
Total Requests (without replies): 3707
Total Requests (including replies): 3743
Successful Responses: 3707
Timeout Responses: 0
Total Failed Responses: 0
% Success: 100.0
% Timeout: 0.0
% Failed: 0.0
Response Time (Successful requests):
[max] 63ms, [min] 5ms
[avg] 16.16833018613434 ms, [stdev] 6.758083450537485ms
Throughput: 61.7627457514162 requests per second
Goodput: 61.7627457514162 requests per second
Retry Rate (Successful requests): 1.0097113568923657
Sending WIPE-OUT to all servers...
[ ... Completed in 0.019 seconds ]
[OK]
Waiting for 5 seconds for the cache to be cleared!
[ ... TEST Completed in 65.05 seconds ]
[Info] Pausing for 6 seconds to wait for the cache to be cleared!

[ TEST Random Front-End Performance (16 clients, 60.0 seconds): PUT -> GET (closed loop) ]
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12350
[Test isAlive] for 3.99.144.144:12353
[Test isAlive] for 3.99.144.144:12371
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12386
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12379
[Test isAlive] for 3.99.144.144:12377
[Test isAlive] for 3.99.144.144:12385
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12349
[Test isAlive] for 3.99.144.144:12378
[Test isAlive] for 3.99.144.144:12364
[Test isAlive] for 3.99.144.144:12369
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12368
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12372
[Test isAlive] for 3.99.144.144:12381
[ ... Completed in 0.009 seconds ]
TEST_PASSED
Stage 2 Random Front-End: Test Status: TEST_PASSED
Clients: 16
Total Requests (without replies): 60395
Total Requests (including replies): 61151
Successful Responses: 60395
Timeout Responses: 0
Total Failed Responses: 0
% Success: 100.0
% Timeout: 0.0
% Failed: 0.0
Response Time (Successful requests):
[max] 143ms, [min] 5ms
[avg] 15.878582664127824 ms, [stdev] 7.2778825387254065ms
Throughput: 1006.3149826712877 requests per second
Goodput: 1006.3149826712877 requests per second
Retry Rate (Successful requests): 1.0125175925159366
Sending WIPE-OUT to all servers...
[ ... Completed in 0.035 seconds ]
[OK]
Waiting for 5 seconds for the cache to be cleared!
[ ... TEST Completed in 65.085 seconds ]
[Info] Pausing for 6 seconds to wait for the cache to be cleared!

[ TEST Random Front-End Performance (256 clients, 60.0 seconds): PUT -> GET (closed loop) ]
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12350
[Test isAlive] for 3.99.144.144:12353
[Test isAlive] for 3.99.144.144:12371
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12386
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12379
[Test isAlive] for 3.99.144.144:12377
[Test isAlive] for 3.99.144.144:12385
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12349
[Test isAlive] for 3.99.144.144:12378
[Test isAlive] for 3.99.144.144:12364
[Test isAlive] for 3.99.144.144:12369
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12368
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12372
[Test isAlive] for 3.99.144.144:12381
[ ... Completed in 0.01 seconds ]
TEST_PASSED
[ CPU load (loadavg, last one minute) 1.15 ]
Stage 2 Random Front-End: Test Status: TEST_PASSED
Clients: 256
Total Requests (without replies): 968049
Total Requests (including replies): 980072
Successful Responses: 968049
Timeout Responses: 0
Total Failed Responses: 0
% Success: 100.0
% Timeout: 0.0
% Failed: 0.0
Response Time (Successful requests):
[max] 301ms, [min] 5ms
[avg] 15.852719232187626 ms, [stdev] 7.235505971449028ms
Throughput: 16119.64232191028 requests per second
Goodput: 16119.64232191028 requests per second
Retry Rate (Successful requests): 1.0124198258559225
Sending WIPE-OUT to all servers...
[ ... Completed in 0.028 seconds ]
[OK]
Waiting for 5 seconds for the cache to be cleared!
[ ... TEST Completed in 65.436 seconds ]
[Info] Pausing for 6 seconds to wait for the cache to be cleared!

[ TEST Random Front-End Performance (512 clients, 60.0 seconds): PUT -> GET (closed loop) ]
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12350
[Test isAlive] for 3.99.144.144:12353
[Test isAlive] for 3.99.144.144:12371
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12386
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12379
[Test isAlive] for 3.99.144.144:12377
[Test isAlive] for 3.99.144.144:12385
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12349
[Test isAlive] for 3.99.144.144:12378
[Test isAlive] for 3.99.144.144:12364
[Test isAlive] for 3.99.144.144:12369
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12368
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12372
[Test isAlive] for 3.99.144.144:12381
[ ... Completed in 0.009 seconds ]
TEST_PASSED
[ CPU load (loadavg, last one minute) 1.91 ]
Stage 2 Random Front-End: Test Status: TEST_PASSED
Clients: 512
Total Requests (without replies): 1909772
Total Requests (including replies): 1935791
Successful Responses: 1909772
Timeout Responses: 0
Total Failed Responses: 0
% Success: 100.0
% Timeout: 0.0
% Failed: 0.0
Response Time (Successful requests):
[max] 302ms, [min] 5ms
[avg] 16.063671998542233 ms, [stdev] 7.443267883546286ms
Throughput: 31798.265039377944 requests per second
Goodput: 31798.265039377944 requests per second
Retry Rate (Successful requests): 1.0136241394260677
Sending WIPE-OUT to all servers...
[ ... Completed in 0.035 seconds ]
[OK]
Waiting for 5 seconds for the cache to be cleared!
[ ... TEST Completed in 65.806 seconds ]
[Info] Pausing for 6 seconds to wait for the cache to be cleared!

[Test isAlive] for 3.99.144.144:12382
[Test isAlive] for 3.99.144.144:12380
[Test isAlive] for 3.99.144.144:12365
[Test isAlive] for 3.99.144.144:12367
[Test isAlive] for 3.99.144.144:12357
[Test isAlive] for 3.99.144.144:12354
[ ... Completed in 10.003 seconds ]
Waiting 10 seconds before resuming nodes
[Info] Resuming 6 servers.
Server 3.99.144.144:12380 resumed
Server 3.99.144.144:12382 resumed
Server 3.99.144.144:12367 resumed
Server 3.99.144.144:12365 resumed
Server 3.99.144.144:12357 resumed
Server 3.99.144.144:12354 resumed
[ ... Completed in 8.017 seconds ]
Waiting 60 seconds after resuming nodes
Sending WIPE-OUT to all servers...
[ ... Completed in 0.024 seconds ]
[OK]
Waiting for 5 seconds for the cache to be cleared!

[ TEST Concurrent Store/Load v2; Stage 3; (Clients = 128, Limit = 40000 PUTs) ]
Issuing PUTS ... 
[DONE in 6.94s]
[Info] Concurrent Store/Load v2; Stage 3; PUT Phase; Of 40000 stored keys: 40000 SUCCESS, 0 TIMEOUT, 0 KEY-NOT-FOUND, 0 ERROR-UID, 0 ERROR-INVALID-GET (value or version), 0 ERROR-OTHER.
[Info] Concurrent Store/Load v2; Stage 3; PUT Phase; Expected Key Loss: 0.0%. Actual Key Loss: 0.0%
[Info] Concurrent Store/Load v2; Stage 3; PUT Phase; Total Value Size (MiB): 19.119232177734375 MiB.
[Info] Concurrent Store/Load v2; Stage 3; PUT Phase; SUCCESS Response Count: 40000.0.
[Info] Concurrent Store/Load v2; Stage 3; PUT Phase; % SUCCESS: 100.0%.
[Info] Concurrent Store/Load v2; Stage 3; PUT Phase; % FAILED (Not SUCCESS): 0.0%.
[Info] Concurrent Store/Load v2; Stage 3; PUT Phase; % TIMEOUT: 0.0%.
[Info] Concurrent Store/Load v2; Stage 3; PUT Phase; % KEY-NOT-FOUND: 0.0%.
[Info] Concurrent Store/Load v2; Stage 3; PUT Phase; % ERROR-UID: 0.0%.
[Info] Concurrent Store/Load v2; Stage 3; PUT Phase; % ERROR-INVALID-GET (value or version): 0.0%.
[Info] Concurrent Store/Load v2; Stage 3; PUT Phase; % ERROR-OTHER: 0.0%.
[Info] PUTs % SUCCESS Ok: 100.0% Success.(90.0% minimum required).
Checking that 40 nodes are alive
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12349
[Test isAlive] for 3.99.144.144:12350
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12353
[Test isAlive] for 3.99.144.144:12354
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12357
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12364
[Test isAlive] for 3.99.144.144:12365
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12367
[Test isAlive] for 3.99.144.144:12368
[Test isAlive] for 3.99.144.144:12369
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12371
[Test isAlive] for 3.99.144.144:12372
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12377
[Test isAlive] for 3.99.144.144:12378
[Test isAlive] for 3.99.144.144:12379
[Test isAlive] for 3.99.144.144:12380
[Test isAlive] for 3.99.144.144:12381
[Test isAlive] for 3.99.144.144:12382
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12385
[Test isAlive] for 3.99.144.144:12386
[ ... Completed in 0.01 seconds ]
TEST_PASSED
[ ... TEST Completed in 6.951 seconds ]

[Info] Preparing for the sequential consistency Test (Stage 3)
[Info] To suspend 12 total node(s) in sequential consistency test.

[ TEST Concurrent Store/Load v2; Stage 3a; (Clients = 128, Limit = 20000 PUTs) ]
Issuing RELOADS ... 
[DONE in 3.595s]
[Info] Concurrent Store/Load v2; Stage 3a; PUT Phase; Of 20000 stored keys: 20000 SUCCESS, 0 TIMEOUT, 0 KEY-NOT-FOUND, 0 ERROR-UID, 0 ERROR-INVALID-GET (value or version), 0 ERROR-OTHER.
[Info] Concurrent Store/Load v2; Stage 3a; PUT Phase; Expected Key Loss: 0.0%. Actual Key Loss: 0.0%
[Info] Concurrent Store/Load v2; Stage 3a; PUT Phase; Total Value Size (MiB): 9.56334114074707 MiB.
[Info] Concurrent Store/Load v2; Stage 3a; PUT Phase; SUCCESS Response Count: 20000.0.
[Info] Concurrent Store/Load v2; Stage 3a; PUT Phase; % SUCCESS: 100.0%.
[Info] Concurrent Store/Load v2; Stage 3a; PUT Phase; % FAILED (Not SUCCESS): 0.0%.
[Info] Concurrent Store/Load v2; Stage 3a; PUT Phase; % TIMEOUT: 0.0%.
[Info] Concurrent Store/Load v2; Stage 3a; PUT Phase; % KEY-NOT-FOUND: 0.0%.
[Info] Concurrent Store/Load v2; Stage 3a; PUT Phase; % ERROR-UID: 0.0%.
[Info] Concurrent Store/Load v2; Stage 3a; PUT Phase; % ERROR-INVALID-GET (value or version): 0.0%.
[Info] Concurrent Store/Load v2; Stage 3a; PUT Phase; % ERROR-OTHER: 0.0%.
[Info] PUTs % SUCCESS Ok: 100.0% Success.(90.0% minimum required).
Checking that 40 nodes are alive
[Test isAlive] for 3.99.144.144:12378
[Test isAlive] for 3.99.144.144:12353
[Test isAlive] for 3.99.144.144:12372
[Test isAlive] for 3.99.144.144:12377
[Test isAlive] for 3.99.144.144:12365
[Test isAlive] for 3.99.144.144:12371
[Test isAlive] for 3.99.144.144:12379
[Test isAlive] for 3.99.144.144:12364
[Test isAlive] for 3.99.144.144:12368
[Test isAlive] for 3.99.144.144:12357
[Test isAlive] for 3.99.144.144:12350
[Test isAlive] for 3.99.144.144:12369
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12354
[Test isAlive] for 3.99.144.144:12386
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12382
[Test isAlive] for 3.99.144.144:12381
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12367
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12385
[Test isAlive] for 3.99.144.144:12380
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12349
[ ... Completed in 0.01 seconds ]
TEST_PASSED
[ ... TEST Completed in 3.606 seconds ]
[Info] Will suspend 3.99.144.144:12378
[Info] Will suspend 3.99.144.144:12353
[Info] Will suspend 3.99.144.144:12372
Waiting 10 seconds before suspending nodes
[Info] Suspending 3 servers.
Server 3.99.144.144:12372 suspended
Server 3.99.144.144:12378 suspended
Server 3.99.144.144:12353 suspended
[ ... Completed in 4.121 seconds ]
Waiting 30 seconds after suspending nodes
[Info] Stage 3 Scenario: Alive Nodes: 37. Dead Nodes: 3
[ ... TEST Completed in 47.727 seconds ]


[ TEST Sequential Consistency; Stage 3a; (Clients = 128, Limit = 40000 GETS) ]
Checking that 3 nodes are suspended
[Test isAlive] for 3.99.144.144:12378
[Test isAlive] for 3.99.144.144:12353
[Test isAlive] for 3.99.144.144:12372
[ ... Completed in 10.003 seconds ]
Checking that 37 nodes are alive
[Test isAlive] for 3.99.144.144:12377
[Test isAlive] for 3.99.144.144:12365
[Test isAlive] for 3.99.144.144:12371
[Test isAlive] for 3.99.144.144:12364
[Test isAlive] for 3.99.144.144:12379
[Test isAlive] for 3.99.144.144:12368
[Test isAlive] for 3.99.144.144:12357
[Test isAlive] for 3.99.144.144:12350
[Test isAlive] for 3.99.144.144:12369
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12354
[Test isAlive] for 3.99.144.144:12386
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12382
[Test isAlive] for 3.99.144.144:12381
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12367
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12385
[Test isAlive] for 3.99.144.144:12380
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12349
[ ... Completed in 0.009 seconds ]
Sending GetMembershipCount to all servers... 
[Info] Server 3.99.144.144:12377 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12365 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12371 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12379 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12364 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12368 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12369 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12357 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12363 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12350 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12354 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12355 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12370 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12376 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12386 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12361 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12375 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12360 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12351 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12382 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12381 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12383 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12358 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12348 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12366 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12367 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12384 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12359 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12362 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12347 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12352 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12374 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12385 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12380 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12373 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12356 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12349 replied with membership count = 37 (expected count = 37)
[ ... Completed in 0.008 seconds ]
[Ok] All servers returned the expected membership count (37)
Attempting retrieval of keys... 
Issuing GETs ... 
[DONE in 3.425s]
[Info] Sequential Consistency; Stage 3a; GET Phase; Of 40000 stored keys: 40000 SUCCESS, 0 TIMEOUT, 0 KEY-NOT-FOUND, 0 ERROR-UID, 0 ERROR-INVALID-GET (value or version), 0 ERROR-OTHER.
[Info] Sequential Consistency; Stage 3a; GET Phase; Expected Key Loss: 0.0%. Actual Key Loss: 0.0%
[Info] Sequential Consistency; Stage 3a; GET Phase; Total Value Size (MiB): 19.132648468017578 MiB.
[Info] Sequential Consistency; Stage 3a; GET Phase; SUCCESS Response Count: 40000.0.
[Info] Sequential Consistency; Stage 3a; GET Phase; % SUCCESS: 100.0%.
[Info] Sequential Consistency; Stage 3a; GET Phase; % FAILED (Not SUCCESS): 0.0%.
[Info] Sequential Consistency; Stage 3a; GET Phase; % TIMEOUT: 0.0%.
[Info] Sequential Consistency; Stage 3a; GET Phase; % KEY-NOT-FOUND: 0.0%.
[Info] Sequential Consistency; Stage 3a; GET Phase; % ERROR-UID: 0.0%.
[Info] Sequential Consistency; Stage 3a; GET Phase; % ERROR-INVALID-GET (value or version): 0.0%.
[Info] Sequential Consistency; Stage 3a; GET Phase; % ERROR-OTHER: 0.0%.
[Info] Sequential Consistency; Stage 3a; GET Phase; NOT-STORED keys Count: 0.0.
Checking that 37 nodes are alive
[Test isAlive] for 3.99.144.144:12377
[Test isAlive] for 3.99.144.144:12365
[Test isAlive] for 3.99.144.144:12371
[Test isAlive] for 3.99.144.144:12379
[Test isAlive] for 3.99.144.144:12364
[Test isAlive] for 3.99.144.144:12368
[Test isAlive] for 3.99.144.144:12357
[Test isAlive] for 3.99.144.144:12350
[Test isAlive] for 3.99.144.144:12369
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12354
[Test isAlive] for 3.99.144.144:12386
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12382
[Test isAlive] for 3.99.144.144:12381
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12367
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12385
[Test isAlive] for 3.99.144.144:12380
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12349
[ ... Completed in 0.009 seconds ]
TEST_PASSED
[ TEST Concurrent Store/Load v2; Stage 3b; (Clients = 128, Limit = 20000 PUTs) ]
Issuing RELOADS ... 
[DONE in 3.625s]
[Info] Concurrent Store/Load v2; Stage 3b; PUT Phase; Of 20000 stored keys: 20000 SUCCESS, 0 TIMEOUT, 0 KEY-NOT-FOUND, 0 ERROR-UID, 0 ERROR-INVALID-GET (value or version), 0 ERROR-OTHER.
[Info] Concurrent Store/Load v2; Stage 3b; PUT Phase; Expected Key Loss: 0.0%. Actual Key Loss: 0.0%
[Info] Concurrent Store/Load v2; Stage 3b; PUT Phase; Total Value Size (MiB): 9.573698043823242 MiB.
[Info] Concurrent Store/Load v2; Stage 3b; PUT Phase; SUCCESS Response Count: 20000.0.
[Info] Concurrent Store/Load v2; Stage 3b; PUT Phase; % SUCCESS: 100.0%.
[Info] Concurrent Store/Load v2; Stage 3b; PUT Phase; % FAILED (Not SUCCESS): 0.0%.
[Info] Concurrent Store/Load v2; Stage 3b; PUT Phase; % TIMEOUT: 0.0%.
[Info] Concurrent Store/Load v2; Stage 3b; PUT Phase; % KEY-NOT-FOUND: 0.0%.
[Info] Concurrent Store/Load v2; Stage 3b; PUT Phase; % ERROR-UID: 0.0%.
[Info] Concurrent Store/Load v2; Stage 3b; PUT Phase; % ERROR-INVALID-GET (value or version): 0.0%.
[Info] Concurrent Store/Load v2; Stage 3b; PUT Phase; % ERROR-OTHER: 0.0%.
[Info] PUTs % SUCCESS Ok: 100.0% Success.(90.0% minimum required).
Checking that 37 nodes are alive
[Test isAlive] for 3.99.144.144:12377
[Test isAlive] for 3.99.144.144:12365
[Test isAlive] for 3.99.144.144:12371
[Test isAlive] for 3.99.144.144:12379
[Test isAlive] for 3.99.144.144:12364
[Test isAlive] for 3.99.144.144:12368
[Test isAlive] for 3.99.144.144:12357
[Test isAlive] for 3.99.144.144:12350
[Test isAlive] for 3.99.144.144:12369
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12354
[Test isAlive] for 3.99.144.144:12386
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12382
[Test isAlive] for 3.99.144.144:12381
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12367
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12385
[Test isAlive] for 3.99.144.144:12380
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12349
[ ... Completed in 0.01 seconds ]
TEST_PASSED
[ ... TEST Completed in 3.635 seconds ]
[Info] Will suspend 3.99.144.144:12377
[Info] Will suspend 3.99.144.144:12365
[Info] Will suspend 3.99.144.144:12371
Waiting 10 seconds before suspending nodes
[Info] Suspending 3 servers.
Server 3.99.144.144:12377 suspended
Server 3.99.144.144:12365 suspended
Server 3.99.144.144:12371 suspended
[ ... Completed in 4.114 seconds ]
Waiting 30 seconds after suspending nodes
[Info] Stage 3 Scenario: Alive Nodes: 34. Dead Nodes: 6
[ ... TEST Completed in 47.75 seconds ]


[ TEST Sequential Consistency; Stage 3b; (Clients = 128, Limit = 40000 GETS) ]
Checking that 6 nodes are suspended
[Test isAlive] for 3.99.144.144:12378
[Test isAlive] for 3.99.144.144:12353
[Test isAlive] for 3.99.144.144:12372
[Test isAlive] for 3.99.144.144:12377
[Test isAlive] for 3.99.144.144:12365
[Test isAlive] for 3.99.144.144:12371
[ ... Completed in 10.003 seconds ]
Checking that 34 nodes are alive
[Test isAlive] for 3.99.144.144:12379
[Test isAlive] for 3.99.144.144:12364
[Test isAlive] for 3.99.144.144:12368
[Test isAlive] for 3.99.144.144:12357
[Test isAlive] for 3.99.144.144:12350
[Test isAlive] for 3.99.144.144:12369
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12354
[Test isAlive] for 3.99.144.144:12386
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12382
[Test isAlive] for 3.99.144.144:12381
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12367
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12385
[Test isAlive] for 3.99.144.144:12380
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12349
[ ... Completed in 0.009 seconds ]
Sending GetMembershipCount to all servers... 
[Info] Server 3.99.144.144:12379 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12364 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12368 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12350 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12369 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12357 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12363 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12354 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12386 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12355 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12376 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12370 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12361 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12375 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12360 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12382 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12381 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12351 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12383 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12348 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12358 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12366 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12384 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12367 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12359 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12347 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12362 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12374 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12352 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12385 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12380 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12373 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12356 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12349 replied with membership count = 34 (expected count = 34)
[ ... Completed in 0.009 seconds ]
[Ok] All servers returned the expected membership count (34)
Attempting retrieval of keys... 
Issuing GETs ... 
[DONE in 3.48s]
[Info] Sequential Consistency; Stage 3b; GET Phase; Of 40000 stored keys: 40000 SUCCESS, 0 TIMEOUT, 0 KEY-NOT-FOUND, 0 ERROR-UID, 0 ERROR-INVALID-GET (value or version), 0 ERROR-OTHER.
[Info] Sequential Consistency; Stage 3b; GET Phase; Expected Key Loss: 0.0%. Actual Key Loss: 0.0%
[Info] Sequential Consistency; Stage 3b; GET Phase; Total Value Size (MiB): 19.07344150543213 MiB.
[Info] Sequential Consistency; Stage 3b; GET Phase; SUCCESS Response Count: 40000.0.
[Info] Sequential Consistency; Stage 3b; GET Phase; % SUCCESS: 100.0%.
[Info] Sequential Consistency; Stage 3b; GET Phase; % FAILED (Not SUCCESS): 0.0%.
[Info] Sequential Consistency; Stage 3b; GET Phase; % TIMEOUT: 0.0%.
[Info] Sequential Consistency; Stage 3b; GET Phase; % KEY-NOT-FOUND: 0.0%.
[Info] Sequential Consistency; Stage 3b; GET Phase; % ERROR-UID: 0.0%.
[Info] Sequential Consistency; Stage 3b; GET Phase; % ERROR-INVALID-GET (value or version): 0.0%.
[Info] Sequential Consistency; Stage 3b; GET Phase; % ERROR-OTHER: 0.0%.
[Info] Sequential Consistency; Stage 3b; GET Phase; NOT-STORED keys Count: 0.0.
Checking that 34 nodes are alive
[Test isAlive] for 3.99.144.144:12379
[Test isAlive] for 3.99.144.144:12364
[Test isAlive] for 3.99.144.144:12368
[Test isAlive] for 3.99.144.144:12357
[Test isAlive] for 3.99.144.144:12350
[Test isAlive] for 3.99.144.144:12369
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12354
[Test isAlive] for 3.99.144.144:12386
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12382
[Test isAlive] for 3.99.144.144:12381
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12367
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12385
[Test isAlive] for 3.99.144.144:12380
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12349
[ ... Completed in 0.01 seconds ]
TEST_PASSED
[ TEST Concurrent Store/Load v2; Stage 3c; (Clients = 128, Limit = 20000 PUTs) ]
Issuing RELOADS ... 
[DONE in 4.384s]
[Info] Concurrent Store/Load v2; Stage 3c; PUT Phase; Of 20000 stored keys: 19991 SUCCESS, 9 TIMEOUT, 0 KEY-NOT-FOUND, 0 ERROR-UID, 0 ERROR-INVALID-GET (value or version), 0 ERROR-OTHER.
[Info] Concurrent Store/Load v2; Stage 3c; PUT Phase; Expected Key Loss: 0.0%. Actual Key Loss: 0.045%
[Info] Concurrent Store/Load v2; Stage 3c; PUT Phase; Total Value Size (MiB): 9.488884925842285 MiB.
[Info] Concurrent Store/Load v2; Stage 3c; PUT Phase; SUCCESS Response Count: 19991.0.
[Info] Concurrent Store/Load v2; Stage 3c; PUT Phase; % SUCCESS: 99.955%.
[Info] Concurrent Store/Load v2; Stage 3c; PUT Phase; % FAILED (Not SUCCESS): 0.045%.
[Info] Concurrent Store/Load v2; Stage 3c; PUT Phase; % TIMEOUT: 0.045%.
[Info] Concurrent Store/Load v2; Stage 3c; PUT Phase; % KEY-NOT-FOUND: 0.0%.
[Info] Concurrent Store/Load v2; Stage 3c; PUT Phase; % ERROR-UID: 0.0%.
[Info] Concurrent Store/Load v2; Stage 3c; PUT Phase; % ERROR-INVALID-GET (value or version): 0.0%.
[Info] Concurrent Store/Load v2; Stage 3c; PUT Phase; % ERROR-OTHER: 0.0%.
[Info] PUTs % SUCCESS Ok: 99.955% Success.(90.0% minimum required).
Checking that 34 nodes are alive
[Test isAlive] for 3.99.144.144:12379
[Test isAlive] for 3.99.144.144:12364
[Test isAlive] for 3.99.144.144:12368
[Test isAlive] for 3.99.144.144:12357
[Test isAlive] for 3.99.144.144:12350
[Test isAlive] for 3.99.144.144:12369
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12354
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12386
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12382
[Test isAlive] for 3.99.144.144:12381
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12367
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12385
[Test isAlive] for 3.99.144.144:12380
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12349
[ ... Completed in 0.057 seconds ]
TEST_PASSED
[ ... TEST Completed in 4.441 seconds ]
[Info] Will suspend 3.99.144.144:12379
[Info] Will suspend 3.99.144.144:12364
[Info] Will suspend 3.99.144.144:12368
Waiting 10 seconds before suspending nodes
[Info] Suspending 3 servers.
Server 3.99.144.144:12364 suspended
Server 3.99.144.144:12379 suspended
Server 3.99.144.144:12368 suspended
[ ... Completed in 4.092 seconds ]
Waiting 30 seconds after suspending nodes
[Info] Stage 3 Scenario: Alive Nodes: 31. Dead Nodes: 9
[ ... TEST Completed in 48.534 seconds ]


[ TEST Sequential Consistency; Stage 3c; (Clients = 128, Limit = 40000 GETS) ]
Checking that 9 nodes are suspended
[Test isAlive] for 3.99.144.144:12378
[Test isAlive] for 3.99.144.144:12353
[Test isAlive] for 3.99.144.144:12372
[Test isAlive] for 3.99.144.144:12377
[Test isAlive] for 3.99.144.144:12365
[Test isAlive] for 3.99.144.144:12371
[Test isAlive] for 3.99.144.144:12379
[Test isAlive] for 3.99.144.144:12364
[Test isAlive] for 3.99.144.144:12368
[ ... Completed in 10.004 seconds ]
Checking that 31 nodes are alive
[Test isAlive] for 3.99.144.144:12357
[Test isAlive] for 3.99.144.144:12350
[Test isAlive] for 3.99.144.144:12369
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12354
[Test isAlive] for 3.99.144.144:12386
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12382
[Test isAlive] for 3.99.144.144:12381
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12367
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12385
[Test isAlive] for 3.99.144.144:12380
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12349
[ ... Completed in 0.008 seconds ]
Sending GetMembershipCount to all servers... 
[Info] Server 3.99.144.144:12357 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12350 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12363 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12369 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12354 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12386 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12355 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12376 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12370 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12361 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12375 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12360 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12382 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12381 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12351 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12383 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12348 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12358 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12366 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12384 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12359 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12367 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12347 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12362 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12374 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12352 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12385 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12380 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12373 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12356 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12349 replied with membership count = 31 (expected count = 31)
[ ... Completed in 0.008 seconds ]
[Ok] All servers returned the expected membership count (31)
Attempting retrieval of keys... 
Issuing GETs ... 
[DONE in 3.384s]
[Info] Sequential Consistency; Stage 3c; GET Phase; Of 39992 stored keys: 39992 SUCCESS, 0 TIMEOUT, 0 KEY-NOT-FOUND, 0 ERROR-UID, 0 ERROR-INVALID-GET (value or version), 0 ERROR-OTHER.
[Info] Sequential Consistency; Stage 3c; GET Phase; Expected Key Loss: 0.0%. Actual Key Loss: 0.0%
[Info] Sequential Consistency; Stage 3c; GET Phase; Total Value Size (MiB): 19.029239654541016 MiB.
[Info] Sequential Consistency; Stage 3c; GET Phase; SUCCESS Response Count: 39992.0.
[Info] Sequential Consistency; Stage 3c; GET Phase; % SUCCESS: 100.0%.
[Info] Sequential Consistency; Stage 3c; GET Phase; % FAILED (Not SUCCESS): 0.0%.
[Info] Sequential Consistency; Stage 3c; GET Phase; % TIMEOUT: 0.0%.
[Info] Sequential Consistency; Stage 3c; GET Phase; % KEY-NOT-FOUND: 0.0%.
[Info] Sequential Consistency; Stage 3c; GET Phase; % ERROR-UID: 0.0%.
[Info] Sequential Consistency; Stage 3c; GET Phase; % ERROR-INVALID-GET (value or version): 0.0%.
[Info] Sequential Consistency; Stage 3c; GET Phase; % ERROR-OTHER: 0.0%.
[Info] Sequential Consistency; Stage 3c; GET Phase; NOT-STORED keys Count: 8.0.
Checking that 31 nodes are alive
[Test isAlive] for 3.99.144.144:12357
[Test isAlive] for 3.99.144.144:12350
[Test isAlive] for 3.99.144.144:12369
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12354
[Test isAlive] for 3.99.144.144:12386
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12382
[Test isAlive] for 3.99.144.144:12381
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12367
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12385
[Test isAlive] for 3.99.144.144:12380
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12349
[ ... Completed in 0.009 seconds ]
TEST_PASSED
[ TEST Concurrent Store/Load v2; Stage 3d; (Clients = 128, Limit = 20000 PUTs) ]
Issuing RELOADS ... 
[DONE in 3.569s]
[Info] Concurrent Store/Load v2; Stage 3d; PUT Phase; Of 20000 stored keys: 20000 SUCCESS, 0 TIMEOUT, 0 KEY-NOT-FOUND, 0 ERROR-UID, 0 ERROR-INVALID-GET (value or version), 0 ERROR-OTHER.
[Info] Concurrent Store/Load v2; Stage 3d; PUT Phase; Expected Key Loss: 0.0%. Actual Key Loss: 0.0%
[Info] Concurrent Store/Load v2; Stage 3d; PUT Phase; Total Value Size (MiB): 9.57327651977539 MiB.
[Info] Concurrent Store/Load v2; Stage 3d; PUT Phase; SUCCESS Response Count: 20000.0.
[Info] Concurrent Store/Load v2; Stage 3d; PUT Phase; % SUCCESS: 100.0%.
[Info] Concurrent Store/Load v2; Stage 3d; PUT Phase; % FAILED (Not SUCCESS): 0.0%.
[Info] Concurrent Store/Load v2; Stage 3d; PUT Phase; % TIMEOUT: 0.0%.
[Info] Concurrent Store/Load v2; Stage 3d; PUT Phase; % KEY-NOT-FOUND: 0.0%.
[Info] Concurrent Store/Load v2; Stage 3d; PUT Phase; % ERROR-UID: 0.0%.
[Info] Concurrent Store/Load v2; Stage 3d; PUT Phase; % ERROR-INVALID-GET (value or version): 0.0%.
[Info] Concurrent Store/Load v2; Stage 3d; PUT Phase; % ERROR-OTHER: 0.0%.
[Info] PUTs % SUCCESS Ok: 100.0% Success.(90.0% minimum required).
Checking that 31 nodes are alive
[Test isAlive] for 3.99.144.144:12357
[Test isAlive] for 3.99.144.144:12350
[Test isAlive] for 3.99.144.144:12369
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12354
[Test isAlive] for 3.99.144.144:12386
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12382
[Test isAlive] for 3.99.144.144:12381
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12367
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12385
[Test isAlive] for 3.99.144.144:12380
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12349
[ ... Completed in 0.01 seconds ]
TEST_PASSED
[ ... TEST Completed in 3.579 seconds ]
[Info] Will suspend 3.99.144.144:12357
[Info] Will suspend 3.99.144.144:12350
[Info] Will suspend 3.99.144.144:12369
Waiting 10 seconds before suspending nodes
[Info] Suspending 3 servers.
Server 3.99.144.144:12350 suspended
Server 3.99.144.144:12357 suspended
Server 3.99.144.144:12369 suspended
[ ... Completed in 4.117 seconds ]
Waiting 30 seconds after suspending nodes
[Info] Stage 3 Scenario: Alive Nodes: 28. Dead Nodes: 12
[ ... TEST Completed in 47.696 seconds ]


[ TEST Sequential Consistency; Stage 3d; (Clients = 128, Limit = 40000 GETS) ]
Checking that 12 nodes are suspended
[Test isAlive] for 3.99.144.144:12378
[Test isAlive] for 3.99.144.144:12353
[Test isAlive] for 3.99.144.144:12372
[Test isAlive] for 3.99.144.144:12377
[Test isAlive] for 3.99.144.144:12365
[Test isAlive] for 3.99.144.144:12371
[Test isAlive] for 3.99.144.144:12379
[Test isAlive] for 3.99.144.144:12364
[Test isAlive] for 3.99.144.144:12368
[Test isAlive] for 3.99.144.144:12357
[Test isAlive] for 3.99.144.144:12350
[Test isAlive] for 3.99.144.144:12369
[ ... Completed in 10.004 seconds ]
Checking that 28 nodes are alive
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12354
[Test isAlive] for 3.99.144.144:12386
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12382
[Test isAlive] for 3.99.144.144:12381
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12367
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12385
[Test isAlive] for 3.99.144.144:12380
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12349
[ ... Completed in 0.008 seconds ]
Sending GetMembershipCount to all servers... 
[Info] Server 3.99.144.144:12363 replied with membership count = 28 (expected count = 28)
[Info] Server 3.99.144.144:12354 replied with membership count = 28 (expected count = 28)
[Info] Server 3.99.144.144:12386 replied with membership count = 28 (expected count = 28)
[Info] Server 3.99.144.144:12355 replied with membership count = 28 (expected count = 28)
[Info] Server 3.99.144.144:12376 replied with membership count = 28 (expected count = 28)
[Info] Server 3.99.144.144:12370 replied with membership count = 28 (expected count = 28)
[Info] Server 3.99.144.144:12375 replied with membership count = 28 (expected count = 28)
[Info] Server 3.99.144.144:12361 replied with membership count = 28 (expected count = 28)
[Info] Server 3.99.144.144:12360 replied with membership count = 28 (expected count = 28)
[Info] Server 3.99.144.144:12382 replied with membership count = 28 (expected count = 28)
[Info] Server 3.99.144.144:12381 replied with membership count = 28 (expected count = 28)
[Info] Server 3.99.144.144:12351 replied with membership count = 28 (expected count = 28)
[Info] Server 3.99.144.144:12383 replied with membership count = 28 (expected count = 28)
[Info] Server 3.99.144.144:12348 replied with membership count = 28 (expected count = 28)
[Info] Server 3.99.144.144:12358 replied with membership count = 28 (expected count = 28)
[Info] Server 3.99.144.144:12366 replied with membership count = 28 (expected count = 28)
[Info] Server 3.99.144.144:12367 replied with membership count = 28 (expected count = 28)
[Info] Server 3.99.144.144:12384 replied with membership count = 28 (expected count = 28)
[Info] Server 3.99.144.144:12347 replied with membership count = 28 (expected count = 28)
[Info] Server 3.99.144.144:12362 replied with membership count = 28 (expected count = 28)
[Info] Server 3.99.144.144:12359 replied with membership count = 28 (expected count = 28)
[Info] Server 3.99.144.144:12352 replied with membership count = 28 (expected count = 28)
[Info] Server 3.99.144.144:12385 replied with membership count = 28 (expected count = 28)
[Info] Server 3.99.144.144:12380 replied with membership count = 28 (expected count = 28)
[Info] Server 3.99.144.144:12373 replied with membership count = 28 (expected count = 28)
[Info] Server 3.99.144.144:12356 replied with membership count = 28 (expected count = 28)
[Info] Server 3.99.144.144:12349 replied with membership count = 28 (expected count = 28)
[Info] Server 3.99.144.144:12374 replied with membership count = 28 (expected count = 28)
[ ... Completed in 0.047 seconds ]
[Ok] All servers returned the expected membership count (28)
Attempting retrieval of keys... 
Issuing GETs ... 
[DONE in 3.416s]
[Info] Sequential Consistency; Stage 3d; GET Phase; Of 39995 stored keys: 39995 SUCCESS, 0 TIMEOUT, 0 KEY-NOT-FOUND, 0 ERROR-UID, 0 ERROR-INVALID-GET (value or version), 0 ERROR-OTHER.
[Info] Sequential Consistency; Stage 3d; GET Phase; Expected Key Loss: 0.0%. Actual Key Loss: 0.0%
[Info] Sequential Consistency; Stage 3d; GET Phase; Total Value Size (MiB): 19.092496871948242 MiB.
[Info] Sequential Consistency; Stage 3d; GET Phase; SUCCESS Response Count: 39995.0.
[Info] Sequential Consistency; Stage 3d; GET Phase; % SUCCESS: 100.0%.
[Info] Sequential Consistency; Stage 3d; GET Phase; % FAILED (Not SUCCESS): 0.0%.
[Info] Sequential Consistency; Stage 3d; GET Phase; % TIMEOUT: 0.0%.
[Info] Sequential Consistency; Stage 3d; GET Phase; % KEY-NOT-FOUND: 0.0%.
[Info] Sequential Consistency; Stage 3d; GET Phase; % ERROR-UID: 0.0%.
[Info] Sequential Consistency; Stage 3d; GET Phase; % ERROR-INVALID-GET (value or version): 0.0%.
[Info] Sequential Consistency; Stage 3d; GET Phase; % ERROR-OTHER: 0.0%.
[Info] Sequential Consistency; Stage 3d; GET Phase; NOT-STORED keys Count: 5.0.
Checking that 28 nodes are alive
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12354
[Test isAlive] for 3.99.144.144:12386
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12382
[Test isAlive] for 3.99.144.144:12381
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12367
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12385
[Test isAlive] for 3.99.144.144:12380
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12349
[ ... Completed in 0.01 seconds ]
TEST_PASSED
[ TEST Concurrent Store/Load v2; Stage 3e; (Clients = 128, Limit = 20000 PUTs) ]
Issuing RELOADS ... 
[DONE in 3.551s]
[Info] Concurrent Store/Load v2; Stage 3e; PUT Phase; Of 20000 stored keys: 20000 SUCCESS, 0 TIMEOUT, 0 KEY-NOT-FOUND, 0 ERROR-UID, 0 ERROR-INVALID-GET (value or version), 0 ERROR-OTHER.
[Info] Concurrent Store/Load v2; Stage 3e; PUT Phase; Expected Key Loss: 0.0%. Actual Key Loss: 0.0%
[Info] Concurrent Store/Load v2; Stage 3e; PUT Phase; Total Value Size (MiB): 9.589224815368652 MiB.
[Info] Concurrent Store/Load v2; Stage 3e; PUT Phase; SUCCESS Response Count: 20000.0.
[Info] Concurrent Store/Load v2; Stage 3e; PUT Phase; % SUCCESS: 100.0%.
[Info] Concurrent Store/Load v2; Stage 3e; PUT Phase; % FAILED (Not SUCCESS): 0.0%.
[Info] Concurrent Store/Load v2; Stage 3e; PUT Phase; % TIMEOUT: 0.0%.
[Info] Concurrent Store/Load v2; Stage 3e; PUT Phase; % KEY-NOT-FOUND: 0.0%.
[Info] Concurrent Store/Load v2; Stage 3e; PUT Phase; % ERROR-UID: 0.0%.
[Info] Concurrent Store/Load v2; Stage 3e; PUT Phase; % ERROR-INVALID-GET (value or version): 0.0%.
[Info] Concurrent Store/Load v2; Stage 3e; PUT Phase; % ERROR-OTHER: 0.0%.
[Info] PUTs % SUCCESS Ok: 100.0% Success.(90.0% minimum required).
Checking that 28 nodes are alive
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12354
[Test isAlive] for 3.99.144.144:12386
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12382
[Test isAlive] for 3.99.144.144:12381
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12367
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12385
[Test isAlive] for 3.99.144.144:12380
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12349
[ ... Completed in 0.009 seconds ]
TEST_PASSED
[ ... TEST Completed in 3.561 seconds ]
[Info] Will resume 3.99.144.144:12369
[Info] Will resume 3.99.144.144:12372
[Info] Will resume 3.99.144.144:12357
Waiting 10 seconds before resuming nodes
[Info] Resuming 3 servers.
Server 3.99.144.144:12369 resumed
Server 3.99.144.144:12372 resumed
Server 3.99.144.144:12357 resumed
[ ... Completed in 4.114 seconds ]
Waiting 30 seconds after resuming nodes
[Info] Stage 3e Scenario: Alive Nodes: 31. Dead Nodes: 9
[ ... TEST Completed in 47.676 seconds ]


[ TEST Sequential Consistency; Stage 3e; (Clients = 128, Limit = 40000 GETS) ]
Checking that 9 nodes are suspended
[Test isAlive] for 3.99.144.144:12371
[Test isAlive] for 3.99.144.144:12365
[Test isAlive] for 3.99.144.144:12378
[Test isAlive] for 3.99.144.144:12377
[Test isAlive] for 3.99.144.144:12368
[Test isAlive] for 3.99.144.144:12350
[Test isAlive] for 3.99.144.144:12364
[Test isAlive] for 3.99.144.144:12379
[Test isAlive] for 3.99.144.144:12353
[ ... Completed in 10.003 seconds ]
Checking that 31 nodes are alive
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12354
[Test isAlive] for 3.99.144.144:12386
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12382
[Test isAlive] for 3.99.144.144:12381
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12367
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12385
[Test isAlive] for 3.99.144.144:12380
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12349
[Test isAlive] for 3.99.144.144:12369
[Test isAlive] for 3.99.144.144:12372
[Test isAlive] for 3.99.144.144:12357
[ ... Completed in 0.009 seconds ]
Sending GetMembershipCount to all servers... 
[Info] Server 3.99.144.144:12363 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12354 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12386 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12355 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12376 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12375 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12370 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12361 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12360 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12382 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12351 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12381 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12383 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12348 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12358 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12366 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12367 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12384 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12359 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12347 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12362 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12374 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12352 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12385 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12373 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12380 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12356 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12349 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12372 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12369 replied with membership count = 31 (expected count = 31)
[Info] Server 3.99.144.144:12357 replied with membership count = 31 (expected count = 31)
[ ... Completed in 0.008 seconds ]
[Ok] All servers returned the expected membership count (31)
Attempting retrieval of keys... 
Issuing GETs ... 
[DONE in 3.438s]
[Info] Sequential Consistency; Stage 3e; GET Phase; Of 39999 stored keys: 39977 SUCCESS, 0 TIMEOUT, 0 KEY-NOT-FOUND, 0 ERROR-UID, 22 ERROR-INVALID-GET (value or version), 0 ERROR-OTHER.
[Info] Sequential Consistency; Stage 3e; GET Phase; Expected Key Loss: 0.0%. Actual Key Loss: 0.05500137503437586%
[Info] Sequential Consistency; Stage 3e; GET Phase; Total Value Size (MiB): 19.152201652526855 MiB.
[Info] Sequential Consistency; Stage 3e; GET Phase; SUCCESS Response Count: 39977.0.
[Info] Sequential Consistency; Stage 3e; GET Phase; % SUCCESS: 99.94499862496562%.
[Info] Sequential Consistency; Stage 3e; GET Phase; % FAILED (Not SUCCESS): 0.05500137503437586%.
[Info] Sequential Consistency; Stage 3e; GET Phase; % TIMEOUT: 0.0%.
[Info] Sequential Consistency; Stage 3e; GET Phase; % KEY-NOT-FOUND: 0.0%.
[Info] Sequential Consistency; Stage 3e; GET Phase; % ERROR-UID: 0.0%.
[Info] Sequential Consistency; Stage 3e; GET Phase; % ERROR-INVALID-GET (value or version): 0.05500137503437586%.
[Info] Sequential Consistency; Stage 3e; GET Phase; % ERROR-OTHER: 0.0%.
[Info] Sequential Consistency; Stage 3e; GET Phase; NOT-STORED keys Count: 1.0.
Checking that 31 nodes are alive
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12354
[Test isAlive] for 3.99.144.144:12386
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12382
[Test isAlive] for 3.99.144.144:12381
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12367
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12385
[Test isAlive] for 3.99.144.144:12380
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12349
[Test isAlive] for 3.99.144.144:12369
[Test isAlive] for 3.99.144.144:12372
[Test isAlive] for 3.99.144.144:12357
[ ... Completed in 0.009 seconds ]
TEST_PASSED
[ TEST Concurrent Store/Load v2; Stage 3f; (Clients = 128, Limit = 20000 PUTs) ]
Issuing RELOADS ... 
[DONE in 3.508s]
[Info] Concurrent Store/Load v2; Stage 3f; PUT Phase; Of 20000 stored keys: 20000 SUCCESS, 0 TIMEOUT, 0 KEY-NOT-FOUND, 0 ERROR-UID, 0 ERROR-INVALID-GET (value or version), 0 ERROR-OTHER.
[Info] Concurrent Store/Load v2; Stage 3f; PUT Phase; Expected Key Loss: 0.0%. Actual Key Loss: 0.0%
[Info] Concurrent Store/Load v2; Stage 3f; PUT Phase; Total Value Size (MiB): 9.530343055725098 MiB.
[Info] Concurrent Store/Load v2; Stage 3f; PUT Phase; SUCCESS Response Count: 20000.0.
[Info] Concurrent Store/Load v2; Stage 3f; PUT Phase; % SUCCESS: 100.0%.
[Info] Concurrent Store/Load v2; Stage 3f; PUT Phase; % FAILED (Not SUCCESS): 0.0%.
[Info] Concurrent Store/Load v2; Stage 3f; PUT Phase; % TIMEOUT: 0.0%.
[Info] Concurrent Store/Load v2; Stage 3f; PUT Phase; % KEY-NOT-FOUND: 0.0%.
[Info] Concurrent Store/Load v2; Stage 3f; PUT Phase; % ERROR-UID: 0.0%.
[Info] Concurrent Store/Load v2; Stage 3f; PUT Phase; % ERROR-INVALID-GET (value or version): 0.0%.
[Info] Concurrent Store/Load v2; Stage 3f; PUT Phase; % ERROR-OTHER: 0.0%.
[Info] PUTs % SUCCESS Ok: 100.0% Success.(90.0% minimum required).
Checking that 31 nodes are alive
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12354
[Test isAlive] for 3.99.144.144:12386
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12382
[Test isAlive] for 3.99.144.144:12381
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12367
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12385
[Test isAlive] for 3.99.144.144:12380
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12349
[Test isAlive] for 3.99.144.144:12369
[Test isAlive] for 3.99.144.144:12372
[Test isAlive] for 3.99.144.144:12357
[ ... Completed in 0.009 seconds ]
TEST_PASSED
[ ... TEST Completed in 3.517 seconds ]
[Info] Will resume 3.99.144.144:12371
[Info] Will resume 3.99.144.144:12378
[Info] Will resume 3.99.144.144:12368
Waiting 10 seconds before resuming nodes
[Info] Resuming 3 servers.
Server 3.99.144.144:12378 resumed
Server 3.99.144.144:12368 resumed
Server 3.99.144.144:12371 resumed
[ ... Completed in 4.114 seconds ]
Waiting 30 seconds after resuming nodes
[Info] Stage 3f Scenario: Alive Nodes: 34. Dead Nodes: 6
[ ... TEST Completed in 47.631 seconds ]


[ TEST Sequential Consistency; Stage 3f; (Clients = 128, Limit = 40000 GETS) ]
Checking that 6 nodes are suspended
[Test isAlive] for 3.99.144.144:12365
[Test isAlive] for 3.99.144.144:12377
[Test isAlive] for 3.99.144.144:12350
[Test isAlive] for 3.99.144.144:12364
[Test isAlive] for 3.99.144.144:12379
[Test isAlive] for 3.99.144.144:12353
[ ... Completed in 10.004 seconds ]
Checking that 34 nodes are alive
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12354
[Test isAlive] for 3.99.144.144:12386
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12382
[Test isAlive] for 3.99.144.144:12381
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12367
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12385
[Test isAlive] for 3.99.144.144:12380
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12349
[Test isAlive] for 3.99.144.144:12369
[Test isAlive] for 3.99.144.144:12372
[Test isAlive] for 3.99.144.144:12357
[Test isAlive] for 3.99.144.144:12371
[Test isAlive] for 3.99.144.144:12378
[Test isAlive] for 3.99.144.144:12368
[ ... Completed in 0.009 seconds ]
Sending GetMembershipCount to all servers... 
[Info] Server 3.99.144.144:12363 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12354 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12386 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12355 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12376 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12361 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12370 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12360 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12375 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12382 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12381 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12351 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12383 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12348 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12358 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12367 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12366 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12384 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12359 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12362 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12374 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12347 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12352 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12385 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12380 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12373 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12356 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12349 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12369 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12372 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12371 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12357 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12368 replied with membership count = 34 (expected count = 34)
[Info] Server 3.99.144.144:12378 replied with membership count = 34 (expected count = 34)
[ ... Completed in 0.008 seconds ]
[Ok] All servers returned the expected membership count (34)
Attempting retrieval of keys... 
Issuing GETs ... 
[DONE in 3.413s]
[Info] Sequential Consistency; Stage 3f; GET Phase; Of 40000 stored keys: 39988 SUCCESS, 0 TIMEOUT, 0 KEY-NOT-FOUND, 0 ERROR-UID, 12 ERROR-INVALID-GET (value or version), 0 ERROR-OTHER.
[Info] Sequential Consistency; Stage 3f; GET Phase; Expected Key Loss: 0.0%. Actual Key Loss: 0.03%
[Info] Sequential Consistency; Stage 3f; GET Phase; Total Value Size (MiB): 19.11678695678711 MiB.
[Info] Sequential Consistency; Stage 3f; GET Phase; SUCCESS Response Count: 39988.0.
[Info] Sequential Consistency; Stage 3f; GET Phase; % SUCCESS: 99.97%.
[Info] Sequential Consistency; Stage 3f; GET Phase; % FAILED (Not SUCCESS): 0.03%.
[Info] Sequential Consistency; Stage 3f; GET Phase; % TIMEOUT: 0.0%.
[Info] Sequential Consistency; Stage 3f; GET Phase; % KEY-NOT-FOUND: 0.0%.
[Info] Sequential Consistency; Stage 3f; GET Phase; % ERROR-UID: 0.0%.
[Info] Sequential Consistency; Stage 3f; GET Phase; % ERROR-INVALID-GET (value or version): 0.03%.
[Info] Sequential Consistency; Stage 3f; GET Phase; % ERROR-OTHER: 0.0%.
[Info] Sequential Consistency; Stage 3f; GET Phase; NOT-STORED keys Count: 0.0.
Checking that 34 nodes are alive
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12354
[Test isAlive] for 3.99.144.144:12386
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12382
[Test isAlive] for 3.99.144.144:12381
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12367
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12385
[Test isAlive] for 3.99.144.144:12380
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12349
[Test isAlive] for 3.99.144.144:12369
[Test isAlive] for 3.99.144.144:12372
[Test isAlive] for 3.99.144.144:12357
[Test isAlive] for 3.99.144.144:12371
[Test isAlive] for 3.99.144.144:12378
[Test isAlive] for 3.99.144.144:12368
[ ... Completed in 0.009 seconds ]
TEST_PASSED
[ TEST Concurrent Store/Load v2; Stage 3g; (Clients = 128, Limit = 20000 PUTs) ]
Issuing RELOADS ... 
[DONE in 3.625s]
[Info] Concurrent Store/Load v2; Stage 3g; PUT Phase; Of 20000 stored keys: 20000 SUCCESS, 0 TIMEOUT, 0 KEY-NOT-FOUND, 0 ERROR-UID, 0 ERROR-INVALID-GET (value or version), 0 ERROR-OTHER.
[Info] Concurrent Store/Load v2; Stage 3g; PUT Phase; Expected Key Loss: 0.0%. Actual Key Loss: 0.0%
[Info] Concurrent Store/Load v2; Stage 3g; PUT Phase; Total Value Size (MiB): 9.61846923828125 MiB.
[Info] Concurrent Store/Load v2; Stage 3g; PUT Phase; SUCCESS Response Count: 20000.0.
[Info] Concurrent Store/Load v2; Stage 3g; PUT Phase; % SUCCESS: 100.0%.
[Info] Concurrent Store/Load v2; Stage 3g; PUT Phase; % FAILED (Not SUCCESS): 0.0%.
[Info] Concurrent Store/Load v2; Stage 3g; PUT Phase; % TIMEOUT: 0.0%.
[Info] Concurrent Store/Load v2; Stage 3g; PUT Phase; % KEY-NOT-FOUND: 0.0%.
[Info] Concurrent Store/Load v2; Stage 3g; PUT Phase; % ERROR-UID: 0.0%.
[Info] Concurrent Store/Load v2; Stage 3g; PUT Phase; % ERROR-INVALID-GET (value or version): 0.0%.
[Info] Concurrent Store/Load v2; Stage 3g; PUT Phase; % ERROR-OTHER: 0.0%.
[Info] PUTs % SUCCESS Ok: 100.0% Success.(90.0% minimum required).
Checking that 34 nodes are alive
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12354
[Test isAlive] for 3.99.144.144:12386
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12382
[Test isAlive] for 3.99.144.144:12381
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12367
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12385
[Test isAlive] for 3.99.144.144:12380
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12349
[Test isAlive] for 3.99.144.144:12369
[Test isAlive] for 3.99.144.144:12372
[Test isAlive] for 3.99.144.144:12357
[Test isAlive] for 3.99.144.144:12371
[Test isAlive] for 3.99.144.144:12378
[Test isAlive] for 3.99.144.144:12368
[ ... Completed in 0.009 seconds ]
TEST_PASSED
[ ... TEST Completed in 3.635 seconds ]
[Info] Will resume 3.99.144.144:12365
[Info] Will resume 3.99.144.144:12350
[Info] Will resume 3.99.144.144:12379
Waiting 10 seconds before resuming nodes
[Info] Resuming 3 servers.
Server 3.99.144.144:12350 resumed
Server 3.99.144.144:12365 resumed
Server 3.99.144.144:12379 resumed
[ ... Completed in 4.424 seconds ]
Waiting 30 seconds after resuming nodes
[Info] Stage 3g Scenario: Alive Nodes: 37. Dead Nodes: 3
[ ... TEST Completed in 48.059 seconds ]


[ TEST Sequential Consistency; Stage 3g; (Clients = 128, Limit = 40000 GETS) ]
Checking that 3 nodes are suspended
[Test isAlive] for 3.99.144.144:12377
[Test isAlive] for 3.99.144.144:12364
[Test isAlive] for 3.99.144.144:12353
[ ... Completed in 10.004 seconds ]
Checking that 37 nodes are alive
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12354
[Test isAlive] for 3.99.144.144:12386
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12382
[Test isAlive] for 3.99.144.144:12381
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12367
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12385
[Test isAlive] for 3.99.144.144:12380
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12349
[Test isAlive] for 3.99.144.144:12369
[Test isAlive] for 3.99.144.144:12372
[Test isAlive] for 3.99.144.144:12357
[Test isAlive] for 3.99.144.144:12371
[Test isAlive] for 3.99.144.144:12378
[Test isAlive] for 3.99.144.144:12368
[Test isAlive] for 3.99.144.144:12365
[Test isAlive] for 3.99.144.144:12350
[Test isAlive] for 3.99.144.144:12379
[ ... Completed in 0.009 seconds ]
Sending GetMembershipCount to all servers... 
[Info] Server 3.99.144.144:12363 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12354 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12375 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12376 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12386 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12370 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12382 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12355 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12361 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12351 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12381 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12360 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12383 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12348 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12366 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12358 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12367 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12359 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12384 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12347 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12374 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12362 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12352 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12385 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12380 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12373 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12356 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12369 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12349 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12372 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12357 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12371 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12378 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12368 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12350 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12365 replied with membership count = 37 (expected count = 37)
[Info] Server 3.99.144.144:12379 replied with membership count = 37 (expected count = 37)
[ ... Completed in 0.009 seconds ]
[Ok] All servers returned the expected membership count (37)
Attempting retrieval of keys... 
Issuing GETs ... 
[DONE in 3.435s]
[Info] Sequential Consistency; Stage 3g; GET Phase; Of 40000 stored keys: 39969 SUCCESS, 0 TIMEOUT, 0 KEY-NOT-FOUND, 0 ERROR-UID, 31 ERROR-INVALID-GET (value or version), 0 ERROR-OTHER.
[Info] Sequential Consistency; Stage 3g; GET Phase; Expected Key Loss: 0.0%. Actual Key Loss: 0.0775%
[Info] Sequential Consistency; Stage 3g; GET Phase; Total Value Size (MiB): 19.1300630569458 MiB.
[Info] Sequential Consistency; Stage 3g; GET Phase; SUCCESS Response Count: 39969.0.
[Info] Sequential Consistency; Stage 3g; GET Phase; % SUCCESS: 99.9225%.
[Info] Sequential Consistency; Stage 3g; GET Phase; % FAILED (Not SUCCESS): 0.0775%.
[Info] Sequential Consistency; Stage 3g; GET Phase; % TIMEOUT: 0.0%.
[Info] Sequential Consistency; Stage 3g; GET Phase; % KEY-NOT-FOUND: 0.0%.
[Info] Sequential Consistency; Stage 3g; GET Phase; % ERROR-UID: 0.0%.
[Info] Sequential Consistency; Stage 3g; GET Phase; % ERROR-INVALID-GET (value or version): 0.0775%.
[Info] Sequential Consistency; Stage 3g; GET Phase; % ERROR-OTHER: 0.0%.
[Info] Sequential Consistency; Stage 3g; GET Phase; NOT-STORED keys Count: 0.0.
Checking that 37 nodes are alive
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12354
[Test isAlive] for 3.99.144.144:12386
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12382
[Test isAlive] for 3.99.144.144:12381
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12367
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12385
[Test isAlive] for 3.99.144.144:12380
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12349
[Test isAlive] for 3.99.144.144:12369
[Test isAlive] for 3.99.144.144:12372
[Test isAlive] for 3.99.144.144:12357
[Test isAlive] for 3.99.144.144:12371
[Test isAlive] for 3.99.144.144:12368
[Test isAlive] for 3.99.144.144:12378
[Test isAlive] for 3.99.144.144:12365
[Test isAlive] for 3.99.144.144:12350
[Test isAlive] for 3.99.144.144:12379
[ ... Completed in 0.01 seconds ]
TEST_PASSED
[ TEST Concurrent Store/Load v2; Stage 3h; (Clients = 128, Limit = 20000 PUTs) ]
Issuing RELOADS ... 
[DONE in 3.588s]
[Info] Concurrent Store/Load v2; Stage 3h; PUT Phase; Of 20000 stored keys: 20000 SUCCESS, 0 TIMEOUT, 0 KEY-NOT-FOUND, 0 ERROR-UID, 0 ERROR-INVALID-GET (value or version), 0 ERROR-OTHER.
[Info] Concurrent Store/Load v2; Stage 3h; PUT Phase; Expected Key Loss: 0.0%. Actual Key Loss: 0.0%
[Info] Concurrent Store/Load v2; Stage 3h; PUT Phase; Total Value Size (MiB): 9.634326934814453 MiB.
[Info] Concurrent Store/Load v2; Stage 3h; PUT Phase; SUCCESS Response Count: 20000.0.
[Info] Concurrent Store/Load v2; Stage 3h; PUT Phase; % SUCCESS: 100.0%.
[Info] Concurrent Store/Load v2; Stage 3h; PUT Phase; % FAILED (Not SUCCESS): 0.0%.
[Info] Concurrent Store/Load v2; Stage 3h; PUT Phase; % TIMEOUT: 0.0%.
[Info] Concurrent Store/Load v2; Stage 3h; PUT Phase; % KEY-NOT-FOUND: 0.0%.
[Info] Concurrent Store/Load v2; Stage 3h; PUT Phase; % ERROR-UID: 0.0%.
[Info] Concurrent Store/Load v2; Stage 3h; PUT Phase; % ERROR-INVALID-GET (value or version): 0.0%.
[Info] Concurrent Store/Load v2; Stage 3h; PUT Phase; % ERROR-OTHER: 0.0%.
[Info] PUTs % SUCCESS Ok: 100.0% Success.(90.0% minimum required).
Checking that 37 nodes are alive
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12354
[Test isAlive] for 3.99.144.144:12386
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12382
[Test isAlive] for 3.99.144.144:12381
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12367
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12385
[Test isAlive] for 3.99.144.144:12380
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12349
[Test isAlive] for 3.99.144.144:12372
[Test isAlive] for 3.99.144.144:12369
[Test isAlive] for 3.99.144.144:12357
[Test isAlive] for 3.99.144.144:12371
[Test isAlive] for 3.99.144.144:12378
[Test isAlive] for 3.99.144.144:12368
[Test isAlive] for 3.99.144.144:12365
[Test isAlive] for 3.99.144.144:12350
[Test isAlive] for 3.99.144.144:12379
[ ... Completed in 0.01 seconds ]
TEST_PASSED
[ ... TEST Completed in 3.598 seconds ]
[Info] Will resume 3.99.144.144:12377
[Info] Will resume 3.99.144.144:12353
Waiting 10 seconds before resuming nodes
[Info] Resuming 2 servers.
Server 3.99.144.144:12377 resumed
Server 3.99.144.144:12353 resumed
[ ... Completed in 2.835 seconds ]
Waiting 30 seconds after resuming nodes
[Info] Stage 3h Scenario: Alive Nodes: 39. Dead Nodes: 1
[ ... TEST Completed in 46.433 seconds ]


[ TEST Sequential Consistency; Stage 3h; (Clients = 128, Limit = 40000 GETS) ]
Checking that 1 nodes are suspended
[Test isAlive] for 3.99.144.144:12364
[ ... Completed in 10.003 seconds ]
Checking that 39 nodes are alive
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12354
[Test isAlive] for 3.99.144.144:12386
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12382
[Test isAlive] for 3.99.144.144:12381
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12367
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12385
[Test isAlive] for 3.99.144.144:12380
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12349
[Test isAlive] for 3.99.144.144:12369
[Test isAlive] for 3.99.144.144:12372
[Test isAlive] for 3.99.144.144:12357
[Test isAlive] for 3.99.144.144:12371
[Test isAlive] for 3.99.144.144:12378
[Test isAlive] for 3.99.144.144:12368
[Test isAlive] for 3.99.144.144:12365
[Test isAlive] for 3.99.144.144:12350
[Test isAlive] for 3.99.144.144:12379
[Test isAlive] for 3.99.144.144:12377
[Test isAlive] for 3.99.144.144:12353
[ ... Completed in 0.01 seconds ]
Sending GetMembershipCount to all servers... 
[Info] Server 3.99.144.144:12363 replied with membership count = 39 (expected count = 39)
[Info] Server 3.99.144.144:12354 replied with membership count = 39 (expected count = 39)
[Info] Server 3.99.144.144:12355 replied with membership count = 39 (expected count = 39)
[Info] Server 3.99.144.144:12386 replied with membership count = 39 (expected count = 39)
[Info] Server 3.99.144.144:12376 replied with membership count = 39 (expected count = 39)
[Info] Server 3.99.144.144:12370 replied with membership count = 39 (expected count = 39)
[Info] Server 3.99.144.144:12375 replied with membership count = 39 (expected count = 39)
[Info] Server 3.99.144.144:12360 replied with membership count = 39 (expected count = 39)
[Info] Server 3.99.144.144:12361 replied with membership count = 39 (expected count = 39)
[Info] Server 3.99.144.144:12382 replied with membership count = 39 (expected count = 39)
[Info] Server 3.99.144.144:12381 replied with membership count = 39 (expected count = 39)
[Info] Server 3.99.144.144:12383 replied with membership count = 39 (expected count = 39)
[Info] Server 3.99.144.144:12351 replied with membership count = 39 (expected count = 39)
[Info] Server 3.99.144.144:12348 replied with membership count = 39 (expected count = 39)
[Info] Server 3.99.144.144:12358 replied with membership count = 39 (expected count = 39)
[Info] Server 3.99.144.144:12366 replied with membership count = 39 (expected count = 39)
[Info] Server 3.99.144.144:12367 replied with membership count = 39 (expected count = 39)
[Info] Server 3.99.144.144:12384 replied with membership count = 39 (expected count = 39)
[Info] Server 3.99.144.144:12359 replied with membership count = 39 (expected count = 39)
[Info] Server 3.99.144.144:12347 replied with membership count = 39 (expected count = 39)
[Info] Server 3.99.144.144:12362 replied with membership count = 39 (expected count = 39)
[Info] Server 3.99.144.144:12374 replied with membership count = 39 (expected count = 39)
[Info] Server 3.99.144.144:12352 replied with membership count = 39 (expected count = 39)
[Info] Server 3.99.144.144:12380 replied with membership count = 39 (expected count = 39)
[Info] Server 3.99.144.144:12385 replied with membership count = 39 (expected count = 39)
[Info] Server 3.99.144.144:12356 replied with membership count = 39 (expected count = 39)
[Info] Server 3.99.144.144:12373 replied with membership count = 39 (expected count = 39)
[Info] Server 3.99.144.144:12349 replied with membership count = 39 (expected count = 39)
[Info] Server 3.99.144.144:12369 replied with membership count = 39 (expected count = 39)
[Info] Server 3.99.144.144:12372 replied with membership count = 39 (expected count = 39)
[Info] Server 3.99.144.144:12357 replied with membership count = 39 (expected count = 39)
[Info] Server 3.99.144.144:12378 replied with membership count = 39 (expected count = 39)
[Info] Server 3.99.144.144:12371 replied with membership count = 39 (expected count = 39)
[Info] Server 3.99.144.144:12365 replied with membership count = 39 (expected count = 39)
[Info] Server 3.99.144.144:12368 replied with membership count = 39 (expected count = 39)
[Info] Server 3.99.144.144:12350 replied with membership count = 39 (expected count = 39)
[Info] Server 3.99.144.144:12379 replied with membership count = 39 (expected count = 39)
[Info] Server 3.99.144.144:12377 replied with membership count = 39 (expected count = 39)
[Info] Server 3.99.144.144:12353 replied with membership count = 39 (expected count = 39)
[ ... Completed in 0.009 seconds ]
[Ok] All servers returned the expected membership count (39)
Attempting retrieval of keys... 
Issuing GETs ... 
[DONE in 3.454s]
[Info] Sequential Consistency; Stage 3h; GET Phase; Of 40000 stored keys: 39980 SUCCESS, 0 TIMEOUT, 0 KEY-NOT-FOUND, 0 ERROR-UID, 20 ERROR-INVALID-GET (value or version), 0 ERROR-OTHER.
[Info] Sequential Consistency; Stage 3h; GET Phase; Expected Key Loss: 0.0%. Actual Key Loss: 0.05%
[Info] Sequential Consistency; Stage 3h; GET Phase; Total Value Size (MiB): 19.244829177856445 MiB.
[Info] Sequential Consistency; Stage 3h; GET Phase; SUCCESS Response Count: 39980.0.
[Info] Sequential Consistency; Stage 3h; GET Phase; % SUCCESS: 99.95%.
[Info] Sequential Consistency; Stage 3h; GET Phase; % FAILED (Not SUCCESS): 0.05%.
[Info] Sequential Consistency; Stage 3h; GET Phase; % TIMEOUT: 0.0%.
[Info] Sequential Consistency; Stage 3h; GET Phase; % KEY-NOT-FOUND: 0.0%.
[Info] Sequential Consistency; Stage 3h; GET Phase; % ERROR-UID: 0.0%.
[Info] Sequential Consistency; Stage 3h; GET Phase; % ERROR-INVALID-GET (value or version): 0.05%.
[Info] Sequential Consistency; Stage 3h; GET Phase; % ERROR-OTHER: 0.0%.
[Info] Sequential Consistency; Stage 3h; GET Phase; NOT-STORED keys Count: 0.0.
Checking that 39 nodes are alive
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12354
[Test isAlive] for 3.99.144.144:12386
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12382
[Test isAlive] for 3.99.144.144:12381
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12367
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12385
[Test isAlive] for 3.99.144.144:12380
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12349
[Test isAlive] for 3.99.144.144:12369
[Test isAlive] for 3.99.144.144:12372
[Test isAlive] for 3.99.144.144:12357
[Test isAlive] for 3.99.144.144:12371
[Test isAlive] for 3.99.144.144:12378
[Test isAlive] for 3.99.144.144:12368
[Test isAlive] for 3.99.144.144:12365
[Test isAlive] for 3.99.144.144:12350
[Test isAlive] for 3.99.144.144:12379
[Test isAlive] for 3.99.144.144:12377
[Test isAlive] for 3.99.144.144:12353
[ ... Completed in 0.01 seconds ]
TEST_PASSED
Waiting 10 seconds before resuming nodes
[Info] Resuming 1 servers.
Server 3.99.144.144:12364 resumed
[ ... Completed in 1.538 seconds ]

Sending WIPE-OUT to all servers...
[ ... Completed in 0.025 seconds ]
[OK]
Waiting for 5 seconds for the cache to be cleared!
Waiting 60 seconds

[ TEST Concurrent Store/Load v3; Stage 4; (Clients = 128, Limit = 5000 PUTs) ]
Issuing PUTS ... 
[DONE in 0.969s]
[Info] PUTs % SUCCESS Ok: 100.0% Success.(90.0% minimum required).
Confirming keys were stored correctly.
Issuing GETs ... 
[DONE in 0.538s]
[Info] GETs % SUCCESS Ok: 100.0% Success.(90.0% minimum required).
Checking that 40 nodes are alive
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12349
[Test isAlive] for 3.99.144.144:12350
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12353
[Test isAlive] for 3.99.144.144:12354
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12357
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12362
[Test isAlive] for 3.99.144.144:12363
[Test isAlive] for 3.99.144.144:12364
[Test isAlive] for 3.99.144.144:12365
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12367
[Test isAlive] for 3.99.144.144:12368
[Test isAlive] for 3.99.144.144:12369
[Test isAlive] for 3.99.144.144:12370
[Test isAlive] for 3.99.144.144:12371
[Test isAlive] for 3.99.144.144:12372
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12377
[Test isAlive] for 3.99.144.144:12378
[Test isAlive] for 3.99.144.144:12379
[Test isAlive] for 3.99.144.144:12380
[Test isAlive] for 3.99.144.144:12381
[Test isAlive] for 3.99.144.144:12382
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12385
[Test isAlive] for 3.99.144.144:12386
[ ... Completed in 0.01 seconds ]
TEST_PASSED
[ ... TEST Completed in 1.518 seconds ]
Confirming randomly selected keys were stored correctly.
Issuing GETs ... 
[DONE in 0.089s]
Sequential Consistency 2; Update values stage 4a Increment 1;
Updating stored KV pairs ... 
Issuing RELOADS ... 
[DONE in 0.175s]
TEST_PASSED
Sequential Consistency 2; Update values stage 4a Increment 2;
Updating stored KV pairs ... 
Issuing RELOADS ... 
[DONE in 0.133s]
TEST_PASSED
Sequential Consistency 2; Update values stage 4a Increment 3;
Updating stored KV pairs ... 
Issuing RELOADS ... 
[DONE in 0.134s]
TEST_PASSED
Sequential Consistency 2; Update values stage 4a Increment 4;
Updating stored KV pairs ... 
Issuing RELOADS ... 
[DONE in 0.154s]
TEST_PASSED
Sequential Consistency 2; Update values stage 4a Increment 5;
Updating stored KV pairs ... 
Issuing RELOADS ... 
[DONE in 0.138s]
TEST_PASSED
[ ... TEST Completed in 0.834 seconds ]

[Info] Preparing for the sequential consistency Test 2
[Info] Will suspend 3.99.144.144:12355
[Info] Will suspend 3.99.144.144:12365
[Info] Will suspend 3.99.144.144:12359
[Info] Suspending 3 servers.
Server 3.99.144.144:12355 suspended
Server 3.99.144.144:12359 suspended
Server 3.99.144.144:12365 suspended
[ ... Completed in 4.129 seconds ]
Waiting 15 seconds after suspend
[ ... TEST Completed in 19.129 seconds ]
[Info] Stage 4a Scenario: Alive Nodes: 37. Dead Nodes: 3
Confirming randomly selected keys were stored correctly.
Issuing GETs ... 
[DONE in 0.093s]
Sequential Consistency 2; Update values stage 4b Increment 1;
Updating stored KV pairs ... 
Issuing RELOADS ... 
[DONE in 0.134s]
TEST_PASSED
Sequential Consistency 2; Update values stage 4b Increment 2;
Updating stored KV pairs ... 
Issuing RELOADS ... 
[DONE in 0.16s]
TEST_PASSED
Sequential Consistency 2; Update values stage 4b Increment 3;
Updating stored KV pairs ... 
Issuing RELOADS ... 
[DONE in 0.154s]
TEST_PASSED
[ ... TEST Completed in 0.546 seconds ]

[Info] Preparing for the sequential consistency Test 2
[Info] Will suspend 3.99.144.144:12364
[Info] Will suspend 3.99.144.144:12366
[Info] Will suspend 3.99.144.144:12377
[Info] Suspending 3 servers.
Server 3.99.144.144:12364 suspended
Server 3.99.144.144:12377 suspended
Server 3.99.144.144:12366 suspended
[ ... Completed in 4.391 seconds ]
Waiting 15 seconds after suspend
[ ... TEST Completed in 19.392 seconds ]
[Info] Stage 4b Scenario: Alive Nodes: 34. Dead Nodes: 6
Confirming randomly selected keys were stored correctly.
Issuing GETs ... 
[DONE in 0.095s]
Sequential Consistency 2; Update values stage 4c Increment 1;
Updating stored KV pairs ... 
Issuing RELOADS ... 
[DONE in 0.169s]
TEST_PASSED
Sequential Consistency 2; Update values stage 4c Increment 2;
Updating stored KV pairs ... 
Issuing RELOADS ... 
[DONE in 0.133s]
TEST_PASSED
Sequential Consistency 2; Update values stage 4c Increment 3;
Updating stored KV pairs ... 
Issuing RELOADS ... 
[DONE in 0.133s]
TEST_PASSED
[ ... TEST Completed in 0.535 seconds ]

[Info] Preparing for the sequential consistency Test 2
[Info] Will resume 3.99.144.144:12355
[Info] Will resume 3.99.144.144:12365
[Info] Will resume 3.99.144.144:12359
[Info] Resuming 3 servers.
Server 3.99.144.144:12359 resumed
Server 3.99.144.144:12365 resumed
Server 3.99.144.144:12355 resumed
[ ... Completed in 4.094 seconds ]
Waiting 15 seconds after resume
[ ... TEST Completed in 19.094 seconds ]
[Info] Stage 4c Scenario: Alive Nodes: 37. Dead Nodes: 3
Confirming randomly selected keys were stored correctly.
Issuing GETs ... 
[DONE in 0.093s]
Sequential Consistency 2; Update values stage 4d Increment 1;
Updating stored KV pairs ... 
Issuing RELOADS ... 
[DONE in 0.21s]
TEST_PASSED
Sequential Consistency 2; Update values stage 4d Increment 2;
Updating stored KV pairs ... 
Issuing RELOADS ... 
[DONE in 0.134s]
TEST_PASSED
Sequential Consistency 2; Update values stage 4d Increment 3;
Updating stored KV pairs ... 
Issuing RELOADS ... 
[DONE in 0.133s]
TEST_PASSED
Sequential Consistency 2; Update values stage 4d Increment 4;
Updating stored KV pairs ... 
Issuing RELOADS ... 
[DONE in 0.132s]
TEST_PASSED
[ ... TEST Completed in 0.708 seconds ]

[Info] Preparing for the sequential consistency Test 2
[Info] Will suspend 3.99.144.144:12381
[Info] Will suspend 3.99.144.144:12372
[Info] Will suspend 3.99.144.144:12367
[Info] Suspending 3 servers.
Server 3.99.144.144:12381 suspended
Server 3.99.144.144:12367 suspended
Server 3.99.144.144:12372 suspended
[ ... Completed in 4.168 seconds ]
Waiting 15 seconds after suspend
[ ... TEST Completed in 19.168 seconds ]
[Info] Stage 4d Scenario: Alive Nodes: 34. Dead Nodes: 6
Confirming randomly selected keys were stored correctly.
Issuing GETs ... 
[DONE in 0.095s]
Sequential Consistency 2; Update values stage 4e Increment 1;
Updating stored KV pairs ... 
Issuing RELOADS ... 
[DONE in 0.131s]
TEST_PASSED
Sequential Consistency 2; Update values stage 4e Increment 2;
Updating stored KV pairs ... 
Issuing RELOADS ... 
[DONE in 0.132s]
TEST_PASSED
Sequential Consistency 2; Update values stage 4e Increment 3;
Updating stored KV pairs ... 
Issuing RELOADS ... 
[DONE in 0.246s]
TEST_PASSED
Sequential Consistency 2; Update values stage 4e Increment 4;
Updating stored KV pairs ... 
Issuing RELOADS ... 
[DONE in 0.136s]
TEST_PASSED
Sequential Consistency 2; Update values stage 4e Increment 5;
Updating stored KV pairs ... 
Issuing RELOADS ... 
[DONE in 0.132s]
TEST_PASSED
[ ... TEST Completed in 0.879 seconds ]

[Info] Preparing for the sequential consistency Test 2
[Info] Will suspend 3.99.144.144:12384
[Info] Will suspend 3.99.144.144:12353
[Info] Will suspend 3.99.144.144:12368
[Info] Suspending 3 servers.
Server 3.99.144.144:12353 suspended
Server 3.99.144.144:12368 suspended
Server 3.99.144.144:12384 suspended
[ ... Completed in 4.092 seconds ]
Waiting 15 seconds after suspend
[ ... TEST Completed in 19.092 seconds ]
[Info] Stage 4e Scenario: Alive Nodes: 31. Dead Nodes: 9
Confirming randomly selected keys were stored correctly.
Issuing GETs ... 
[DONE in 0.092s]
Sequential Consistency 2; Update values stage 4f Increment 1;
Updating stored KV pairs ... 
Issuing RELOADS ... 
[DONE in 0.211s]
TEST_PASSED
[ ... TEST Completed in 0.305 seconds ]

[Info] Preparing for the sequential consistency Test 2
[Info] Will resume 3.99.144.144:12364
[Info] Will resume 3.99.144.144:12366
[Info] Will resume 3.99.144.144:12377
[Info] Resuming 3 servers.
Server 3.99.144.144:12366 resumed
Server 3.99.144.144:12364 resumed
Server 3.99.144.144:12377 resumed
[ ... Completed in 4.1 seconds ]
Waiting 15 seconds after resume
[ ... TEST Completed in 19.1 seconds ]
[Info] Stage 4f Scenario: Alive Nodes: 34. Dead Nodes: 6
Confirming randomly selected keys were stored correctly.
Issuing GETs ... 
[DONE in 0.091s]
Sequential Consistency 2; Update values stage 4g Increment 1;
Updating stored KV pairs ... 
Issuing RELOADS ... 
[DONE in 0.135s]
TEST_PASSED
Sequential Consistency 2; Update values stage 4g Increment 2;
Updating stored KV pairs ... 
Issuing RELOADS ... 
[DONE in 0.131s]
TEST_PASSED
Sequential Consistency 2; Update values stage 4g Increment 3;
Updating stored KV pairs ... 
Issuing RELOADS ... 
[DONE in 0.149s]
TEST_PASSED
Sequential Consistency 2; Update values stage 4g Increment 4;
Updating stored KV pairs ... 
Issuing RELOADS ... 
[DONE in 0.134s]
TEST_PASSED
[ ... TEST Completed in 0.645 seconds ]

[Info] Preparing for the sequential consistency Test 2
[Info] Will suspend 3.99.144.144:12362
[Info] Will suspend 3.99.144.144:12382
[Info] Will suspend 3.99.144.144:12363
[Info] Suspending 3 servers.
Server 3.99.144.144:12363 suspended
Server 3.99.144.144:12362 suspended
Server 3.99.144.144:12382 suspended
[ ... Completed in 4.094 seconds ]
Waiting 15 seconds after suspend
[ ... TEST Completed in 19.094 seconds ]
[Info] Stage 4g Scenario: Alive Nodes: 31. Dead Nodes: 9
Confirming randomly selected keys were stored correctly.
Issuing GETs ... 
[DONE in 0.087s]
Sequential Consistency 2; Update values stage 4h Increment 1;
Updating stored KV pairs ... 
Issuing RELOADS ... 
[DONE in 0.134s]
TEST_PASSED
Sequential Consistency 2; Update values stage 4h Increment 2;
Updating stored KV pairs ... 
Issuing RELOADS ... 
[DONE in 0.134s]
TEST_PASSED
Sequential Consistency 2; Update values stage 4h Increment 3;
Updating stored KV pairs ... 
Issuing RELOADS ... 
[DONE in 0.133s]
TEST_PASSED
[ ... TEST Completed in 0.492 seconds ]

[Info] Preparing for the sequential consistency Test 2
[Info] Will suspend 3.99.144.144:12370
[Info] Will suspend 3.99.144.144:12380
[Info] Will suspend 3.99.144.144:12379
[Info] Suspending 3 servers.
Server 3.99.144.144:12370 suspended
Server 3.99.144.144:12380 suspended
Server 3.99.144.144:12379 suspended
[ ... Completed in 4.124 seconds ]
Waiting 15 seconds after suspend
[ ... TEST Completed in 19.124 seconds ]
[Info] Stage 4h Scenario: Alive Nodes: 28. Dead Nodes: 12
Confirming randomly selected keys were stored correctly.
Issuing GETs ... 
[DONE in 0.093s]
Sequential Consistency 2; Update values stage 4i Increment 1;
Updating stored KV pairs ... 
Issuing RELOADS ... 
[DONE in 0.128s]
TEST_PASSED
Sequential Consistency 2; Update values stage 4i Increment 2;
Updating stored KV pairs ... 
Issuing RELOADS ... 
[DONE in 0.167s]
TEST_PASSED
Sequential Consistency 2; Update values stage 4i Increment 3;
Updating stored KV pairs ... 
Issuing RELOADS ... 
[DONE in 0.173s]
TEST_PASSED
Sequential Consistency 2; Update values stage 4i Increment 4;
Updating stored KV pairs ... 
Issuing RELOADS ... 
[DONE in 0.133s]
TEST_PASSED
Sequential Consistency 2; Update values stage 4i Increment 5;
Updating stored KV pairs ... 
Issuing RELOADS ... 
[DONE in 0.134s]
TEST_PASSED
[ ... TEST Completed in 0.835 seconds ]

[Info] Preparing for the sequential consistency Test 2
[Info] Will resume 3.99.144.144:12381
[Info] Will resume 3.99.144.144:12372
[Info] Will resume 3.99.144.144:12367
[Info] Resuming 3 servers.
Server 3.99.144.144:12372 resumed
Server 3.99.144.144:12381 resumed
Server 3.99.144.144:12367 resumed
[ ... Completed in 4.109 seconds ]
Waiting 15 seconds after resume
[ ... TEST Completed in 19.109 seconds ]
[Info] Stage 4i Scenario: Alive Nodes: 31. Dead Nodes: 9
Confirming randomly selected keys were stored correctly.
Issuing GETs ... 
[DONE in 0.085s]
Sequential Consistency 2; Update values stage 4j Increment 1;
Updating stored KV pairs ... 
Issuing RELOADS ... 
[DONE in 0.205s]
TEST_PASSED
Sequential Consistency 2; Update values stage 4j Increment 2;
Updating stored KV pairs ... 
Issuing RELOADS ... 
[DONE in 0.214s]
TEST_PASSED
Sequential Consistency 2; Update values stage 4j Increment 3;
Updating stored KV pairs ... 
Issuing RELOADS ... 
[DONE in 0.168s]
TEST_PASSED
Sequential Consistency 2; Update values stage 4j Increment 4;
Updating stored KV pairs ... 
Issuing RELOADS ... 
[DONE in 0.135s]
TEST_PASSED
Sequential Consistency 2; Update values stage 4j Increment 5;
Updating stored KV pairs ... 
Issuing RELOADS ... 
[DONE in 0.14s]
TEST_PASSED
[ ... TEST Completed in 0.954 seconds ]

[Info] Preparing for the sequential consistency Test 2
[Info] Will resume 3.99.144.144:12384
[Info] Will resume 3.99.144.144:12353
[Info] Will resume 3.99.144.144:12368
[Info] Resuming 3 servers.
Server 3.99.144.144:12368 resumed
Server 3.99.144.144:12353 resumed
Server 3.99.144.144:12384 resumed
[ ... Completed in 4.48 seconds ]
Waiting 15 seconds after resume
[ ... TEST Completed in 19.48 seconds ]
[Info] Stage 4j Scenario: Alive Nodes: 34. Dead Nodes: 6


[ TEST Stage 4; Verify Sequential Consistency 2; (Clients = 128, Limit = 5000 GETS) ]
Attempting retrieval of keys... 
Issuing GETs ... 
[DONE in 0.456s]
[Info] Stage 4; Verify Sequential Consistency 2; GET Phase; Of 5000 stored keys: 5000 SUCCESS, 0 TIMEOUT, 0 KEY-NOT-FOUND, 0 ERROR-UID, 0 ERROR-INVALID-GET (value or version), 0 ERROR-OTHER.
[Info] Stage 4; Verify Sequential Consistency 2; GET Phase; Expected Key Loss: 0.0%. Actual Key Loss: 0.0%
[Info] Stage 4; Verify Sequential Consistency 2; GET Phase; Total Value Size (MiB): 2.44140625 MiB.
[Info] Stage 4; Verify Sequential Consistency 2; GET Phase; SUCCESS Response Count: 5000.0.
[Info] Stage 4; Verify Sequential Consistency 2; GET Phase; % SUCCESS: 100.0%.
[Info] Stage 4; Verify Sequential Consistency 2; GET Phase; % FAILED (Not SUCCESS): 0.0%.
[Info] Stage 4; Verify Sequential Consistency 2; GET Phase; % TIMEOUT: 0.0%.
[Info] Stage 4; Verify Sequential Consistency 2; GET Phase; % KEY-NOT-FOUND: 0.0%.
[Info] Stage 4; Verify Sequential Consistency 2; GET Phase; % ERROR-UID: 0.0%.
[Info] Stage 4; Verify Sequential Consistency 2; GET Phase; % ERROR-INVALID-GET (value or version): 0.0%.
[Info] Stage 4; Verify Sequential Consistency 2; GET Phase; % ERROR-OTHER: 0.0%.
[Info] Stage 4; Verify Sequential Consistency 2; GET Phase; NOT-STORED keys Count:                      0.0.
[Info] Stage 4; Verify Sequential Consistency 2; GET Phase; Retrieved Increments Count (tr):            19000.0
[Info] Stage 4; Verify Sequential Consistency 2; GET Phase; Timed Out Increments Count (tt):            0.0
[Info] Stage 4; Verify Sequential Consistency 2; GET Phase; Successful Increments Count (ts):           19000.0
[Info] Stage 4; Verify Sequential Consistency 2; GET Phase; SC Violations Count (tr<ts):                0.0
[Info] Stage 4; Verify Sequential Consistency 2; GET Phase; SC Compliant Count (ts<=tr<=ts+tt):         5000.0
[Info] Stage 4; Verify Sequential Consistency 2; GET Phase; SC Buggy-Implementation Count (tr>tt+ts):   0.0

[Info] Stage 4; Verify Sequential Consistency 2; GET Phase; % SC Compliant:                     100.0
[Info] Stage 4; Verify Sequential Consistency 2; GET Phase; % SC Violations:                    0.0
[Info] Stage 4; Verify Sequential Consistency 2; GET Phase; % SC Buggy-Implementation:          0.0
[Info] Stage 4; Verify Sequential Consistency 2; GET Phase; % Retrieved Increments (tr/ts):     100.0
[Info] Stage 4; Verify Sequential Consistency 2; GET Phase; % Timed Out Increments (tt/ts):     0.0
Checking that 34 nodes are alive
[Test isAlive] for 3.99.144.144:12383
[Test isAlive] for 3.99.144.144:12361
[Test isAlive] for 3.99.144.144:12352
[Test isAlive] for 3.99.144.144:12357
[Test isAlive] for 3.99.144.144:12375
[Test isAlive] for 3.99.144.144:12373
[Test isAlive] for 3.99.144.144:12385
[Test isAlive] for 3.99.144.144:12356
[Test isAlive] for 3.99.144.144:12374
[Test isAlive] for 3.99.144.144:12378
[Test isAlive] for 3.99.144.144:12350
[Test isAlive] for 3.99.144.144:12347
[Test isAlive] for 3.99.144.144:12348
[Test isAlive] for 3.99.144.144:12386
[Test isAlive] for 3.99.144.144:12349
[Test isAlive] for 3.99.144.144:12354
[Test isAlive] for 3.99.144.144:12351
[Test isAlive] for 3.99.144.144:12360
[Test isAlive] for 3.99.144.144:12371
[Test isAlive] for 3.99.144.144:12369
[Test isAlive] for 3.99.144.144:12358
[Test isAlive] for 3.99.144.144:12376
[Test isAlive] for 3.99.144.144:12355
[Test isAlive] for 3.99.144.144:12365
[Test isAlive] for 3.99.144.144:12359
[Test isAlive] for 3.99.144.144:12364
[Test isAlive] for 3.99.144.144:12366
[Test isAlive] for 3.99.144.144:12377
[Test isAlive] for 3.99.144.144:12381
[Test isAlive] for 3.99.144.144:12372
[Test isAlive] for 3.99.144.144:12367
[Test isAlive] for 3.99.144.144:12384
[Test isAlive] for 3.99.144.144:12353
[Test isAlive] for 3.99.144.144:12368
[ ... Completed in 0.009 seconds ]
TEST_PASSED
Waiting 10 seconds before resuming nodes
[Info] Resuming 6 servers.
Server 3.99.144.144:12382 resumed
Server 3.99.144.144:12362 resumed
Server 3.99.144.144:12380 resumed
Server 3.99.144.144:12379 resumed
Server 3.99.144.144:12363 resumed
Server 3.99.144.144:12370 resumed
[ ... Completed in 8.581 seconds ]

Results Summary:
Client AWS Instance Type : c5a.16xlarge
Client Local Hostname : ip-172-31-14-244
Client Local IP : ip-172-31-14-244/172.31.14.244
Concurrent Store/Load v2; Stage 0; : 2.0
Concurrent Store/Load v2; Stage 0; GET Phase; % ERROR-INVALID-GET (value or version) : 0.0
Concurrent Store/Load v2; Stage 0; GET Phase; % ERROR-OTHER : 0.0
Concurrent Store/Load v2; Stage 0; GET Phase; % ERROR-UID : 0.0
Concurrent Store/Load v2; Stage 0; GET Phase; % FAILED (Not SUCCESS) : 0.0
Concurrent Store/Load v2; Stage 0; GET Phase; % KEY-NOT-FOUND : 0.0
Concurrent Store/Load v2; Stage 0; GET Phase; % SUCCESS : 100.0
Concurrent Store/Load v2; Stage 0; GET Phase; % TIMEOUT : 0.0
Concurrent Store/Load v2; Stage 0; GET Phase; SUCCESS Response Count : 40000.0
Concurrent Store/Load v2; Stage 0; GET Phase; Total Value Size (MiB) : 19.120856285095215
Concurrent Store/Load v2; Stage 0; PUT Phase; % ERROR-INVALID-GET (value or version) : 0.0
Concurrent Store/Load v2; Stage 0; PUT Phase; % ERROR-OTHER : 0.0
Concurrent Store/Load v2; Stage 0; PUT Phase; % ERROR-UID : 0.0
Concurrent Store/Load v2; Stage 0; PUT Phase; % FAILED (Not SUCCESS) : 0.0
Concurrent Store/Load v2; Stage 0; PUT Phase; % KEY-NOT-FOUND : 0.0
Concurrent Store/Load v2; Stage 0; PUT Phase; % SUCCESS : 100.0
Concurrent Store/Load v2; Stage 0; PUT Phase; % TIMEOUT : 0.0
Concurrent Store/Load v2; Stage 0; PUT Phase; SUCCESS Response Count : 40000.0
Concurrent Store/Load v2; Stage 0; PUT Phase; Total Value Size (MiB) : 19.120856285095215
Concurrent Store/Load v2; Stage 2a; : 2.0
Concurrent Store/Load v2; Stage 2a; GET Phase; % ERROR-INVALID-GET (value or version) : 0.0
Concurrent Store/Load v2; Stage 2a; GET Phase; % ERROR-OTHER : 0.0
Concurrent Store/Load v2; Stage 2a; GET Phase; % ERROR-UID : 0.0
Concurrent Store/Load v2; Stage 2a; GET Phase; % FAILED (Not SUCCESS) : 0.0
Concurrent Store/Load v2; Stage 2a; GET Phase; % KEY-NOT-FOUND : 0.0
Concurrent Store/Load v2; Stage 2a; GET Phase; % SUCCESS : 100.0
Concurrent Store/Load v2; Stage 2a; GET Phase; % TIMEOUT : 0.0
Concurrent Store/Load v2; Stage 2a; GET Phase; SUCCESS Response Count : 20000.0
Concurrent Store/Load v2; Stage 2a; GET Phase; Total Value Size (MiB) : 9.514667510986328
Concurrent Store/Load v2; Stage 2a; PUT Phase; % ERROR-INVALID-GET (value or version) : 0.0
Concurrent Store/Load v2; Stage 2a; PUT Phase; % ERROR-OTHER : 0.0
Concurrent Store/Load v2; Stage 2a; PUT Phase; % ERROR-UID : 0.0
Concurrent Store/Load v2; Stage 2a; PUT Phase; % FAILED (Not SUCCESS) : 0.0
Concurrent Store/Load v2; Stage 2a; PUT Phase; % KEY-NOT-FOUND : 0.0
Concurrent Store/Load v2; Stage 2a; PUT Phase; % SUCCESS : 100.0
Concurrent Store/Load v2; Stage 2a; PUT Phase; % TIMEOUT : 0.0
Concurrent Store/Load v2; Stage 2a; PUT Phase; SUCCESS Response Count : 20000.0
Concurrent Store/Load v2; Stage 2a; PUT Phase; Total Value Size (MiB) : 9.514667510986328
Concurrent Store/Load v2; Stage 3; : 2.0
Concurrent Store/Load v2; Stage 3; PUT Phase; % ERROR-INVALID-GET (value or version) : 0.0
Concurrent Store/Load v2; Stage 3; PUT Phase; % ERROR-OTHER : 0.0
Concurrent Store/Load v2; Stage 3; PUT Phase; % ERROR-UID : 0.0
Concurrent Store/Load v2; Stage 3; PUT Phase; % FAILED (Not SUCCESS) : 0.0
Concurrent Store/Load v2; Stage 3; PUT Phase; % KEY-NOT-FOUND : 0.0
Concurrent Store/Load v2; Stage 3; PUT Phase; % SUCCESS : 100.0
Concurrent Store/Load v2; Stage 3; PUT Phase; % TIMEOUT : 0.0
Concurrent Store/Load v2; Stage 3; PUT Phase; SUCCESS Response Count : 40000.0
Concurrent Store/Load v2; Stage 3; PUT Phase; Total Value Size (MiB) : 19.119232177734375
Concurrent Store/Load v2; Stage 3a; : 2.0
Concurrent Store/Load v2; Stage 3a; PUT Phase; % ERROR-INVALID-GET (value or version) : 0.0
Concurrent Store/Load v2; Stage 3a; PUT Phase; % ERROR-OTHER : 0.0
Concurrent Store/Load v2; Stage 3a; PUT Phase; % ERROR-UID : 0.0
Concurrent Store/Load v2; Stage 3a; PUT Phase; % FAILED (Not SUCCESS) : 0.0
Concurrent Store/Load v2; Stage 3a; PUT Phase; % KEY-NOT-FOUND : 0.0
Concurrent Store/Load v2; Stage 3a; PUT Phase; % SUCCESS : 100.0
Concurrent Store/Load v2; Stage 3a; PUT Phase; % TIMEOUT : 0.0
Concurrent Store/Load v2; Stage 3a; PUT Phase; SUCCESS Response Count : 20000.0
Concurrent Store/Load v2; Stage 3a; PUT Phase; Total Value Size (MiB) : 9.56334114074707
Concurrent Store/Load v2; Stage 3b; : 2.0
Concurrent Store/Load v2; Stage 3b; PUT Phase; % ERROR-INVALID-GET (value or version) : 0.0
Concurrent Store/Load v2; Stage 3b; PUT Phase; % ERROR-OTHER : 0.0
Concurrent Store/Load v2; Stage 3b; PUT Phase; % ERROR-UID : 0.0
Concurrent Store/Load v2; Stage 3b; PUT Phase; % FAILED (Not SUCCESS) : 0.0
Concurrent Store/Load v2; Stage 3b; PUT Phase; % KEY-NOT-FOUND : 0.0
Concurrent Store/Load v2; Stage 3b; PUT Phase; % SUCCESS : 100.0
Concurrent Store/Load v2; Stage 3b; PUT Phase; % TIMEOUT : 0.0
Concurrent Store/Load v2; Stage 3b; PUT Phase; SUCCESS Response Count : 20000.0
Concurrent Store/Load v2; Stage 3b; PUT Phase; Total Value Size (MiB) : 9.573698043823242
Concurrent Store/Load v2; Stage 3c; : 2.0
Concurrent Store/Load v2; Stage 3c; PUT Phase; % ERROR-INVALID-GET (value or version) : 0.0
Concurrent Store/Load v2; Stage 3c; PUT Phase; % ERROR-OTHER : 0.0
Concurrent Store/Load v2; Stage 3c; PUT Phase; % ERROR-UID : 0.0
Concurrent Store/Load v2; Stage 3c; PUT Phase; % FAILED (Not SUCCESS) : 0.045
Concurrent Store/Load v2; Stage 3c; PUT Phase; % KEY-NOT-FOUND : 0.0
Concurrent Store/Load v2; Stage 3c; PUT Phase; % SUCCESS : 99.955
Concurrent Store/Load v2; Stage 3c; PUT Phase; % TIMEOUT : 0.045
Concurrent Store/Load v2; Stage 3c; PUT Phase; SUCCESS Response Count : 19991.0
Concurrent Store/Load v2; Stage 3c; PUT Phase; Total Value Size (MiB) : 9.488884925842285
Concurrent Store/Load v2; Stage 3d; : 2.0
Concurrent Store/Load v2; Stage 3d; PUT Phase; % ERROR-INVALID-GET (value or version) : 0.0
Concurrent Store/Load v2; Stage 3d; PUT Phase; % ERROR-OTHER : 0.0
Concurrent Store/Load v2; Stage 3d; PUT Phase; % ERROR-UID : 0.0
Concurrent Store/Load v2; Stage 3d; PUT Phase; % FAILED (Not SUCCESS) : 0.0
Concurrent Store/Load v2; Stage 3d; PUT Phase; % KEY-NOT-FOUND : 0.0
Concurrent Store/Load v2; Stage 3d; PUT Phase; % SUCCESS : 100.0
Concurrent Store/Load v2; Stage 3d; PUT Phase; % TIMEOUT : 0.0
Concurrent Store/Load v2; Stage 3d; PUT Phase; SUCCESS Response Count : 20000.0
Concurrent Store/Load v2; Stage 3d; PUT Phase; Total Value Size (MiB) : 9.57327651977539
Concurrent Store/Load v2; Stage 3e; : 2.0
Concurrent Store/Load v2; Stage 3e; PUT Phase; % ERROR-INVALID-GET (value or version) : 0.0
Concurrent Store/Load v2; Stage 3e; PUT Phase; % ERROR-OTHER : 0.0
Concurrent Store/Load v2; Stage 3e; PUT Phase; % ERROR-UID : 0.0
Concurrent Store/Load v2; Stage 3e; PUT Phase; % FAILED (Not SUCCESS) : 0.0
Concurrent Store/Load v2; Stage 3e; PUT Phase; % KEY-NOT-FOUND : 0.0
Concurrent Store/Load v2; Stage 3e; PUT Phase; % SUCCESS : 100.0
Concurrent Store/Load v2; Stage 3e; PUT Phase; % TIMEOUT : 0.0
Concurrent Store/Load v2; Stage 3e; PUT Phase; SUCCESS Response Count : 20000.0
Concurrent Store/Load v2; Stage 3e; PUT Phase; Total Value Size (MiB) : 9.589224815368652
Concurrent Store/Load v2; Stage 3f; : 2.0
Concurrent Store/Load v2; Stage 3f; PUT Phase; % ERROR-INVALID-GET (value or version) : 0.0
Concurrent Store/Load v2; Stage 3f; PUT Phase; % ERROR-OTHER : 0.0
Concurrent Store/Load v2; Stage 3f; PUT Phase; % ERROR-UID : 0.0
Concurrent Store/Load v2; Stage 3f; PUT Phase; % FAILED (Not SUCCESS) : 0.0
Concurrent Store/Load v2; Stage 3f; PUT Phase; % KEY-NOT-FOUND : 0.0
Concurrent Store/Load v2; Stage 3f; PUT Phase; % SUCCESS : 100.0
Concurrent Store/Load v2; Stage 3f; PUT Phase; % TIMEOUT : 0.0
Concurrent Store/Load v2; Stage 3f; PUT Phase; SUCCESS Response Count : 20000.0
Concurrent Store/Load v2; Stage 3f; PUT Phase; Total Value Size (MiB) : 9.530343055725098
Concurrent Store/Load v2; Stage 3g; : 2.0
Concurrent Store/Load v2; Stage 3g; PUT Phase; % ERROR-INVALID-GET (value or version) : 0.0
Concurrent Store/Load v2; Stage 3g; PUT Phase; % ERROR-OTHER : 0.0
Concurrent Store/Load v2; Stage 3g; PUT Phase; % ERROR-UID : 0.0
Concurrent Store/Load v2; Stage 3g; PUT Phase; % FAILED (Not SUCCESS) : 0.0
Concurrent Store/Load v2; Stage 3g; PUT Phase; % KEY-NOT-FOUND : 0.0
Concurrent Store/Load v2; Stage 3g; PUT Phase; % SUCCESS : 100.0
Concurrent Store/Load v2; Stage 3g; PUT Phase; % TIMEOUT : 0.0
Concurrent Store/Load v2; Stage 3g; PUT Phase; SUCCESS Response Count : 20000.0
Concurrent Store/Load v2; Stage 3g; PUT Phase; Total Value Size (MiB) : 9.61846923828125
Concurrent Store/Load v2; Stage 3h; : 2.0
Concurrent Store/Load v2; Stage 3h; PUT Phase; % ERROR-INVALID-GET (value or version) : 0.0
Concurrent Store/Load v2; Stage 3h; PUT Phase; % ERROR-OTHER : 0.0
Concurrent Store/Load v2; Stage 3h; PUT Phase; % ERROR-UID : 0.0
Concurrent Store/Load v2; Stage 3h; PUT Phase; % FAILED (Not SUCCESS) : 0.0
Concurrent Store/Load v2; Stage 3h; PUT Phase; % KEY-NOT-FOUND : 0.0
Concurrent Store/Load v2; Stage 3h; PUT Phase; % SUCCESS : 100.0
Concurrent Store/Load v2; Stage 3h; PUT Phase; % TIMEOUT : 0.0
Concurrent Store/Load v2; Stage 3h; PUT Phase; SUCCESS Response Count : 20000.0
Concurrent Store/Load v2; Stage 3h; PUT Phase; Total Value Size (MiB) : 9.634326934814453
Local At-Most-Once Policy : 2.0
Local At-Most-Once Policy Empty REMs : 2.0
Local At-Most-Once Policy Error Return Code : 2.0
Local At-Most-Once Policy Full Test : 2.0
Local Basic PUT GET  : 2.0
Local Client Capacity Test : 2.0
Local Double PUT with version : 2.0
Local Double Remove : 2.0
Local Invalid Command : 2.0
Local Invalid Key Test : 2.0
Local Invalid Value Test : 2.0
Local Remove Operation : 2.0
Local Shutdown Test : 2.0
Local Single node: 1 Client(s) % Failed : 0.0
Local Single node: 1 Client(s) % Success : 100.0
Local Single node: 1 Client(s) % Timeout : 0.0
Local Single node: 1 Client(s) Average Response Time (ms) : 1.0
Local Single node: 1 Client(s) Goodput (Requests per Second) : 10539.0
Local Single node: 1 Client(s) Maximum Response Time (ms) : 11.0
Local Single node: 1 Client(s) STDEV Response Time (ms) : 1.0
Local Single node: 1 Client(s) Successful Responses : 632336
Local Single node: 1 Client(s) Throughput (Requests per Second) : 10539.0
Local Single node: 1 Client(s) Timeout Responses : 0
Local Single node: 1 Client(s) Total Failed Responses : 0
Local Single node: 1 Client(s) Total Requests (including retries) : 632336
Local Single node: 1 Client(s) Total Requests (without retries) : 632336
Local Single node: 16 Client(s) % Failed : 0.0
Local Single node: 16 Client(s) % Success : 100.0
Local Single node: 16 Client(s) % Timeout : 0.0
Local Single node: 16 Client(s) Average Response Time (ms) : 1.0
Local Single node: 16 Client(s) Goodput (Requests per Second) : 25028.0
Local Single node: 16 Client(s) Maximum Response Time (ms) : 15.0
Local Single node: 16 Client(s) STDEV Response Time (ms) : 1.0
Local Single node: 16 Client(s) Successful Responses : 1501695
Local Single node: 16 Client(s) Throughput (Requests per Second) : 25028.0
Local Single node: 16 Client(s) Timeout Responses : 0
Local Single node: 16 Client(s) Total Failed Responses : 0
Local Single node: 16 Client(s) Total Requests (including retries) : 1501695
Local Single node: 16 Client(s) Total Requests (without retries) : 1501695
Local Single node: 64 Client(s) % Failed : 0.0
Local Single node: 64 Client(s) % Success : 100.0
Local Single node: 64 Client(s) % Timeout : 0.0
Local Single node: 64 Client(s) Average Response Time (ms) : 3.0
Local Single node: 64 Client(s) Goodput (Requests per Second) : 24914.0
Local Single node: 64 Client(s) Maximum Response Time (ms) : 14.0
Local Single node: 64 Client(s) STDEV Response Time (ms) : 1.0
Local Single node: 64 Client(s) Successful Responses : 1494969
Local Single node: 64 Client(s) Throughput (Requests per Second) : 24914.0
Local Single node: 64 Client(s) Timeout Responses : 0
Local Single node: 64 Client(s) Total Failed Responses : 0
Local Single node: 64 Client(s) Total Requests (including retries) : 1494969
Local Single node: 64 Client(s) Total Requests (without retries) : 1494969
Sequential Consistency; Stage 3a; : 2.0
Sequential Consistency; Stage 3a; GET Phase; % ERROR-INVALID-GET (value or version) : 0.0
Sequential Consistency; Stage 3a; GET Phase; % ERROR-OTHER : 0.0
Sequential Consistency; Stage 3a; GET Phase; % ERROR-UID : 0.0
Sequential Consistency; Stage 3a; GET Phase; % FAILED (Not SUCCESS) : 0.0
Sequential Consistency; Stage 3a; GET Phase; % KEY-NOT-FOUND : 0.0
Sequential Consistency; Stage 3a; GET Phase; % SUCCESS : 100.0
Sequential Consistency; Stage 3a; GET Phase; % TIMEOUT : 0.0
Sequential Consistency; Stage 3a; GET Phase; NOT-STORED keys Count : 0.0
Sequential Consistency; Stage 3a; GET Phase; SUCCESS Response Count : 40000.0
Sequential Consistency; Stage 3a; GET Phase; Total Value Size (MiB) : 19.132648468017578
Sequential Consistency; Stage 3b; : 2.0
Sequential Consistency; Stage 3b; GET Phase; % ERROR-INVALID-GET (value or version) : 0.0
Sequential Consistency; Stage 3b; GET Phase; % ERROR-OTHER : 0.0
Sequential Consistency; Stage 3b; GET Phase; % ERROR-UID : 0.0
Sequential Consistency; Stage 3b; GET Phase; % FAILED (Not SUCCESS) : 0.0
Sequential Consistency; Stage 3b; GET Phase; % KEY-NOT-FOUND : 0.0
Sequential Consistency; Stage 3b; GET Phase; % SUCCESS : 100.0
Sequential Consistency; Stage 3b; GET Phase; % TIMEOUT : 0.0
Sequential Consistency; Stage 3b; GET Phase; NOT-STORED keys Count : 0.0
Sequential Consistency; Stage 3b; GET Phase; SUCCESS Response Count : 40000.0
Sequential Consistency; Stage 3b; GET Phase; Total Value Size (MiB) : 19.07344150543213
Sequential Consistency; Stage 3c; : 2.0
Sequential Consistency; Stage 3c; GET Phase; % ERROR-INVALID-GET (value or version) : 0.0
Sequential Consistency; Stage 3c; GET Phase; % ERROR-OTHER : 0.0
Sequential Consistency; Stage 3c; GET Phase; % ERROR-UID : 0.0
Sequential Consistency; Stage 3c; GET Phase; % FAILED (Not SUCCESS) : 0.0
Sequential Consistency; Stage 3c; GET Phase; % KEY-NOT-FOUND : 0.0
Sequential Consistency; Stage 3c; GET Phase; % SUCCESS : 100.0
Sequential Consistency; Stage 3c; GET Phase; % TIMEOUT : 0.0
Sequential Consistency; Stage 3c; GET Phase; NOT-STORED keys Count : 8.0
Sequential Consistency; Stage 3c; GET Phase; SUCCESS Response Count : 39992.0
Sequential Consistency; Stage 3c; GET Phase; Total Value Size (MiB) : 19.029239654541016
Sequential Consistency; Stage 3d; : 2.0
Sequential Consistency; Stage 3d; GET Phase; % ERROR-INVALID-GET (value or version) : 0.0
Sequential Consistency; Stage 3d; GET Phase; % ERROR-OTHER : 0.0
Sequential Consistency; Stage 3d; GET Phase; % ERROR-UID : 0.0
Sequential Consistency; Stage 3d; GET Phase; % FAILED (Not SUCCESS) : 0.0
Sequential Consistency; Stage 3d; GET Phase; % KEY-NOT-FOUND : 0.0
Sequential Consistency; Stage 3d; GET Phase; % SUCCESS : 100.0
Sequential Consistency; Stage 3d; GET Phase; % TIMEOUT : 0.0
Sequential Consistency; Stage 3d; GET Phase; NOT-STORED keys Count : 5.0
Sequential Consistency; Stage 3d; GET Phase; SUCCESS Response Count : 39995.0
Sequential Consistency; Stage 3d; GET Phase; Total Value Size (MiB) : 19.092496871948242
Sequential Consistency; Stage 3e; : 2.0
Sequential Consistency; Stage 3e; GET Phase; % ERROR-INVALID-GET (value or version) : 0.05500137503437586
Sequential Consistency; Stage 3e; GET Phase; % ERROR-OTHER : 0.0
Sequential Consistency; Stage 3e; GET Phase; % ERROR-UID : 0.0
Sequential Consistency; Stage 3e; GET Phase; % FAILED (Not SUCCESS) : 0.05500137503437586
Sequential Consistency; Stage 3e; GET Phase; % KEY-NOT-FOUND : 0.0
Sequential Consistency; Stage 3e; GET Phase; % SUCCESS : 99.94499862496562
Sequential Consistency; Stage 3e; GET Phase; % TIMEOUT : 0.0
Sequential Consistency; Stage 3e; GET Phase; NOT-STORED keys Count : 1.0
Sequential Consistency; Stage 3e; GET Phase; SUCCESS Response Count : 39977.0
Sequential Consistency; Stage 3e; GET Phase; Total Value Size (MiB) : 19.152201652526855
Sequential Consistency; Stage 3f; : 2.0
Sequential Consistency; Stage 3f; GET Phase; % ERROR-INVALID-GET (value or version) : 0.03
Sequential Consistency; Stage 3f; GET Phase; % ERROR-OTHER : 0.0
Sequential Consistency; Stage 3f; GET Phase; % ERROR-UID : 0.0
Sequential Consistency; Stage 3f; GET Phase; % FAILED (Not SUCCESS) : 0.03
Sequential Consistency; Stage 3f; GET Phase; % KEY-NOT-FOUND : 0.0
Sequential Consistency; Stage 3f; GET Phase; % SUCCESS : 99.97
Sequential Consistency; Stage 3f; GET Phase; % TIMEOUT : 0.0
Sequential Consistency; Stage 3f; GET Phase; NOT-STORED keys Count : 0.0
Sequential Consistency; Stage 3f; GET Phase; SUCCESS Response Count : 39988.0
Sequential Consistency; Stage 3f; GET Phase; Total Value Size (MiB) : 19.11678695678711
Sequential Consistency; Stage 3g; : 2.0
Sequential Consistency; Stage 3g; GET Phase; % ERROR-INVALID-GET (value or version) : 0.0775
Sequential Consistency; Stage 3g; GET Phase; % ERROR-OTHER : 0.0
Sequential Consistency; Stage 3g; GET Phase; % ERROR-UID : 0.0
Sequential Consistency; Stage 3g; GET Phase; % FAILED (Not SUCCESS) : 0.0775
Sequential Consistency; Stage 3g; GET Phase; % KEY-NOT-FOUND : 0.0
Sequential Consistency; Stage 3g; GET Phase; % SUCCESS : 99.9225
Sequential Consistency; Stage 3g; GET Phase; % TIMEOUT : 0.0
Sequential Consistency; Stage 3g; GET Phase; NOT-STORED keys Count : 0.0
Sequential Consistency; Stage 3g; GET Phase; SUCCESS Response Count : 39969.0
Sequential Consistency; Stage 3g; GET Phase; Total Value Size (MiB) : 19.1300630569458
Sequential Consistency; Stage 3h; : 2.0
Sequential Consistency; Stage 3h; GET Phase; % ERROR-INVALID-GET (value or version) : 0.05
Sequential Consistency; Stage 3h; GET Phase; % ERROR-OTHER : 0.0
Sequential Consistency; Stage 3h; GET Phase; % ERROR-UID : 0.0
Sequential Consistency; Stage 3h; GET Phase; % FAILED (Not SUCCESS) : 0.05
Sequential Consistency; Stage 3h; GET Phase; % KEY-NOT-FOUND : 0.0
Sequential Consistency; Stage 3h; GET Phase; % SUCCESS : 99.95
Sequential Consistency; Stage 3h; GET Phase; % TIMEOUT : 0.0
Sequential Consistency; Stage 3h; GET Phase; NOT-STORED keys Count : 0.0
Sequential Consistency; Stage 3h; GET Phase; SUCCESS Response Count : 39980.0
Sequential Consistency; Stage 3h; GET Phase; Total Value Size (MiB) : 19.244829177856445
Sequential consistency test Suspend Node Count : 12
Server Count : 40
Server List : 3.99.144.144_12347_3.99.144.144_12348_3.99.144.144_12349_3.99.144.144_12350_3.99.144.144_12351_3.99.144.144_12352_3.99.144.144_12353_3.99.144.144_12354_3.99.144.144_12355_3.99.144.144_12356_3.99.144.144_12357_3.99.144.144_12358_3.99.144.144_12359_3.99.144.144_12360_3.99.144.144_12361_3.99.144.144_12362_3.99.144.144_12363_3.99.144.144_12364_3.99.144.144_12365_3.99.144.144_12366_3.99.144.144_12367_3.99.144.144_12368_3.99.144.144_12369_3.99.144.144_12370_3.99.144.144_12371_3.99.144.144_12372_3.99.144.144_12373_3.99.144.144_12374_3.99.144.144_12375_3.99.144.144_12376_3.99.144.144_12377_3.99.144.144_12378_3.99.144.144_12379_3.99.144.144_12380_3.99.144.144_12381_3.99.144.144_12382_3.99.144.144_12383_3.99.144.144_12384_3.99.144.144_12385_3.99.144.144_12386_
Single Client Capacity (MiB) for value size of 500 : 41.0
Single Front Node (At-Most-Once) Policy : 2.0
Single Front Node At-Most-Once Policy Empty REMs : 2.0
Single Front Node At-Most-Once Policy Error Return Code : 2.0
Single Front Node At-Most-Once Policy Full Test : 2.0
Single Front Node Basic PUT GET  : 2.0
Single Front Node Double PUT with version : 2.0
Single Front Node Double Remove : 2.0
Single Front Node Invalid Command : 2.0
Single Front Node Invalid Key Test : 2.0
Single Front Node Invalid Value Test : 2.0
Single Front Node Multi-Client Crash : 2.0
Single Front Node Remove Operation : 2.0
Stage 1 Random Front-End Basic : 2.0
Stage 1 Random Front-End: 1 Client(s) % Failed : 0.0
Stage 1 Random Front-End: 1 Client(s) % Success : 100.0
Stage 1 Random Front-End: 1 Client(s) % Timeout : 0.0
Stage 1 Random Front-End: 1 Client(s) Average Response Time (ms) : 17.0
Stage 1 Random Front-End: 1 Client(s) Goodput (Requests per Second) : 60.0
Stage 1 Random Front-End: 1 Client(s) Maximum Response Time (ms) : 63.0
Stage 1 Random Front-End: 1 Client(s) STDEV Response Time (ms) : 8.0
Stage 1 Random Front-End: 1 Client(s) Successful Responses : 3599
Stage 1 Random Front-End: 1 Client(s) Throughput (Requests per Second) : 60.0
Stage 1 Random Front-End: 1 Client(s) Timeout Responses : 0
Stage 1 Random Front-End: 1 Client(s) Total Failed Responses : 0
Stage 1 Random Front-End: 1 Client(s) Total Requests (including replies) : 3643
Stage 1 Random Front-End: 1 Client(s) Total Requests (without replies) : 3599
Stage 1 Random Front-End: 1024 Client(s) % Failed : 0.0
Stage 1 Random Front-End: 1024 Client(s) % Success : 99.90764359957608
Stage 1 Random Front-End: 1024 Client(s) % Timeout : 0.09235640042392006
Stage 1 Random Front-End: 1024 Client(s) Average Response Time (ms) : 25.0
Stage 1 Random Front-End: 1024 Client(s) CPU load (loadavg - last one minute) : 4.35
Stage 1 Random Front-End: 1024 Client(s) Goodput (Requests per Second) : 39306.0
Stage 1 Random Front-End: 1024 Client(s) Maximum Response Time (ms) : 585.0
Stage 1 Random Front-End: 1024 Client(s) STDEV Response Time (ms) : 42.0
Stage 1 Random Front-End: 1024 Client(s) Successful Responses : 2390694
Stage 1 Random Front-End: 1024 Client(s) Throughput (Requests per Second) : 39342.0
Stage 1 Random Front-End: 1024 Client(s) Timeout Responses : 2210
Stage 1 Random Front-End: 1024 Client(s) Total Failed Responses : 0
Stage 1 Random Front-End: 1024 Client(s) Total Requests (including replies) : 2759840
Stage 1 Random Front-End: 1024 Client(s) Total Requests (without replies) : 2392904
Stage 1 Random Front-End: 16 Client(s) % Failed : 0.0
Stage 1 Random Front-End: 16 Client(s) % Success : 100.0
Stage 1 Random Front-End: 16 Client(s) % Timeout : 0.0
Stage 1 Random Front-End: 16 Client(s) Average Response Time (ms) : 17.0
Stage 1 Random Front-End: 16 Client(s) Goodput (Requests per Second) : 998.0
Stage 1 Random Front-End: 16 Client(s) Maximum Response Time (ms) : 141.0
Stage 1 Random Front-End: 16 Client(s) STDEV Response Time (ms) : 8.0
Stage 1 Random Front-End: 16 Client(s) Successful Responses : 59876
Stage 1 Random Front-End: 16 Client(s) Throughput (Requests per Second) : 998.0
Stage 1 Random Front-End: 16 Client(s) Timeout Responses : 0
Stage 1 Random Front-End: 16 Client(s) Total Failed Responses : 0
Stage 1 Random Front-End: 16 Client(s) Total Requests (including replies) : 60620
Stage 1 Random Front-End: 16 Client(s) Total Requests (without replies) : 59876
Stage 1 Random Front-End: 2048 Client(s) % Failed : 0.0
Stage 1 Random Front-End: 2048 Client(s) % Success : 98.43879364790943
Stage 1 Random Front-End: 2048 Client(s) % Timeout : 1.5612063520905717
Stage 1 Random Front-End: 2048 Client(s) Average Response Time (ms) : 46.0
Stage 1 Random Front-End: 2048 Client(s) CPU load (loadavg - last one minute) : 8.44
Stage 1 Random Front-End: 2048 Client(s) Goodput (Requests per Second) : 33102.0
Stage 1 Random Front-End: 2048 Client(s) Maximum Response Time (ms) : 589.0
Stage 1 Random Front-End: 2048 Client(s) STDEV Response Time (ms) : 91.0
Stage 1 Random Front-End: 2048 Client(s) Successful Responses : 2015301
Stage 1 Random Front-End: 2048 Client(s) Throughput (Requests per Second) : 33627.0
Stage 1 Random Front-End: 2048 Client(s) Timeout Responses : 31962
Stage 1 Random Front-End: 2048 Client(s) Total Failed Responses : 0
Stage 1 Random Front-End: 2048 Client(s) Total Requests (including replies) : 2937533
Stage 1 Random Front-End: 2048 Client(s) Total Requests (without replies) : 2047263
Stage 1 Random Front-End: 256 Client(s) % Failed : 0.0
Stage 1 Random Front-End: 256 Client(s) % Success : 100.0
Stage 1 Random Front-End: 256 Client(s) % Timeout : 0.0
Stage 1 Random Front-End: 256 Client(s) Average Response Time (ms) : 16.0
Stage 1 Random Front-End: 256 Client(s) CPU load (loadavg - last one minute) : 0.43
Stage 1 Random Front-End: 256 Client(s) Goodput (Requests per Second) : 16062.0
Stage 1 Random Front-End: 256 Client(s) Maximum Response Time (ms) : 301.0
Stage 1 Random Front-End: 256 Client(s) STDEV Response Time (ms) : 8.0
Stage 1 Random Front-End: 256 Client(s) Successful Responses : 964620
Stage 1 Random Front-End: 256 Client(s) Throughput (Requests per Second) : 16062.0
Stage 1 Random Front-End: 256 Client(s) Timeout Responses : 0
Stage 1 Random Front-End: 256 Client(s) Total Failed Responses : 0
Stage 1 Random Front-End: 256 Client(s) Total Requests (including replies) : 976791
Stage 1 Random Front-End: 256 Client(s) Total Requests (without replies) : 964620
Stage 1 Suspend Node Count : 24
Stage 2 Distributed Shutdown Honoured : 2.0
Stage 2 Random Front-End Basic : 2.0
Stage 2 Random Front-End: 1 Client(s) % Failed : 0.0
Stage 2 Random Front-End: 1 Client(s) % Success : 100.0
Stage 2 Random Front-End: 1 Client(s) % Timeout : 0.0
Stage 2 Random Front-End: 1 Client(s) Average Response Time (ms) : 17.0
Stage 2 Random Front-End: 1 Client(s) Goodput (Requests per Second) : 62.0
Stage 2 Random Front-End: 1 Client(s) Maximum Response Time (ms) : 63.0
Stage 2 Random Front-End: 1 Client(s) STDEV Response Time (ms) : 7.0
Stage 2 Random Front-End: 1 Client(s) Successful Responses : 3707
Stage 2 Random Front-End: 1 Client(s) Throughput (Requests per Second) : 62.0
Stage 2 Random Front-End: 1 Client(s) Timeout Responses : 0
Stage 2 Random Front-End: 1 Client(s) Total Failed Responses : 0
Stage 2 Random Front-End: 1 Client(s) Total Requests (including replies) : 3743
Stage 2 Random Front-End: 1 Client(s) Total Requests (without replies) : 3707
Stage 2 Random Front-End: 16 Client(s) % Failed : 0.0
Stage 2 Random Front-End: 16 Client(s) % Success : 100.0
Stage 2 Random Front-End: 16 Client(s) % Timeout : 0.0
Stage 2 Random Front-End: 16 Client(s) Average Response Time (ms) : 16.0
Stage 2 Random Front-End: 16 Client(s) Goodput (Requests per Second) : 1007.0
Stage 2 Random Front-End: 16 Client(s) Maximum Response Time (ms) : 143.0
Stage 2 Random Front-End: 16 Client(s) STDEV Response Time (ms) : 8.0
Stage 2 Random Front-End: 16 Client(s) Successful Responses : 60395
Stage 2 Random Front-End: 16 Client(s) Throughput (Requests per Second) : 1007.0
Stage 2 Random Front-End: 16 Client(s) Timeout Responses : 0
Stage 2 Random Front-End: 16 Client(s) Total Failed Responses : 0
Stage 2 Random Front-End: 16 Client(s) Total Requests (including replies) : 61151
Stage 2 Random Front-End: 16 Client(s) Total Requests (without replies) : 60395
Stage 2 Random Front-End: 256 Client(s) % Failed : 0.0
Stage 2 Random Front-End: 256 Client(s) % Success : 100.0
Stage 2 Random Front-End: 256 Client(s) % Timeout : 0.0
Stage 2 Random Front-End: 256 Client(s) Average Response Time (ms) : 16.0
Stage 2 Random Front-End: 256 Client(s) CPU load (loadavg - last one minute) : 1.15
Stage 2 Random Front-End: 256 Client(s) Goodput (Requests per Second) : 16120.0
Stage 2 Random Front-End: 256 Client(s) Maximum Response Time (ms) : 301.0
Stage 2 Random Front-End: 256 Client(s) STDEV Response Time (ms) : 8.0
Stage 2 Random Front-End: 256 Client(s) Successful Responses : 968049
Stage 2 Random Front-End: 256 Client(s) Throughput (Requests per Second) : 16120.0
Stage 2 Random Front-End: 256 Client(s) Timeout Responses : 0
Stage 2 Random Front-End: 256 Client(s) Total Failed Responses : 0
Stage 2 Random Front-End: 256 Client(s) Total Requests (including replies) : 980072
Stage 2 Random Front-End: 256 Client(s) Total Requests (without replies) : 968049
Stage 2 Random Front-End: 512 Client(s) % Failed : 0.0
Stage 2 Random Front-End: 512 Client(s) % Success : 100.0
Stage 2 Random Front-End: 512 Client(s) % Timeout : 0.0
Stage 2 Random Front-End: 512 Client(s) Average Response Time (ms) : 17.0
Stage 2 Random Front-End: 512 Client(s) CPU load (loadavg - last one minute) : 1.91
Stage 2 Random Front-End: 512 Client(s) Goodput (Requests per Second) : 31799.0
Stage 2 Random Front-End: 512 Client(s) Maximum Response Time (ms) : 302.0
Stage 2 Random Front-End: 512 Client(s) STDEV Response Time (ms) : 8.0
Stage 2 Random Front-End: 512 Client(s) Successful Responses : 1909772
Stage 2 Random Front-End: 512 Client(s) Throughput (Requests per Second) : 31799.0
Stage 2 Random Front-End: 512 Client(s) Timeout Responses : 0
Stage 2 Random Front-End: 512 Client(s) Total Failed Responses : 0
Stage 2 Random Front-End: 512 Client(s) Total Requests (including replies) : 1935791
Stage 2 Random Front-End: 512 Client(s) Total Requests (without replies) : 1909772
Stage 2 Resumed Node Count : 18
Stage 4; Verify Sequential Consistency 2; : 2.0
Stage 4; Verify Sequential Consistency 2; GET Phase; % ERROR-INVALID-GET (value or version) : 0.0
Stage 4; Verify Sequential Consistency 2; GET Phase; % ERROR-OTHER : 0.0
Stage 4; Verify Sequential Consistency 2; GET Phase; % ERROR-UID : 0.0
Stage 4; Verify Sequential Consistency 2; GET Phase; % FAILED (Not SUCCESS) : 0.0
Stage 4; Verify Sequential Consistency 2; GET Phase; % KEY-NOT-FOUND : 0.0
Stage 4; Verify Sequential Consistency 2; GET Phase; % Retrieved Increments (tr/ts) : 100.0
Stage 4; Verify Sequential Consistency 2; GET Phase; % SC Buggy-Implementation : 0.0
Stage 4; Verify Sequential Consistency 2; GET Phase; % SC Compliant : 100.0
Stage 4; Verify Sequential Consistency 2; GET Phase; % SC Violations : 0.0
Stage 4; Verify Sequential Consistency 2; GET Phase; % SUCCESS : 100.0
Stage 4; Verify Sequential Consistency 2; GET Phase; % TIMEOUT : 0.0
Stage 4; Verify Sequential Consistency 2; GET Phase; % Timed Out Increments (tt/ts) : 0.0
Stage 4; Verify Sequential Consistency 2; GET Phase; NOT-STORED keys Count : 0.0
Stage 4; Verify Sequential Consistency 2; GET Phase; Retrieved Increments Count Sum (tr) : 19000.0
Stage 4; Verify Sequential Consistency 2; GET Phase; SC Buggy-Implementation Count (tr>tt+ts) : 0.0
Stage 4; Verify Sequential Consistency 2; GET Phase; SC Compliant Count (ts<=tr<=ts+tt) : 5000.0
Stage 4; Verify Sequential Consistency 2; GET Phase; SC Violations Count (tr<ts) : 0.0
Stage 4; Verify Sequential Consistency 2; GET Phase; SUCCESS Response Count : 5000.0
Stage 4; Verify Sequential Consistency 2; GET Phase; Successful Increments Count Sum (ts) : 19000.0
Stage 4; Verify Sequential Consistency 2; GET Phase; Timed-out Increments Count Sum (tt) : 0.0
Stage 4; Verify Sequential Consistency 2; GET Phase; Total Value Size (MiB) : 2.44140625
Suspend Verify v2; Stage 1; Key Recovery; : 2.0
Suspend Verify v2; Stage 1; Key Recovery; % ERROR-INVALID-GET (value or version) : 0.0
Suspend Verify v2; Stage 1; Key Recovery; % ERROR-OTHER : 0.0
Suspend Verify v2; Stage 1; Key Recovery; % ERROR-UID : 0.0
Suspend Verify v2; Stage 1; Key Recovery; % FAILED (Not SUCCESS) : 0.46
Suspend Verify v2; Stage 1; Key Recovery; % KEY-NOT-FOUND : 0.46
Suspend Verify v2; Stage 1; Key Recovery; % SUCCESS : 99.54
Suspend Verify v2; Stage 1; Key Recovery; % TIMEOUT : 0.0
Suspend Verify v2; Stage 1; Key Recovery; SUCCESS Response Count : 39816.0
Suspend Verify v2; Stage 1; Key Recovery; Total Value Size (MiB) : 19.03245449066162
Suspend Verify v2; Stage 2b; Key Recovery; : 2.0
Suspend Verify v2; Stage 2b; Key Recovery; % ERROR-INVALID-GET (value or version) : 0.0
Suspend Verify v2; Stage 2b; Key Recovery; % ERROR-OTHER : 0.0
Suspend Verify v2; Stage 2b; Key Recovery; % ERROR-UID : 0.0
Suspend Verify v2; Stage 2b; Key Recovery; % FAILED (Not SUCCESS) : 0.0
Suspend Verify v2; Stage 2b; Key Recovery; % KEY-NOT-FOUND : 0.0
Suspend Verify v2; Stage 2b; Key Recovery; % SUCCESS : 100.0
Suspend Verify v2; Stage 2b; Key Recovery; % TIMEOUT : 0.0
Suspend Verify v2; Stage 2b; Key Recovery; SUCCESS Response Count : 40000.0
Suspend Verify v2; Stage 2b; Key Recovery; Total Value Size (MiB) : 19.120856285095215
Suspend Verify v2; Stage 2c; Key Recovery; : 2.0
Suspend Verify v2; Stage 2c; Key Recovery; % ERROR-INVALID-GET (value or version) : 0.0
Suspend Verify v2; Stage 2c; Key Recovery; % ERROR-OTHER : 0.0
Suspend Verify v2; Stage 2c; Key Recovery; % ERROR-UID : 0.0
Suspend Verify v2; Stage 2c; Key Recovery; % FAILED (Not SUCCESS) : 0.015
Suspend Verify v2; Stage 2c; Key Recovery; % KEY-NOT-FOUND : 0.015
Suspend Verify v2; Stage 2c; Key Recovery; % SUCCESS : 99.985
Suspend Verify v2; Stage 2c; Key Recovery; % TIMEOUT : 0.0
Suspend Verify v2; Stage 2c; Key Recovery; SUCCESS Response Count : 19997.0
Suspend Verify v2; Stage 2c; Key Recovery; Total Value Size (MiB) : 9.51309871673584
Test Start Time : Fri Apr 14 04:16:09 UTC 2023
Result Verification Code : 10B70185CB311EE931119E954AAE2A15
[Info] Number of collected result fields: 468
[Info] Submission Successful!
[Info] Results server is up. Secret is valid.
[Info] Results successfully submitted.
[Info] Completed A11 Tests.

[TA-Note] Resuming any remaining dead nodes (so you do not have to redeploy your service)!
Waiting 10 seconds before resuming nodes
[Info] Resuming 1 servers.
Server 3.99.144.144:12364 resumed
[ ... Completed in 1.523 seconds ]

Testing completed in 2554.327s