Adding a new Alternator binding

[m-szymon]

I am wondering if it would make sense to copy DynamoDB binding, add it as a new Alternator binding and continue development there.

It seems that main YCSB repo is not very actively developed.
It has not merged PR with TTL support.
There is PR updating SDK (#3) - it would be nice to refresh it and merge in our repo.
I would like to switch from using legacy parameters (#11).
There are some other minor changes I would like to introduce (option for error on null read, secondary index support in scan).

With all these changes, there will be no line untouched (apart from general plugin skeleton).
So under the same name we will have actually different binding then in the main repo and chances that anything will be merged are very slim.

I think that it would be better to just start an "Alternator" binding.
What do you think? @fruch @ScyllaPiotr @radoslawcybulski

Migrated to Jira: QATOOLS-5

[radoslawcybulski]

My two cents:

• ycsb quality is decent, but not overwhelming
• it is written in java, not really fast - currently i'm using 9 (nine) loader nodes (c7i.2xlarge) - this is 8 * 9 = 72 virtual cpus and 16 * 9 = 144 GB ram - to push 3 node cluster (3 x i3.4xlarge - 16 * 3 = 48 vcpu) to 100%. I need more vcpus for loaders than for dbase...
• not actively developed
• do we care about other databases, than scylla / cassandra and dynamodb?

We pretty much already have our own "fork", so we might as well continue further and just ignore original repo.
Alternatively - write our own (i know every programmer wants to rewrite...) - there're rust and go bindings for hdr histogram, dynamodb and cassandra cql.

[ScyllaPiotr]

@radoslawcybulski maybe somewhat departed from the main issue of this discussion thread, but at the same time he raised a more important question - taking into account all the disadvantages of YCSB, do we wish to invest more time in its development? Let's maybe consider this subject first.

@fee-mendes based on your experience with testing performance of databases, what is your recommendation, would you suggest us to depart from YCSB towards some other environment?

[fee-mendes]

@fee-mendes based on your experience with testing performance of databases, what is your recommendation, would you suggest us to depart from YCSB towards some other environment?

Personally, I'd ditch YCSB. Most YCSB runs out there rely primarily on workload[a-b], which are very simplistic (we could come up with our own tooling probably within 100 lines of code if we aim to "just" reproduce these profiles).

On the contrary, workloade and workloadf are more interesting, as they stress things like Scans and RMW. Unfortunately, YCSB apparently doesn't support ConditionExpressions (something DynamoDB users rely on quite frequently), and thus have almost no insights on how Alternator compares.

On the performance front, @radoslawcybulski feedback isn't really surprising. It is quite painful to get YCSB to perform decently. Running a distributed load testing infrastructure to stress a distributed system is just insane :(

I understand this issue primarily aims to discuss how to stress-test Alternator, but another weird aspect on YCSB is how it handles ThrottledRequests. In particular, I remember previously observing it would often backoff quite aggressively, though never went ahead to understand what the actual policy was. Thus, as throttling hits, the observed client-side latencies become very irrealistic compared to CloudWatch metrics. Interestingly enough, YCSB shouldn't need the user to specify a rate when working with DynamoDB - as it theoretically could infer the available capacity and pre-warm throughput and prevent throttling. :(

All that said, I really like latte. It's easily extensible, performant (single decently-sized loader should easily get the 500K ops), and rune scripts allows developers to build profiles in the query language native to the database. It also differentiate between threads and concurrency, which is quite important and absent in tools like YCSB or cassandra-stress.

Hope it helps.

[m-szymon]

I understand this issue primarily aims to discuss how to stress-test Alternator,

My primary aim was to discuss how to approach the fact that DynamoDB binding in the original repo is not maintained and patches are not merged. Actually whole repo is not very active (although there are some updates). May it change in the future?

But the bigger question is if we should use it at all.
From what I see there is nobody to advocate for YCSB as a permanent solution.
Updating binding for DynamoDB/Alternator specific API is quite easy, but if its core is not trustworthy or to complex to use right, we should move from it as fast as possible

Unfortunately, YCSB apparently doesn't support ConditionExpressions

That is one of the things I would like to change.

[radoslawcybulski]

Oh, the issue with throttling in YCSB... on start (several seconds of the run itself) YCSB throttles violently and as a result if your run is short (for example 5 minutes), your P99 will be through the roof. For example https://argus.scylladb.com/tests/scylla-cluster-tests/8b939844-f9ab-48e4-b358-47cac6b10480/results - 7.88 ms.

[fruch]

Oh, the issue with throttling in YCSB... on start (several seconds of the run itself) YCSB throttles violently and as a result if your run is short (for example 5 minutes), your P99 will be through the roof. For example https://argus.scylladb.com/tests/scylla-cluster-tests/8b939844-f9ab-48e4-b358-47cac6b10480/results - 7.88 ms.

I don't know if any other stress tool is doing much better when it comes to a 5min run, 20-30min is usually the lowest we try to measure

[fruch]

@m-szymon

one point to keep in mind, as we have for CQL, is that one want to support the clients in all kind of languages
for CQL we focused first on Java, then on golang, and then on rust. (we never work stress tool for C++ or for python)

so one might still want to keep YCSB as a java reference for using alternator (even if something else is implemented in golang or rust)

[ScyllaPiotr]

As for Team Alternator, we'll be using YCSB only as long as we must. With a ZPP project that just launched, in 2026H2 we intend to switch to latte.

[dani-tweig]

Closing this issue as it was moved to Jira.
Please continue the thread in https://scylladb.atlassian.net/browse/QATOOLS-5