This utility is used to generate load on Kafka. In the simplest form it can generate x messages per second where x is configurable. This utility has currently been enhanced to generate a near real scenario which follows the life cycle of an event and its child entities ( markets and selections).
The life cycle is as follows.
- Event/Market/Selection Creation
- Event/Market/Selection Updates
- Events kick off (isEventStarted = true)
- Event/Market/Selection Updates
- Selections/Markets/Events Resulted and Settled
Use Maven build to build the jar. Once built the following runnable jar will be created
kafka-load-generator-0.0.1-SNAPSHOT-jar-with-dependencies.jar
For setting up a runtime create a folder and copy the jar in it. As an example let us assume the root folder is /opt/kafkaloadgenerator. The following folder structure will be needed
/opt/kafkaloadgenerator
/opt/kafkaloadgenerator/config
/opt/kafkaloadgenerator/data
/opt/kafkaloadgenerator will contain the jar.
/opt/kafkaloadgenerator/config contains the main configuration file and the Logger config file
kafkaclient.properties
Logger.properties
/opt/kafkaloadgenerator/data contains all the template json files ( for event/market/selection creates, event inplays and other updates)
PushTest_EventCreate.json
PushTest_EventInplay.json
PushTest_EventResult.json
PushTest_EventUpdate.json
PushTest_MarketCreate.json
PushTest_MarketResult.json
PushTest_MarketUpdate.json
PushTest_SelectionCreate.json
PushTest_SelectionResult.json
PushTest_SelectionUpdate.json
PushTest_SelectionUpdatePrice.json
All the template json files have certain key fields that will be replaced by the test utility at runtime
${event} will be replaced by eventKey
${market} will be replaced by the marketKey
${selection} will be replaced by the selectionKey
${toplevel} will be replaced by catergory class and type details
${messageid} will be replaced by a unique number per request
${recordModifiedTime} will be replaced by the time the message is generated and sent to Kafka
-9999 will be replaced by the displayOrder
-1111 will be replaced by a random price numerator
-2222 will be replaced by a random price denominator
3333.33 will be replaced by a random decimal price
Details of each parameter as given below
[client.TPS] - This is the message rate per second that you want to generate.
[client.maxtime] - This is duration of the test in seconds.
[client.producer.payload.dir] - This is the folder where the template json data is.
[client.producer.payload.fileprefix] - All templace files must have this prefix. PushTest_ as an example
[client.producer.payload.toplevels.count] - This is the total number of sport ot parent combination that will be used. In the below examples 6 top level cominations were used.
[client.producer.payload.toplevels.0] - The value has a format categoryid-classid-typeid:Percentage OF events for this sport as shown below in following examples
[client.producer.payload.toplevels.1]=34-266-10449:20 - Here Tennis ( 34 ) , All Tennis (266) and Wimbledon ( 10449) are the hierarchy and 20% of the events will be traded for this combination.
[client.producer.payload.toplevels.2]=6-13-35:40
[client.producer.payload.toplevels.3]=10-58-162:40
[client.producer.payload.toplevels.4]=31-259-2703:40
[client.producer.payload.toplevels.5]=18-195-1260:40
[client.producer.payload.eventid.range] - This is the eventid range. Keey the range to 250 and use 3 digit numbers. (e.g. 200-450)
[client.producer.payload.marketid.prefix.range] - MarketId prefix range . the actual market id will be this range number + the event id. (e.g 70-72 will generate 3 markets under event event. with the IDs 70200,71200 and 72200 for the event id 200)
[client.producer.payload.selectionid.prefix.range] - Selection ID Prefix range. For the POV keep this to 3 (e.g 80-82 wil generate 8070200,8170200,8270200 for market 70200 and for event 200)
[client.producer.inplaytime] - This is the time in seconds after which events are made inplay (Recommended value 5 minutes)
[client.producer.resulttime] - This is the time in seconds after which results will be sent. This should be a little before the total time of test (Recommended value is 25 minutes for a 30 minute test)
#Kafka
[kafka.bootstrap.servers] - This is the Kafka Bootstrap Server URL
[kafka.topic] - This is the delta topic on which our Kafka Streams will be listening on.
[kafka.listener.groupName] This utility will also have a consumer which shows you the latency of consumption. This property is the consumer group name. Recommended value netty.loadtest.consumer
Please leave all other properties as is
From /opt/kafkaloadgenerator execute the following command
java -Dconfig=./config -jar kafka-load-generator-0.0.1-SNAPSHOT-jar-with-dependencies.jar ./config/kafkaclient.properties
If the utility is running fine you will see the following in the logs
Stats: TPS(avg):1044, TPS(last):1012, TIME: 1341, sent:1400, ok:1400, error:0(c/t/d/o: 0/0/1598/0), avg_latency:7
Stats: TPS(avg):1024, TPS(last):1000, TIME: 2344, sent:2400, ok:2400, error:0(c/t/d/o: 0/0/1598/0), avg_latency:6
Stats: TPS(avg):1016, TPS(last):1000, TIME: 3348, sent:3400, ok:3400, error:0(c/t/d/o: 0/0/1598/0), avg_latency:5
Stats: TPS(avg):1011, TPS(last):1000, TIME: 4352, sent:4400, ok:4400, error:0(c/t/d/o: 0/0/1598/0), avg_latency:5
Stats: TPS(avg):1008, TPS(last):1000, TIME: 5356, sent:5400, ok:5400, error:0(c/t/d/o: 0/0/1598/0), avg_latency:5
Stats: TPS(avg):1006, TPS(last):1000, TIME: 6359, sent:6400, ok:6400, error:0(c/t/d/o: 0/0/1598/0), avg_latency:5
which shows the tps and the latency of the internal consumer.