Skip to content

Conversation

@whitedr
Copy link

@whitedr whitedr commented Mar 17, 2015

I know more is needed for this (like testing...), but this is the beginnings of a kafka input/output plugin. This is using https://github.com/SOHU-Co/kafka-node/ and the HighLevelProducer/HighLevelConsumer interfaces to produce/consume messages to kafka topics. On the output side, I also created a kafka_river output that is the same as the kafka output but formats the messages published to kafka with an JSON envelope that can be indexed into ElasticSearch with the https://github.com/clippPR/elasticsearch-river-kafka ES river. An example of this JSON envelope structure can be seen here: https://github.com/clippPR/elasticsearch-river-kafka/blob/master/src/main/java/org/elasticsearch/river/kafka/JsonMessageHandler.java#L35

One thing I wasn't completely sure of was whether to have the kafka_river as a separate output type or have it be a filter. I ended up following the example of the output_elasticsearch_zeromq.

@bpaquet
Copy link
Owner

bpaquet commented Sep 22, 2016

Can you add documentation for this great plugin ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants