Dedupe on Couchbase for real time streaming json (flink)

[ashubitm]

Hi ,I am trying to dedupe real time streaming json with destination as couchbase .I am trying to do this call for dedupe from flink but not able to perform .
Can you please help with config file for couchbase and how to call our larsga/Dedupe from flink ?

[uderline]

Hi,
There are no couchbase datasource - you can eventually make your own based on other datasources.
I am not familiar with couchbase and flink. Can you make a request with couchbase, then make a deduplication with the returned json ?
To call the deduplication function:

• Make the config (properties, match listener, data source ...)
• Start the processor
• Make the records
• Call the dedup function

[ashubitm]

Thanks ,
What i am trying to do is to process a stream of jsons ( source) against couchbase DB (destination).
Calling the json from couchbase may not be a gr8 idea from performance point of view and how many records to pull will be another thing .
If MongoDB can have a direct destination why not couchbase ? If you can help around this will be gr8 .
My purpose is if there are duplicates in the stream that is already present in destination i should not be saving those .

[uderline]

Sorry, I misread your first post. No problem for the json streaming if you're using flink or any other tool designed for streaming.
MongoDB has a data source connection but not a destination connection. The matching records are always saved in the match listener whereas the ones with no matches are not saved. You will need to make the destination connection.

[kyriehan89]

hi @ashubitm ,

I also have the same case, need to do dedup from couchbase,have you found the solution?

Thanks