Using several websockets with high incoming data rate and queues

[yimyom]

Hi,

I'm looking for an efficient pattern to do the following:

• I need to connect to several websocket which send a lot of data at a high rate (it's common to see 20-30 messages/second)
• I want to store the incoming data in queues, one for each websocket.
• then I have a main process which reads the queues and process them asynchronously.

Now here is my problem: in an ideal world, I would like to have one thread for each websocket and its queue and the main thread will read from the queue at its own rate. Now, I don't want to have slow IPC like in most of the R packages (futures, ipc, promises, etc...) which rely on local files. I need very fast communications (like shared memory for example).

How would you approach this problem?

(I know it's not an "issue" with websocket per se but it's still related to it, because I'd like to have not only the internal websocket in a thread but also the $onMessage method too)