Add SQS / SNS Support

[goliel]

Websockets are not very good when you want to process a lot of messages in a large pipeline due to the lack of consumers that can process the messages.

Would like to see some kind of SQS support, this way multiple consumers can subscribe and process the messages as scale out rather than scale up.

[d-Rickyy-b]

This project was initially thought of as a 1:1 replacement for the original certstream server. I thought about implementing something like Kafka support myself, additionally to websockets.

I haven't worked with SNS / SQS yet so I can't say if that's something I'll consider adding. But I'll definitely have a look into it.

Cheers

[alberts-s]

When I first found this repository I thought at first that having support for a queue (e.g SQS) would be useful.

However after giving this more though - maybe it would be better to prioritize adding support for storing the messages in an S3-compatible file store? Combining this with the out of the box feature to receive S3 events in SQS (guide here) it would also accomplish the goal of supporting SQS. GCP supports similar mechanism out of the box too.

Moreover supporting an S3-compatible store would allow also to easily query results using AWS Athena/Glue and GCP BigQuery.

[meinder-a]

Well, I thought that having this feature could make it easier to implement other distributed message-queuing services like Kafka, which is also widely used. I'm not entirely sure though. I like the concept of uploading messages to S3, but I’d prefer a small script that reads the messages using Kafka and uploads them to S3 afterwards.

[d-Rickyy-b]

I am currently working on kafka and nsq support as I think kafka is the most widely used and nsq is a neat, lightweight stream processing tool.

The implementation will be interface based, so it will be easy to integrate further services later on. Since I still don't use any AWS or other cloud services, it will be hard for me to test the integration. Adding more dependencies for features most people probably won't use anyway sounds kinda unnecessary.

Hence I am not quite sure how to deal with other kinds of stream processing tools yet. Maybe I push those to separate branches to self-build? Not sure yet.