Add note about fan-out for increased consumer throughput

[harlow]

This blog post is great, I've referenced it several times over the years. In fact we chose to use Kafka over Kinesis primarily around the 5 consumer constraint. However, it looks like that limitation has been addressed and I wanted to make sure that new readers have updated information.

In Aug 2018 AWS introduce fan-out to the Kinesis Data Streams which gives each stream reader dedicated resources.

https://aws.amazon.com/blogs/aws/kds-enhanced-fanout/

At first glance I didn't see any special CSS styling for "UPDATES" in the repo. Let me know if you'd prefer some special formatting via blockquote or whatever.

[brandur]

Thanks Harlow! Glad this article has been helpful, and I'm very happy to see some contributions as I'm not actively using Kinesis anymore and am not aware of the new developments.

Just reading through the linked blog post, it kind of looks like they've ramped up the amount of data throughput allowed, but as far as I can tell, it doesn't look like they ever addressed the 5 reads per second maximum. It's conveniently not mentioned in the blog post, and is still listed on the Kinesis limits page.

I'd certainly be very happy to be wrong as I'd love to see a real Kafka competitor that's not as complicated to run (i.e., in case this is just a problem of outdated documentation), but I'd say that if we mention this update, it should be inline in a paragraph or in a footnote rather than at the top of the section. Putting it at the top implies that the problem has been addressed, and to my mind, it hasn't, and continues to be the system's most problematic drawback even four years later in 2019.

[harlow]

Just reading through the linked blog post, it kind of looks like they've ramped up the amount of data throughput allowed, but as far as I can tell, it doesn't look like they ever addressed the 5 reads per second maximum.

@brandur yep agree the docs aren't clear. I'll reach out to our AWS rep and get some more details.

[brandur]

I'll reach out to our AWS rep and get some more details.

Awesome, thank you! Let me know if you hear back. I'll leave this open until then.

[harlow]

From AWS Rep:

No with EFO the limit does not exist. EFO changes the reads to a push model instead of a pull model so as new data arrives into the stream, it gets delivered to the consumer.

https://docs.aws.amazon.com/streams/latest/dev/introduction-to-enhanced-consumers.html

This throughput is dedicated, which means that consumers that use enhanced fan-out don't have to contend with other consumers that are receiving data from the stream.

OK so I think this must be in conjunction with the HTTP/2 connection enhancement to support streaming over HTTP

[brandur]

Thanks again for following up!

Could you change the comment to look/read something like this (I have one similar case elsewhere so may as well try to match the formatting):

Update (2019/01/10) — Amazon has introduced enhanced fan-out capability which gives each consumer a dedicated 2MB/second of read throughput per shard. Enhanced fan-out uses a streaming protocol, making it unnecessary for consumers to poll continually and use up the read limit.