Enforce a size limit on journaled events

[samandmoore]

Background
In some cases, we have messages being pulled off streams, augmented with additional data, and then plopped back onto another stream. If the message we initially put on the stream is too large (at or too close to the max message size) then we run the risk of breaking stream consumers ability to minimally augment our events and add them to their own streams.

Proposal
We should introduce a size limit for the events to ensure that we don't come within some reasonable threshold of the size limit of a validate message for AWS kinesis streams.

Questions

1. what should the limit be? perhaps 90% of the max message size?
2. how should we enforce this limit?

• reject events that are too large? probably bad dev ex since the size of the message is a function of external factors
• trim fields that are too large?

[jmileham]

I could buy something even more extreme/opinionatedly small max message size, given the intended use of this tool - the max kinesis message size is 1mb, which is way larger than the kinds of payloads this would be expected to use by default. I might even go down to something closer to 4kb if data analysis indicates that's appropriate?

[smudge]

Trimming fields could maybe work for specific use cases, but I don't think it's a good idea for the core gem (which I see as the unopinionated delivery mechanism) to mutate underlying data. Better to leave that to the next layer of abstraction. So as long as it's not a regular occurrence, better to halt delivery and leave an erroring background job to clean up.

(This is where tuning the size cutoff is important. I can pull a distribution of our production payload sizes that could inform a default value for the gem.)

[jmileham]

+1 for reject, especially in the DJ background where you can maybe override the max by a bit and unwedge a job.

[samandmoore]

yea, i'm in favor of accepting the job but rejecting the event at stream enqueue time so that exceptional events can be handled easily and we don't break the end user experience.