Idempotent message handling

[sgryt]

Some of the Qaiain clients will only be occasionally connected. They will have to store pending messages in some durable store (filesystem for iOS, maybe IndexedDB for appCache'd laptop versions in the near future).
This brings about the possibility of having duplicate email-requests send to Qaiain, if the updating of the clients durable store fails, after successfully delivering an email request to Qaiain.
It would be unfortunate to have duplicates, as they will potentially look like spam to recipients, and also just be annoying to the recipients if arriving more than once in their inbox.
One way of avoiding this is to have Qaiain support idempotent messaging - where part of the contract could be a unique identifier, generated by the client, that Qaiain could use to check if the message has already been delivered to the email service backend (Mailgun, or the like).

I haven't spend much time researching alternative ideas, so any feedback is most welcome. Perhaps the email backend service used supports this in itself, or we may need Qaiain to have a persistent storage of already-delivered message identifiers?

[ploeh]

Let's add a unique ID to the message format, but IIRC, not all languages/platforms support GUIDs/UUIDs (Javascript?). There are at least two ways of dealing with this:

• Make the id element or attribute optional. If it has a value, Qaiain will use the value, but otherwise, it'll generate a GUID when it picks up the message. This follows Postel's law in the sense that clients that can't generate unique IDs can still participate, but the tradeoff is that these clients don't get the immutability 'guarantee'.
• Make the id element or attribute required, but a string. Clients can supply any string, but will have to guarantee that the string is unique (or risk losing the message because of replay detection).

However, this, in the end, doesn't guarantee idempotency. It may reduce the risk of replays, but can't totally eliminate them.

In order to perform replay detection, Qaiain would need to maintain a persistent store of all handled IDs. However, since Azure queues don't support distributed transactions, the safest update order would be:

1. Check the replay detection store. If a replay is detected, go to step 4.
2. Send the email.
3. Add the ID to the replay detection store.
4. Delete the message from the queue.

Steps 3 and 4 can fail, which will cause an undetected replay.

While we could consider switching the order of 2 and 3, this would cause messages to be dropped, so IMO isn't, in this case, a good trade-off.

[sgryt]

I vote for the an optional id element, but in string format. There are relatively easy ways of generating pseudo-random strings in JS, to mimic GUID's, but I'm not sure about other, more esoteric, client stacks.

About the update sequence - we could bet on the fact that the joint probability of failure on both device update and azure update, for the same message, is small (will be on the order of the product of the probabilities of each storage update failing independently). Does not seem unreasonable IMO, and losing messages entirely is probably worse than sending a few duplicates.

[ploeh]

Indeed, I, too, think that the probability of failing to write to the replay detection store, after successfully sending an email, is low.

[sgryt]

So, is this the point where Qaiain gets to be a consumer of the Azure AtomEventStore?

[ploeh]

It might, but that's not going to work until AtomEventStore is also open source...

The replay detection store has, AFAICT, extremely simple requirements: essentially, it just has to hold a list of already observed IDs. Basically, this can be done in a single BLOB.

However, neither that nor AtomEventStore scales in this scenario, because that list is only going to grow and grow. A better option would be to simply write a BLOB with the ID as the name (and no content). This would enable Qaiain to attempt to get the BLOB with a particular ID. If the BLOB exists, it's a replay.