enqueue --fifo interacts poorly with callers using exponential backoff

[ryanwitt]

User encountered a problem where the same job was being sent to a fifo queue multiple times due to exponential backoff code wrapping qdone.

In this case, qdone generates a MessageDeduplicationID for each message for each retry, allowing multiple instances of the same message through.

To fix this, we could bring exponential backoff and retry into qdone, pending discovery of whether the errors we see are retryable.

Alternately, we could introduce an option that pre-seeds qdone with a random string that it uses to create a consistent MessageDeduplicationID for each message enqueued. This approach should work for enqueue-batch too.

[ryanwitt]

Upon reviewing the SQS docs and AWS.Request retry docs, we should be retrying internally for the enqueue action. Not clear if enqueue-batch will do this though.

[ryanwitt]

Looks like user moving to enqueue from enqueue-batch in critical sections could solve this, bringing the error handling into the AWS sdk.