Merge pull request #232 from alphagov/ldeb-use-ses-for-submission-emails

Update sequence diagrams for sending submission emails

Author: lfdebrux

diagrams/sequence-diagrams/README.md

@@ -38,15 +38,15 @@ graph TD
     filler --- runner --- admin
 
     notify(GOV.UK Notify)
+    ses(Amazon Simple Email Service)
 
     pay(GOV.UK Pay)
     filler -.- pay -.- runner
 
     filler -.- notify
-    notify --- runner
-    processor --- notify
-
-
+    notify -.- runner
+    ses --- runner
+    processor --- ses
 ```
 
 ## Links to Sequence Diagrams

diagrams/sequence-diagrams/file-upload.md

@@ -138,157 +138,3 @@ runner->>browser: HTTP 302 redirect response
 
 note over user,runner: User completes the rest of the questions
 ```
-
-### User submits their form
-```mermaid
-
----
-title: User submits their form
----
-
-sequenceDiagram
-
-autonumber
-
-actor user as User
-
-participant browser as Browser
-
-participant runner as Forms Runner
-participant runner-db as Forms Runner database
-participant solidqueue-db as Solid Queue database
-
-browser->>runner: GET check your answers
-runner->>browser: HTTP 200 response
-browser->>user: display check your answers page
-
-user->>browser: select "Submit"
-browser->>runner: POST submit form
-runner->>runner-db: save Submission
-runner->>solidqueue-db: enqueue send submission job
-
-runner->>browser: HTTP 302 response
-browser->>runner: GET confirmation page
-runner->>browser: HTTP 200 response
-browser->>user: Display confirmation page
-
-```
-
-### Sending the submission email asynchronously
-
-```mermaid
-
----
-title: Sending the submission email asynchronously 
----
-
-sequenceDiagram
-
-autonumber
-
-participant worker as Forms runner worker
-participant solidqueue-db as Solid Queue database
-participant runner-db as Forms Runner database
-participant s3 as Amazon S3
-participant ses as Amazon SES
-participant sns as Amazon SNS
-participant inbox as Email inbox
-participant sentry as Sentry
-
-actor processor
-
-worker->>solidqueue-db: dequeue send submission job
-worker->>runner-db: get Submission
-worker->>s3: get file(s)
-worker->>ses: send email
-
-break error 
-  alt AWS SDK error and max retries not reached
-    worker->>solidqueue-db: schedule retry
-  else
-    worker->>sentry: send error
-  end
-end
-
-ses->>worker: return message_id
-worker->>runner-db: set mail_message_id on Submission
-worker->>runner-db: update mail_status of Submission to "pending"
-
-ses-)inbox: send email
-note over ses,inbox: happens some time later
-
-alt email sent successfully
-  ses->>sns: send delivery notification
-  processor->>inbox: get form from inbox
-  processor->>processor: process form
-else email bounces
-  ses->>sns: send bounce notification
-  note over ses,sns: We have an SQS queue subscribed to the<br/> SNS topic and a recurring task to poll<br/>the SQS queue.
-end
-
-worker->>worker: Recurring task deletes Submissions<br/>that haven't bounced after 7 days
-```
-
-### Handling email bounces/complaints
-```mermaid
-
----
-title: Handling email bounces/complaints
----
-
-sequenceDiagram
-
-autonumber
-
-participant worker as Solid Queue worker
-participant solidqueue-db as Solid Queue database
-participant runner-db as Forms Runner database
-participant sqs as Amazon SQS
-participant sentry as Sentry
-
-actor support as Forms team tech support
-
-worker->>solidqueue-db: enqueue recurring receive bounces job
-worker->>solidqueue-db: dequeue receive bounces job
-worker->>sqs: get messages from bounces and complaints queue
-alt there is a bounce SQS message
-  worker->>runner-db: get Submission by the message_id in the SQS message
-  worker->>runner-db: update mail_status of Submission to "bounced"
-  worker->>worker: Log with the submission details
-  worker->>sentry: send error event
-  sentry->>support: Alert via Slack
-  support->>support: Identify why the email bounced
-  support->>support: Run rake task to retry submission
-else there is a complaint SQS message
-  worker->>runner-db: get Submission by the message_id in the SQS message
-  worker->>worker: Log with the submission details
-end
-
-```
-
-### Handling successful deliveries
-
-```mermaid
-
----
-title: Handling successful deliveries
----
-
-sequenceDiagram
-
-autonumber
-
-participant worker as Solid Queue worker
-participant solidqueue-db as Solid Queue database
-participant runner-db as Forms Runner database
-participant sqs as Amazon SQS
-
-actor support as Forms team tech support
-
-worker->>solidqueue-db: enqueue recurring receive deliveries job
-worker->>solidqueue-db: dequeue receive deliveries job
-worker->>sqs: get messages from deliveries queue
-worker->>runner-db: get Submission by the message_id in the SQS message
-worker->>runner-db: update mail_status of Submission to "delivered"
-note over worker,runner-db: we don't currently use the "delivered" status for anything other than for information
-```

diagrams/sequence-diagrams/filling-in-a-form.md

@@ -7,7 +7,6 @@ sequenceDiagram
   participant govuk as GOV.UK website
   participant runner as forms-runner
   participant admin as forms-admin
-  participant notify as GOV.UK Notify
   participant inbox as shared email inbox
   actor processor as Form processor
 
@@ -41,13 +40,13 @@ sequenceDiagram
   user->>browser: Click "Submit"
   browser->>runner: POST /form/{form id}/{form slug}/submit-answers
   runner->>admin: GET /api/v2/forms/{form id}/live
-  runner->>notify: client.send_email()
-  runner->>runner: Delete user session
+  runner->>runner: save Submission
+  runner->>runner: enqueue send submission job
   runner-->>browser: 302
   browser->>runner: GET /form/{form id}/{form slug}/submitted
   runner->>admin: GET /api/v2/forms/{form id}/live
   browser-->>user: Show confirmation page
-  notify->>inbox: Send email
+  runner->>inbox: Send email
   inbox->>processor: Read email
 ```
 
@@ -61,7 +60,6 @@ sequenceDiagram
   participant browser as Web Browser
   participant runner as forms-runner
   participant admin as forms-admin
-  participant notify as GOV.UK Notify
 
   user->>browser: Click link to form
   browser->>runner: GET /form/{form id}/{form slug}
@@ -75,8 +73,6 @@ sequenceDiagram
   end
 ```
 
-
-
 ## Brief summary
 
 The users answers are store in Redis in the following structure as they progress through the form.

diagrams/sequence-diagrams/sending-submission-emails.md

@@ -0,0 +1,158 @@
+# Sending submission emails asynchronously
+
+forms-runner uses SolidQueue and Amazon SES/SNS to send submissions asynchronously.
+
+## User submits their form
+
+```mermaid
+
+---
+title: User submits their form
+---
+
+sequenceDiagram
+
+autonumber
+
+actor user as User
+
+participant browser as Browser
+
+participant runner as Forms Runner
+participant runner-db as Forms Runner database
+participant solidqueue-db as Solid Queue database
+
+browser->>runner: GET check your answers
+runner->>browser: HTTP 200 response
+browser->>user: display check your answers page
+
+user->>browser: select "Submit"
+browser->>runner: POST submit form
+runner->>runner-db: save Submission
+runner->>solidqueue-db: enqueue send submission job
+
+runner->>browser: HTTP 302 response
+browser->>runner: GET confirmation page
+runner->>browser: HTTP 200 response
+browser->>user: Display confirmation page
+
+```
+
+## Sending the submission email asynchronously
+
+```mermaid
+
+---
+title: Sending the submission email asynchronously 
+---
+
+sequenceDiagram
+
+autonumber
+
+participant worker as Forms runner worker
+participant solidqueue-db as Solid Queue database
+participant runner-db as Forms Runner database
+participant s3 as Amazon S3
+participant ses as Amazon SES
+participant sns as Amazon SNS
+participant inbox as Email inbox
+participant sentry as Sentry
+
+actor processor
+
+worker->>solidqueue-db: dequeue send submission job
+worker->>runner-db: get Submission
+worker->>s3: get file(s)
+worker->>ses: send email
+
+break error 
+  alt AWS SDK error and max retries not reached
+    worker->>solidqueue-db: schedule retry
+  else
+    worker->>sentry: send error
+  end
+end
+
+ses->>worker: return message_id
+worker->>runner-db: set mail_message_id on Submission
+worker->>runner-db: set the last_delivery_attempt timestamp on Submission
+
+ses-)inbox: send email
+note over ses,inbox: happens some time later
+
+alt email sent successfully
+  ses->>sns: send delivery notification
+  processor->>inbox: get form from inbox
+  processor->>processor: process form
+else email bounces
+  ses->>sns: send bounce notification
+  note over ses,sns: We have an SQS queue subscribed to the<br/> SNS topic and a recurring task to poll<br/>the SQS queue.
+end
+
+worker->>worker: Recurring task deletes Submissions<br/>that haven't bounced after 7 days
+```
+
+## Handling email bounces/complaints
+```mermaid
+
+---
+title: Handling email bounces/complaints
+---
+
+sequenceDiagram
+
+autonumber
+
+participant worker as Solid Queue worker
+participant solidqueue-db as Solid Queue database
+participant runner-db as Forms Runner database
+participant sqs as Amazon SQS
+participant sentry as Sentry
+
+actor support as Forms team tech support
+
+worker->>solidqueue-db: enqueue recurring receive bounces job
+worker->>solidqueue-db: dequeue receive bounces job
+worker->>sqs: get messages from bounces and complaints queue
+alt there is a bounce SQS message
+  worker->>runner-db: get Submission by the message_id in the SQS message
+  worker->>runner-db: update delivery_status of Submission to "bounced"
+  worker->>worker: Log with the submission details
+  worker->>sentry: send error event
+  sentry->>support: Alert via Slack
+  support->>support: Identify why the email bounced
+  support->>support: Run rake task to retry submission
+else there is a complaint SQS message
+  worker->>runner-db: get Submission by the message_id in the SQS message
+  worker->>worker: Log with the submission details
+end
+
+```
+
+## Handling successful deliveries
+
+```mermaid
+
+---
+title: Handling successful deliveries
+---
+
+sequenceDiagram
+
+autonumber
+
+participant worker as Solid Queue worker
+participant solidqueue-db as Solid Queue database
+participant runner-db as Forms Runner database
+participant sqs as Amazon SQS
+
+actor support as Forms team tech support
+
+worker->>solidqueue-db: enqueue recurring receive deliveries job
+worker->>solidqueue-db: dequeue receive deliveries job
+worker->>sqs: get messages from deliveries queue
+worker->>runner-db: get Submission by the message_id in the SQS message
+worker->>worker: log a "form_submission_delivered" event
+note over worker,runner-db: we don't currently use the "delivered" status for anything other than for information
+```