Add another answer data model

[SamJamCul]

Summary

Adding another answer is an upcoming feature for Forms. This will require backend changes in order to build, run and store questions that can have multiple answers.

Problem

Currently, we only support answering a form's questions once per submission.

We want to allow forms to include

• single questions that can be answered multiple times
• sets of multiple questions that can be answered multiple times

Questions will have a minimum and maximum number of answers that can be given, which the form builder will specify for each question or set of questions.

Question sets can take place at any point within a form.

We want to support routing to the start of a question set, and skipping over a question set. Routing will not lead into the middle of a question set, or from within a question set to the outside. We will likely want to also support routing between questions within a question set.

The current data structure for a form in the API is roughly:

erDiagram
  FORM ||--o{ PAGE : "form has many pages"
  FORM { 
    integer id PK
    string name
    string form_slug
  }
  PAGE
  PAGE { 
    integer id PK
    string question_text
    integer position
    integer next_page_id FK
  }
  CONDITION }o--|| PAGE : "page has many conditions"
  CONDITION { 
    integer id PK
    integer check_page_id FK
    integer goto_page_id FK
    string answer_value
  }

Loading

A PAGE contains question data and its position within the form.

A PAGE can have one or more CONDITIONs, which are used to describe conditional routing from that page.

Proposal

Options

Option 1: Separate out "steps" from "questions" and "question sets"

This option separates pages into "Steps" and "Questions", with a new "QuestionSet" type representing a set of questions. A "Step" will either point to a "Question" or "QuestionSet" using the polymorphic associations feature of Rails.

flowchart TD
    Form --- Step1(Step)
    Form --- Step2(Step)
    Form --- Step3(Step)
    Step1 --- Question1(Question)
    Step2 --- QuestionSet(Question Set)
    Step3 --- Question2(Question)
    QuestionSet --- Step4(Step)
    QuestionSet --- Step5(Step)
    Step4 --- SetQuestion1(Question)
    Step5 --- SetQuestion2(Question)

Loading

Forms-api database schema

erDiagram
  FORM ||--o{ STEP : "form has many steps"
  FORM { 
    integer id PK
    string name
    string form_slug
  }
  STEP ||--|| POSITIONABLE: "has one polymorphic positionable"
  STEP { 
    integer id PK
    integer form_id FK
    integer next_step_id FK
    string positionable_type "'Question' or 'QuestionSet'"
    integer positionable_id FK "FK to Question or QuestionSet" 
    integer parent_question_set_id FK "Only has a value if step is in a set"
    integer position
    integer min_answers
    integer max_answers
  }
  CONDITION }o--|| STEP : "step has many conditions"
  CONDITION { 
    integer id PK
    integer check_step_id FK
    integer goto_step_id FK
    string answer_value
  }
  QUESTION_SET ||--|| POSITIONABLE: "question set is a positionable"
  QUESTION_SET ||--o{ STEP : "question set has many steps"
  QUESTION_SET {
    integer id PK
    string name
  }
  QUESTION ||--|| POSITIONABLE: "question is a positionable"
  QUESTION {
    integer id FK
    string question_text
    string answer_type
    jsonb answer_settings
    bool is_optional
  }

Loading

The new STEP entity represents the position of a question or question set in a form.

The new QUESTION entity represents the properties of a question for the STEP they belong to.

The new QUESTION_SET entity represents a set of STEPs that can be anwered multiple times. Each of the STEPs within a QUESTION_SET will have a QUESTION.

We use a polymorphic association on the STEP entity (the 'positionable') to establish a one-to-one association with either a QUESTION or a QUESTION_SET.

Steps have min_answers and max_answers values, which indicate how many times the question or set of questions can be answered.

JSON representation of a form used by Runner

This is the JSON structure for storing a live form that runner retreives from forms-api.

In the proposed structure, the pages field is replaced with steps. Each "step" in the steps array has a type field, which indicates whether it is a Question or a QuestionSet, and a data field which contains either a Question or QuestionSet object.

A QuestionSet object contains a steps field which contains an array of Step objects for that QuestionSet. These Step objects will always have a type: "Question".

{
  "id": 123,
  "name": "All question types form",
  "form_slug": "all-question-types-form",
  "start_page": 5,
  ...
  "steps": [
    {
      "id": 5,
      "next_step_id": 6,
      "position": 1,
      "min_answers": null, // would this have a value if the question can only be answered once?
      "max_answers": null,
      "type": "Question", // this tells us the "data" field contains a question object
      "data": { // this contains the Question attributes
        "question_text": "What is your name?",
        "is_optional": false
        ...
      },
      "routing_conditions": []
    },
    {
      "id": 6,
      "next_step_id": 7,
      "position": 2,
      "min_answers": 1,
      "max_answers": 5,
      "type": "QuestionSet", // this tells us the "data" field contains a question set object
      "data": { // this contains the QuestionSet attributes
        "name": "What addresses have you lived at in the past 3 years?",
        "steps": [
          {
            "id": 10,
            "next_step_id": 11,
            "position": 1,
            "min_answers": null,
            "max_answers": null,
            "type": "Question",
            "data": {
              "question_text": "What was your address?"
              ...
            }
          },
          {
            "id": 11,
            "next_step_id": 12,
            "position": 2,
            "min_answers": null,
            "max_answers": null,
            "type": "Question",
            "data": {
              "question_text": "What date did you start living at this address?"
              ...
            }
          },
          {
            "id": 12,
            "next_step_id": null,
            "position": 3,
            "min_answers": null,
            "max_answers": null,
            "type": "Question",
            "data": {
              "question_text": "What date did you stop living at this address?"
              ...
            }
          }
        ]
      }
    },
    {
      "id": 7,
      "next_step_id": null,
      "position": 3,
      "min_answers": 1,
      "max_answers": 10,
      "type": "Question",
      "data": {
        "question_text": "What countries have you travelled to in the past year?",
        "is_optional": false
        ...
      }
    }
  ]
}

Runner session data structure

When a user is filling out a form, we store their completed answers in a session stored in a Redis database. The session currently has the following data structure:

{
  "session_id": "37e7691c31a61c0169469ebcf2441f34",
  "answers": {
    // at the root level of the answers, the keys are the form IDs
    "123": {
      // at the next level, the keys are the page IDs
      "5": {
        // each page contains key value pairs for the question's form fields
        "date_day": "1",
        "date_month": "1",
        "date_year": "1900",
        "date": "1900-01-01"
      },
      "6": {
        "full_name": "John Doe"
      }
    }
  }
}

For the new data model with question sets we think the data structure of the session would be something like:

{
  "session_id": "37e7691c31a61c0169469ebcf2441f34",
  "answers": {
    // at the root level of the answers, the keys are the form IDs
    "123": {
      // at the next level, the keys are the step IDs for the question or question set
      "5": {
        "full_name": "John Doe"
      },
      "6": {
        // question sets have a nested object with keys corresponding to the step IDs for its questions
        "10": [
          // value for each question is an array with an entry per answer provided
          {
            "address1": "The Whitechapel Building",
            "postcode": "E1 8QS"
          },
          {
            "address1": "Aviation House",
            "address2": "WC2B 6NH"
          }
        ],
        "11": [
          {
            "date": "2018-01-01"
          },
          {
            "date": "2015-01-01"
          }
        ],
        "12": [
          {
            "date": ""
          },
          {
            "date": "2018-01-01"
          }
        ]
      },
      "7": [
        // for single questions that can have multiple answers, the value is an array with an entry per answer given
        {
          "text": "Spain"
        },
        {
          "text": "Italy"
        }
      ]
    }
  }
}

Pros

• The nested "steps" matches quite well with how we might present the questions/sets in the question list in Forms-admin.
• Introducing the idea of a "Step" that can be of different types might allow for us to add steps to a form that aren't questions, for example adding a payment page and allowing its position in the form to be customised.

Cons

• The idea of polymorphism has quite a large cognitive overhead. The data model could be difficult for developers to understand.
• This data structure would require quite a big migration that has lots of moving parts. It could take a long time to implement and might be risky.
• The API between forms-api and form-admin will need to change significantly.
• There is some complexity to serializing and deserializing the JSON representation of a form when dealing with polymorphism.

Option 2: Keep pages holding their position in a form, whilst also being able to assign them to sets

This option keeps the "Pages" model largely the same, with the position remaining a property of a page. A page can have a "question set" associated with it, which holds information about the set it is in, such as the name and minimum and maximum answers.

flowchart TD
    Form --- Page1(Page)
    Form --- Page2(Page)
    Form --- Page3(Page)
    Form --- Page4(Page)
    Form --- Page5(Page)
    Page2 --- Set1(Question Set)
    Page3 --- Set1(Question Set)
    Page4 --- Set1(Question Set)

Loading

Forms-api database schema

erDiagram
  FORM ||--o{ PAGE : "form has many pages"
  FORM { 
    integer id PK
    string name
    string form_slug
  }
  PAGE
  PAGE { 
    integer id PK
    string question_text
    integer next_page_id FK
    integer position_in_form "Null if page is in a set and not the first page of the set?"
    integer position_in_set "Only has a value if page is in a set"
    integer min_answers
    integer max_answers
    integer question_set_id FK "Only has a value if page is in a set"
  }
  
  QUESTION_SET ||--o{ PAGE: "a page can belong to a question set"
  QUESTION_SET { 
    integer id PK
    string name
    integer min_answers
    integer max_answers
  }
  CONDITION }o--|| PAGE : "page has many conditions"
  CONDITION { 
    integer id PK
    integer check_page_id FK
    integer goto_page_id FK
    string answer_value
  }

Loading

In this model, we add a question_set_id foreign key to a PAGE. This is used to identify when a question is part of a question set.

The next_page_id will point to the next question in the form regardless of whether it is in a QUESTION_SET or not. For the last page in a set, the next_page_id will point to the next page after the set. This indicates that once a user has finished providing as many repeat answers to the question set as they need to, they will be directed to this page next. Runner will inevitably need more logic in addition to the existing routing logic to determine which page a user is shown next when sets are introduced.

When the position of a QUESTION_SET within a form is changed, we'd have to update both the next_page_id of the proceeding page, and also the next_page_id of the final page in the set.

Thepostion attribute of a PAGE has been replaced by position_in_form and position_in_set attributes. The position_in_form will be set if the question is not in a question set, or is the first page of a question set. The position_in_set attribute will identifty the position of a question within a set.

JSON representation of a form used by Runner

In the JSON structure, the pages array holds all pages in a form. All entries are at the same level, regardless of whether they belong to a set.

We identify pages in a set by the question_set_id. There is a separate question_sets attribute on the form object which is an array of all the question sets in a form.

An entry in the question_sets array has a first_page, similar to the start_page on a form, to easily identify the first page in the set. We could also store the last_page if this makes the navigation logic in Runner easier.

{
  "id": 123,
  "name": "All question types form",
  "start_page": 5,
  "pages": [
    {
      "id": 5,
      "next_page_id": 6,
      "position_in_form": 1,
      "min_answers": null,
      "max_answers": null,
      "question_text": "What is your name?"
    },
    { // this is the first page in a set
      "id": 6,
      "next_page_id": 7,
      "position_in_form": 2,
      "position_in_set": 1,
      "min_answers": null,
      "max_answers": null,
      "question_text": "What was your address?",
      "question_set_id": 1
    },
    {
      "id": 7,
      "next_page_id": 8,
      "position_in_form": null, // this is null because it is not the first page in the set
      "position_in_set": 2,
      "min_answers": null,
      "max_answers": null,
      "question_text": "What date did you start living at this address?",
      "question_set_id": 1
    },
    { // this is the last page in a set
      "id": 8,
      "next_page_id": 9,
      "position_in_form": null,
      "position_in_set": 3,
      "min_answers": null,
      "max_answers": null,
      "question_text": "What date did you stop living at this address?",
      "question_set_id": 1
    },
    {
      "id": 9,
      "next_page_id": null,
      "position_in_form": 3,
      "min_answers": 1,
      "max_answers": 10,
      "question_text": "What countries have you travelled to in the past year?"
    }
  ],
  "question_sets": [
    {
      "id": 1,
      "name": "What addresses have you lived at in the past 3 years?",
      "min_answers": 1,
      "max_answers": 5,
      "first_page": 6
    }
  ]
}

Runner session data structure

The runner session data structure should not have to change much, except the value associated with a page will be an array when the page is either a single question that can be answered multiple times, or belongs to a question set.

If we wanted to allow for questions within a set to be themselves answered multiple times per set of answers, we might have difficulties with this data structure.

{
  "session_id": "37e7691c31a61c0169469ebcf2441f34",
  "answers": {
    // at the root level of the answers, the keys are the form IDs
    "123": {
      // at the next level, the keys are the page IDs
      "5": {
        "full_name": "John Doe"
      },
      "6": [
        // value for each question in a set is an array with an entry per answer provided
        {
          "address1": "The Whitechapel Building",
          "postcode": "E1 8QS"
        },
        {
          "address1": "Aviation House",
          "address2": "WC2B 6NH"
        }
      ],
      "7": [
        {
          "date": "2018-01-01"
        },
        {
          "date": "2015-01-01"
        }
      ],
      "8": [
        {
          "date": ""
        },
        {
          "date": "2018-01-01"
        }
      ],
      "9": [
        // for single questions that can have multiple answers, the value is an array with an entry per answer given
        {
          "text": "Spain"
        },
        {
          "text": "Italy"
        }
      ]
    }
  }
}

Pros

• The migration for this change is small compared to option 1.
  • The database schema in forms-api doesn't change dramatically
  • The JSON representation of a form used by Runner does not change dramatically
• The API between forms-api and forms-admin will need to be added to, but we should be able to modify the existing API endpoints for adding pages rather than needing a completely new set of endpoints

Cons

• The logic that forms-runner needs to perform in order to determine which page to direct the user to next might be slightly less clear than Option 1. The next_page for the last page in a question set will not be the next page the user sees if they choose to add another answer.
• The data model doesn't map quite as well to what the user sees in forms-admin when creating a form compared with Option 1.

[chao-xian]

We have a bit of time before we need to conclude this by about mid-August-ish...

[stephencdaly]

Although I was involved in writing this, I should probably give my preference. I'm leaning towards option 2 because it seems like the changes will be far less drastic, so less likely to drag on and bring up major difficulties. Option 1 is conceptually quite nice, but in practice probably doesn't give us too much of an advantage over option 2. I think option 2 will also end up being easier to understand in the code, as the polymorphism introduced by option 1 isn't completely straightforward.

From what I can tell for option 2, there are very few changes that need to be made that will affect forms without question sets with the exception of a small migration for the position column. The changes to the datamodel used by runner are also a lot smaller.

[stephencdaly]

So in option 2, pages continue to represent questions. They'll have both a foreign key to the form and also optionally a foreign key to the question set they're in if they're in one. It's a many to one relationship for pages to question sets. So I think you'd phrase that as a page belonging to both forms and question sets?

Happy to chat through things if it doesn't make complete sense how we've written it out.

[chao-xian]

OH I see! Ok that makes more sense but you can understand why I was confused with the unclear naming... 🫠 Would it be worth considering renaming Pages as well?

[stephencdaly]

Calling them Questions rather than Pages would definitely make more sense to me. Is there some historic reason they were called Pages?

[chao-xian]

@thomasiles @DavidBiddle any recollection as the two longest serving devs on the team? Cc @lfdebrux

[DavidBiddle]

I think it was a combination of:

• 'one thing per page' by default meant that questions and pages were basically the same thing
• it left things open so that there could be a page without a question on it (i.e. a page that was just guidance or an exit page)

it's got a bit confusing though to the extent that there's already a card for untangling it: https://trello.com/c/604nINJO/157-figure-out-clean-up-the-difference-between-a-page-a-step-and-a-question

[lfdebrux]

I like the abstraction of having STEP objects; as well as being cleaner conceptually I think it would make future features easier, for instance having multiple questions on one page, or a page with just guidance on.

Would implementing steps be possible in a way that retains backwards compatibility (to ease the migration)?

[stephencdaly]

So for multiple questions on a page, we could end up with something that looks like a question set I guess, but behaves differently in runner. I wonder whether this could also be achieved with option 2. We could have a flag on a set indicating whether the questions should be on a single page. In that case it wouldn't make sense for pages to be called pages though 🤔.

I guess if it's a page with just guidance, option 1 is nice because we could have another "Positionable" type and this would work easily. We also considered that a payment page could be a step if we need to configure the position that is presented in the form.

In terms of backwards compatibility, I think it would be possible but it would be fairly fiddly with lots of steps. I'd envision migrating the database in forms-api, and initially mapping the new data structure onto the old data structure to support the APIs consumed by forms-runner and forms-admin. This migration might be a lot easier if forms-api and forms-admin were combined.

[jamie-o-wilkinson]

In my opinion option 1 is conceptually closer to what we are actually trying to model, which is nice. I find the structure of the JSON and session data much clearer at a glance. It does sound like the migration itself would be pretty involved though, but as others have mentioned perhaps it is worth us paying that cost now to make future features easier.

[chao-xian]

According to @SamJamCul there may be changes to this needed based on changes to design. We have some time as we are going to start with Add an "answer" to a single question.

[SamJamCul]

I've been messing around with another option, which I'd like to get some feedback on!

Option 3: Questions and Sets are equal tier steps

This is some sort of hybrid option, that makes use of the step by step construction - but keeps every question on the same tier, and uses sets as positioning elements for form journeys. There will be a need to be able to identify which questions are contained within a set, and this can be done either by associating them with the set, or by adding another object that represents the end of a set

Option 3a: Sets belong to questions

In this option, steps that come after a set will be connected back to that set, which can be used to determine the set details. When reaching a set, the runner can recognise that it needs to behave differently until the user progresses past the set's last step. I think the advantage here is that the runner more closely matches our mental model of how forms work, as each step should have information about what to do next.

flowchart TD
    Form --- Step1(Step 1)
    Form --- Step2(Step 2)
    Form --- Step3(Step 3)
    Form --- Step4(Step 4)
    Form --- Step5(Step 5)
    Step1 --- Question1(Question 1)
    Step2 --- Set(Set)
    Step3 --- Question2(Question 2)
    Step4 --- Question3(Question 3)
    Step5 --- Question4(Question 4)

Loading

Mermaid diagrams unfortunately don't keep the order of step objects, so joining the set to each contained question results in a messy diagram. Each step that falls within the set will have an attribute that keys it to that set.

I've draw up some extra lines in this image to show what I mean!

Option 3b: Sets have a start and finish step

In this option, the end of the set is demarcated by another object. This makes it extremely easy for the runner to recognise when it's reached the end of a set of questions, and it decouples questions within the set from the set itself. It's not completely necessary to do this though, but I like the idea of being able to shift the start and end points of the set to add and remove questions. I think this might be more palatable if the design changes to the extent that we're using sets more as marker points in the UI.

flowchart TD
    Form --- Step1(Step 1)
    Form --- Step2(Step 2)
    Form --- Step3(Step 3)
    Form --- Step4(Step 4)
    Form --- Step5(Step 5)
    Form --- Step6(Step 6)
    Step1 --- Question1(Question 1)
    Step2 --- Set(Set 1)
    Step3 --- Question2(Question 2)
    Step4 --- Question3(Question 3)
    Step5 --- Question4(Question 4)
    Step6 --- EndSet(EndSet 1)

Loading

[lfdebrux]

Why do we still need the next_step_id? 🤔

[SamJamCul]

I think giving the runner as much help as possible will make things easier, so showing the linked-list structure of steps very clear, even if it's also reflected in the position attribute, is something I'd be more in favour of.

[chao-xian]

I quite like the 3b option of being a hybrid - best of both worlds (does that also mean it's as few changes as 2? or a middle ground between 2 and 1?)

[SamJamCul]

It's more of a middle ground, as we'd still go through the process of building step objects for forms.

[stephencdaly]

From my understanding of this, it looks like it works in a similar way to option 2, but needing more work.

I think the advantage of option 1 is you have the questions clearly grouped in a set. If a set moves position in the form, only the set needs to be updated as the questions are unaware of their position in a form.

Option 2 has the disadvantage that questions have to be slightly more aware of their position in a form so a bit more logic will be required to deal with changing the position of sets, but the advantage that the schema doesn't need to be updated significantly.

[lfdebrux]

@SamJamCul what parts of option 1 depend on the design? I.e. what changes in the design could lead to a change in the data model?

[SamJamCul]

I was mainly thinking about how questions are added and removed from sets - if this is done by moving the set around as a window over a group of, say, 3 questions; then it will be a pain to shift questions out of the question set steps and into the higher level form steps. Not impossible, but I think clunky.

[stephencdaly]

I don't see it as being particularly clunky. In option 1, a step for a question in a set belongs to that set and has a position in the set. If it needs to be moved out of the set, we would need to change the parent of the step and update positions accordingly. Obviously there might be some slightly complex logic about this to ensure all the next steps and position fields are recalculated and updated appropriately, but conceptually I think it's fairly easy to understand.

[lfdebrux]

I've spent a while playing around with these options, to try and understand the pros and cons.

Method

I've coded up some example transformations in https://gist.github.com/lfdebrux/e9ff91c07935302383806f065672a2f0 to try and make things concrete. I tried converting an existing form document and a potential question set record into the different formats, and I then looked at how easy it would be to get data out of the object and/or manipulate it.

Note that the code in the gist is not production ready code or even particularly good; I tried as much as possible to keep all the logic in one function for each transformation because what I wanted to measure was how complicated the operation was, and being more DRY and abstract would have obscured that.

After writing this code and getting it working I analysed it using some of the metrics from RuboCop, here's the results:

Method	MethodLength	AbcSize	CyclomaticComplexity
to_option1	35	35.58	9
to_option2	13	21.21	6
to_option3a	37	40.61	9
to_option3b	47	50.21	10
option1__update_position_and_next_step_ids!	16	25.55	4
option2__update_position_and_next_page_ids!	26	32.45	9
option3a__update_position_and_next_step_ids!	9	12.21	3
option3b__update_position_and_next_step_ids!	9	12.21	3
option1_step	11	12.69	5
option2_page	2	3.74	2
option3a_step	2	3.74	2
option3b_step	2	3.74	2
option1_question_set	3	4.58	2
option2_question_set	2	3.74	2
option3a_question_set	2	3.74	2
option3b_question_set	2	3.74	2
option1_move_up	39	57.98	18
option2_move_up	45	76.69	20
option3a_move_up	35	67.61	19
option3b_move_up	39	73.61	21

Discussion

• It is simpler to move questions about in Option 1 than might be expected, especially because moving question sets is the same as moving a single question
• Retrieving a single question from Option 1 is complicated by the nested structure; however this could be solved by identifying questions by their index(es) rather than id
• In Option 2 question sets have to be identified differently to questions
• Option 2 is the simplest to create but has the most complex logic related to the position in set and position in form attributes
• In all options the book-keeping around position and linking to the next page/step is a significant source of complexity
• Whether or not we want to move questions into/out of a set or not affects the complexity of moving questions in Option 2 most

In Option 1 the information about whether a question is in a set is implicit in the structure of the object. The same is true in Option 3b, but the information is a little harder to determine computationally, so we need to rely on the extra linkage more.

It is important to note though that we do not necessarily have to use the same structures in forms-admin as we use for form documents sent to forms-runner; so moving questions or question sets will probably not be done in the way I've coded here. However I think it is a good proxy for the measuring the complexity of the object structure.

Conclusion

I think this analysis shows the advantage of making a bigger change to the form document structure; I think we should decide on either Option 1 or Option 3b, with a preference for Option 1.

[chao-xian]

This is paused as the feature that requires the next stage is being deprioritised for now. We will return to this when appropriate and leave open for further comments in the time being.