SCOPING: cap number of participants, condition counterbalancing, cap by criteria

[becky-gilbert]

Summary

This is a multi-phase plan to add a related set of researcher usability features:

• Cap the total number of responses. The study should pause automatically when it has this number of responses.
• Condition counterbalancing. Allow researchers to define a number of conditions, and we counterbalance assignment to those conditions. Researchers have access to the condition ID in the experiment.
• Cap the number of responses within conditions. Keep a running total of valid/complete responses within each condition and stop assigning a condition when the cap is reached.
• Cap the number of responses within a given criteria expression (e.g. age range).

These features are all related and so it would be more efficient to get them done all together. However, I'm grouping them into releaseable phases, so that we can decide at each phase whether to continue on to the next phase, release and continue, or release and pivot to other priorities.

Capping total number of participants

Related issue: #772

Question: Which responses count toward the total?

We need to decide how we will consider a response to be countable toward the study's total response count. There are a few straightforward ways to do this, but there are some issues with each of these.

• Participant must be eligible
  • Problem: When a study specifies that the child must not have participated in that same study, valid responses can be labelled ineligible because the participant started the study (even just the first trial) and then went back to do the whole thing. The result is that our response count might underestimate the number of valid responses because it incorrectly treats these responses as ineligible.
• Session must be complete
  • Problem: Sometimes participants get all the way through the study but don't do the exit survey, in which case they are marked as 'incomplete' but their data is still useable. The result is that our response count might underestimate the number of valid responses because it incorrectly treats these responses as incomplete.
  • Problem: In hybrid studies, participants do the consent and then are redirected to an external study. These studies are treated as 'internal' in our system, but the study flow usually does not include an exit frame, which means the responses are always marked as 'incomplete'. The result is that our response count might underestimate the number of valid responses.
  • Problem: For external studies, we do not have any information about whether the participant completed the study or not.
• Must be marked as complete by the researcher
  • Problem: This requires the researcher to continuously monitor the responses and actively mark each one before it is be counted towards the study total. If the researcher doesn't do this (e.g. because a lot of responses come in over an evening/weekend) then the study may collect additional responses beyond the cap. With this requirement, our response count would underestimate the number of valid responses at any given time.

Possible solutions

@mekline please let me know what you think about the question above and the solutions outlined here

• Session must not be a preview
• Participant must be eligible
• For internal studies, response must also be complete
• Responses that meet the criteria above are automatically counted toward the study's total, BUT the researcher can edit the include/exclude boolean associated with each response. This way, we can keep an automated running count of the response count without researcher input, but the researcher can correct this count as necessary.
• Responses for external studies are always treated as complete unless changed by researcher

With this solution, there is still a problem with hybrid studies because these are 'internal' but will always produce incomplete responses. Possible solutions:

• Add a 'hybrid' flag (boolean, defaults to false) to internal studies so that we know to ignore the response's completion status.
• Require researchers to include an exit frame in hybrid studies. (This would cause confusion since the exit survey asks for feedback on a study that the family hasn't done yet, so it would probably require new parameters for modifying the exit survey).

For external studies (and hybrid studies, if we add a flag for them), we will need to consider all responses as 'complete' until the researcher changes the status. We could address this limitation by adding an API endpoint that the researcher puts into the end of their external study, which sends the session ID back to our system with a completion code. This would prevent these studies from automatically pausing prematurely because incomplete responses have been counted toward its current response total.

To do

• In the study model/table, add a field for the response limit. The default value is empty/null, in which case there is no limit (same as status quo). Add max responses field to study #1831
• Add the response limit field to the study create/edit form. This value can be edited at any time, and can be changed without causing the study to be rejected (if already approved). Add max responses field to study #1831
• When a response comes in for that study and is marked as complete, check the study's response count and pause the study if the limit has been reached. Add max responses field to study #1831
• If the study is active and the researcher's edits cause the response count to reach/exceed the limit, then the site should:
  • Show a warning/pop-up stating that the response limit has been reached and the study will now be paused Pause study when max responses limit is reached #1832
  • Automatically pause the study Pause study when max responses limit is reached #1832
• If the study is inactive and the researcher tries to start it, but the response count has reached/exceeded the limit, then the site should:
  • Show a warning/pop-up stating that the response limit has been reached, and "please modify the response limit if you want to collect more responses" Pause study when max responses limit is reached #1832
  • Do not allow the study to start Pause study when max responses limit is reached #1832
• Whenever the study changes from active to paused due to reaching the response limit, a notification email should be sent to the researcher. Pause study when max responses limit is reached #1832

UI updates for studies with a response limit:

• Show the current valid response count and limit on the study details page Add max responses field to study #1831
• In the Individual Responses page, flag each response in the left-hand column of the table as one of: Preview, Valid/Accepted, or Invalid/Rejected/Disregard
• In the Individual Responses page, indicate the number of responses associated with each type.
• For the latter two categories, allow the researcher to change the status (click an icon or checkbox). This action should update all counts on the page.
• Allow researchers to filter the table by response status (valid/invalid/preview).

@mekline please let me know what you think about the above specs

Balancing condition assignment

Related issue: #677

• In the study create/edit form, add field called 'add conditions for counterbalancing'. This starts off empty/null. The researcher can add a value indicating the number of conditions that they want counterbalanced assignment for. A list will appear with one ID (number or letter?) for each condition, along with a text box where the researcher can enter their own corresponding condition label.
• Each response is assigned one of these conditions, and the condition value is passed into the experiment runner (just jsPsych to start with?)
• As a new response is created, the site will assign a condition by cycling through the condition list (either in order or randomly without replacement).
• In the experiment code, the researcher is responsible for checking the condition assignment for that session (e.g. const condition = chsData.getCondition()) and modifying the experiment as necessary for that condition.

The approach above is a reasonable solution for counterbalanced assignment and would likely produce similar response counts per condition. But this just counterbalances at the start of the session, and it does not take into account the true condition counts that are based on the completed/valid responses.

Tracking/capping response counts based on criteria/conditions

Related issue: #772

• In the study create/edit form, add field called 'add a response limit/target based on criteria or conditions'. This starts off empty. The researcher clicks to add a new condition and/or criteria expression (e.g. age range), along with a response limit.
• If the researcher adds a single criteria-based response limit, then the study's total response limit will automatically set to the same number. If they add additional criteria-based limits, then the study total response limit will be the sum of all criteria-based limits.
• The researcher can edit the condition/criteria/limit values at any point, add new ones, or delete existing ones.
• As a new completed response comes in for the study, the site checks to see if any of the condition and/or criteria-based response limits have been reached. If so, that condition will not be assigned and/or any child who meets this criteria expression becomes ineligible.
• Whenever the site needs to determine a child's eligibility (e.g. on the participation page, when sending announcement emails, on the study list page), it will take any criteria groups/limits into account.
• One a participant clicks 'participate now' and a new response is created, the site will look at the set of conditions (that this child is eligible for) and associated counts, and assign the condition with the smallest number of valid responses. If there are multiple conditions with the smallest value, it will select at random.

Questions:

• What if the criteria expressions overlap, in which case a single response might be counted toward both totals? And if the response limit is reached for one criteria expression, what happens if a participant meets that criteria and another one that is still under the limit?
• The implementation of dynamic study eligibility might be tricky. For instance, if a study has some kind of global criteria expression (e.g. age 4-6), as well as limits on other subsets of that criteria (age 4-4.5), and the limit is reached for the subset, then the negation of the subset eligibility would have to be appended to the global eligibility, e.g. "is 4-6 years old" AND "is NOT 4-4.5 years old". This will probably work but needs clear documentation, as it could cause unexpected behavior if, for instance, a researcher includes global criteria in a subset (e.g. "age 4-4.5 AND speaks English", in which case all English-speaking children would be excluded when the 4-4.5 age range response limit is reached).