Design Decision on Representation of Samples

[golharam]

How do we handle sample metadata and fastq files?

Examples of use-cases:
User navigates to project page, the following scenarios exist (ignore existance or lack-of fastq files):
1. Project has no sample metadata
2. Project has samples from demuxed flowcells (hence they have fastq files)
3. Project has samples from vendor (hence they have fastq files)

Now,
1. When a user uploads a samplesheet - what happens?
2. When we auto-generate a samplesheet - this should scan the bucket for fastq files and create a list of samples corresponding to fastq files.
a. What happens where there are multiple sets of fastq files that can be associated with a sample? Do we associate them all to a single sample?

What are the pre-requisites or requirements we are sticking to for a sample to exist? Does a sample need to have fastq files to exist? Or can a sample exist without fastq files?

[golharam]

Tagging @EricSDavis and @accumb3ns for input.

[golharam]

One thing Steve and I discussed was when a samplesheet (with metadata) is uploaded, we can display a git diff style page showing what the changes are. Changes to metadata are in Yellow, deletions in red, additions in Green. This gives the user an opportunity to review changes before accepting.

[golharam]

If samples must have fastq files to exist, then what happens when we have multiple sets of fastq files? Can a sample have multiple sets of fastq files? For Single-Cell data, yes. What about when an a physical sample is sequenced multiple times, e.g. for increased depth of sequencing? Or what happens when we have a set of clipped and unclipped fastq files for a sample?

Additionally, should files be a separate resource associated with samples?

[accumb3ns]

1. Allowing sample metadata to exist with no associated files
   PRO: User can upload metadata for samples that (they believe) will exist in the future.
   CON: This may not be true - examples:

• Biosample was sent for sequencing but library prep failed. Common
• Internally sequenced run generated no usable reads or wrong set of indexes was input
• Vendor delivered sample under a totally different name; now we have 2 sets of records

I think there's a lot of potential for confusion -

• It can be misleading if no data has been delivered since it appears to the user that the samples are "in the project"
• Mismatches/missing data are not immediately obvious, eg 200 samples appear in the project but samplesheet for running only has 198

In contrast, I don't think it's too onerous to mandate that some files for a sample to exist before the user uploads metadata. Then, record creation occurs at the time the files are copied into the bucket, and we are just updating the record with the user's metadata.
At a high level - do we want this to be a store of all sample metadata or a more focused set of metadata attached to data in NGS360's data layer? I'm ok with the latter.

[golharam]

Assuming we require every sample that exists in the database to have associated fastQ files, how will the sample sheet the user uploads look like if a sample contains 10 sets of fast Q files (eg scrna-seq)?

Perhaps we auto generate a sample sheet based on fast Q files already present, then require the user to modify the auto generated sample sheet.

[EricSDavis]

I think it should be allowed to have duplicate "sample ids" as those define the biological identity of the sample. Each sample id within a project should be unique to differentiate them from one another. The composite of attributes attached to that sample should define cases such as technical replicates (re-sequencing for more depth) or alternative FASTQs (trimmed/clipped reads). Therefore, we should disallow samples with exact duplicate attributes.

Since samples can be added from external sources (not necessarily through NGS360), it should be allowed to have samples with no corresponding FASTQ file. However, we should clearly flag these in the table with a warning or highlighted row to indicate that there is no FASTQ so users can correct this or be aware that there is something wrong. This should take care of the scenarios @accumb3ns described above.

For autogenerating samples, NGS360 can scan the bucket and generate a temporary sample table that is downloaded (but not added to the database). The user can sort out the metadata and include/exclude rows and upload the result back to NGS360 where the samples will then be added to the databse.

[EricSDavis]

One thing Steve and I discussed was when a samplesheet (with metadata) is uploaded, we can display a git diff style page showing what the changes are. Changes to metadata are in Yellow, deletions in red, additions in Green. This gives the user an opportunity to review changes before accepting.

This is a good idea

[EricSDavis]

Assuming we require every sample that exists in the database to have associated fastQ files, how will the sample sheet the user uploads look like if a sample contains 10 sets of fast Q files (eg scrna-seq)?

Perhaps we auto generate a sample sheet based on fast Q files already present, then require the user to modify the auto generated sample sheet.

It would have the same sample id Different sample ids for each sample and different attributes (include FASTQ file path) that differentiate them. Attributes can be added to show that they belong together.

Yes, use the FASTQ files present and have the user sort out how the samples should be named/represented.

[EricSDavis]

The ability to track "biologically distinct samples" across projects would be out of scope. This could be done with an interface for adding samples and associating files/metadata, but would potentially be too cumbersome for large-scale experiments with thousands of samples. The trade-off is improved flexibility