Add scatterplots to C pages

[leburnett]

Fixes #78

• Adds interactive scatterplots to the "C" pages of types with synapses within ME, LO or LOP.

• The metrics are the mean of the values calculated individually for each optic lobe side (ME(R) and ME(L) etc.)

• When the page loads, the marker that represents the given type should automatically be highlighted.

• A marker will only be shown if that type has > 50 synapses (across all cells) and >5% of its total synapse number within that region. This is why some types have eyemap plots that are not empty for certain regions but they don't have a marker in the corresponding scatterplot.

• The cell type's marker will remain "highlighted" as you hover over other markers within the plot, but will reduce back to normal size if hovered over.

• One downside to how these plots are currently generated is that they will only be "complete" if pages for all types (or optic lobe types) have been generated (e.g. using pixi run neuview generate-all).

• The plots will be made if you run pixi run subset-medium etc. However, the scatterplots will only have a few markers representing the cell types that have pages.

[floesche]

Running pixi run subset-medium only shows 2 of the 3 scatter plots:

[leburnett]

Yes, this is because the only OL cell type in "subset medium" is Tm3 which only has synapses within ME and LO. As mentioned above, we can extend the medium subset or makes an OL subset with more types so that you can see more points within the scatterplots and a scatterplot for LOP as well.

[floesche]

Tm3 has synapses in LOP as the eyemaps show:

[leburnett]

Hmm, ok. I will look into this today.

[leburnett]

The scatterplot for Tm3 was not made for LOP because the number of synapses within LOP did not reach the threshold_high = self.threshold_service.get_roi_filtering_threshold( profile_name="roi_filtering_strict" ) value in cache_service.save_neuron_data_to_cache or there were fewer than 50 synapses in that region.

                                if clean_name in ["ME", "LO", "LOP"]:
                                    if (roi["pre_percentage"] >= threshold_high and (roi["pre"]+roi["post"])>50
                                        or roi["post_percentage"] >= threshold_high and (roi["pre"]+roi["post"])>50):
                                        include_in_scatter = 1
                                    else:
                                        include_in_scatter = 0
                                    entry["incl_scatter"] = include_in_scatter

I included this because we used the same thresholds in the paper. I have removed it from the last commit but we can put it back if we decide we want it. Now, all types with at least one column occupied in the ME, LO or LOP will be included in that region's scatterplot.

[floesche]

• Would it be possible to make the tooltip in the scatterplots look more similar to the ones in the eyemaps? Currently, the scatterplots look like this (gray background, small font, everything on one line):

The eyemap tooltip looks like this (dark gray or black background, bigger font, one line per information item):

• For the example of pixi run subset-medium (or subset-small), the Tm3 has scales on the LO and LOP scatter plots, but not on ME. Would it be possible to have scales there, too?

[leburnett]

Hi Frank! I think these changes should be incorporated in the most recent commits.
Some things that I want to highlight:

• I hard coded the axis limits to 1-1000. I was debating making this more flexible to be able to work with other datasets in the future, but these plots are only really useful for optic lobe analyses anyway so I think it's fine.
• I updated the "population size" value in the scatterplot to show the mean number of cells of the type per eye - not the total number of cells per type across both eyes because I think it makes more sense to think about these metrics on a per eye basis. I hope it is clear in the plots what the numbers correspond to, but let me know if you think this should be improved.

[floesche]

I added the scatter plots for L and R and did some layout changes. Let me know if you think they are improvements or if we should go back.
Also, the scale for the coverage factors goes from "n" to ">n" (e.g. 10 and >10 in the screenshot above). What does that mean?

[floesche]

Re

Also, the scale for the coverage factors goes from "n" to ">n" (e.g. 10 and >10 in the screenshot above). What does that mean?

I added a "subset-medium-optic" test set, and there it looks much better. Maybe not worth addressing that point…

[floesche]

I had noticed, that some cells had innervation in some ROI (e.g. C2 in LO), but the data still didn't show up in the plots. It turns out there was another thresholding in the code. Instead of just deleting it, I pulled it out to the config file (see min_col_count_theshold from 21434ed onwards) but set it to 0 for CNS. We can also remove it in the future, if that's not useful.

[leburnett]

I added the scatter plots for L and R and did some layout changes. Let me know if you think they are improvements or if we should go back. Also, the scale for the coverage factors goes from "n" to ">n" (e.g. 10 and >10 in the screenshot above). What does that mean?

Yes, I see your comment below this one as well. I think that's just a bug of only having one data point in the plot. I believe the 98th percentile value is used as the colour scale max and then the tick mark is changed to be ">cmax". I think if someone makes a single webpage and wants to look at these plots then they'll have the tooltip box information to help them understand the colour scale.

[leburnett]

I had noticed, that some cells had innervation in some ROI (e.g. C2 in LO), but the data still didn't show up in the plots. It turns out there was another thresholding in the code. Instead of just deleting it, I pulled it out to the config file (see min_col_count_theshold from 21434ed onwards) but set it to 0 for CNS. We can also remove it in the future, if that's not useful.

Thanks for catching this.

[floesche]

I had noticed, that some cells had innervation in some ROI (e.g. C2 in LO), but the data still didn't show up in the plots. It turns out there was another thresholding in the code. Instead of just deleting it, I pulled it out to the config file (see min_col_count_theshold from 21434ed onwards) but set it to 0 for CNS. We can also remove it in the future, if that's not useful.

Thanks for catching this.

It would simplify the code and configuration if this wasn't a parameter and we just always show what's in the database… Let me know what you think.