Integration + alignment of third-party cell segmentation masks

[pool-es]

I'm currently working in Stereomap with stereoseq data, and am trying to optimize the nuclei segmentation using a cell segmentation mask that was generated by applying StarDist on the ssDNA staining image.

You were kind to share a script that aligns the sizes/orientation of the ssDNA/DAPI images and a third party segmentation mask recently: #254

Unfortunately however, for this project, I am running into two issues:

1. I can't get the StarDist mask to align to the cells in Stereomap. After running the script from issue Aligning DAPI/mask images to regist.tif #254 with the StarDist mask and the ssDNA stain loaded into the saw_count run, the StarDist mask is slightly off, see images below:

Nuclei segmentation mask generated by saw_count run:

Nuclei segmentation after using the StarDist mask, aligned to the ssDNA image using the script from issue #254:

I used this script for the mask:
/exports/nieromics-hpc/roan/saw821/saw-8.2.1/anaconda/bin/python /exports/nieromics-hpc/roan/saw821/saw-8.2.1/lib/register/register/main.pyc
-i /exports/nieromics-hpc/emma/Cell_segmentation/saw_realign/D04923E2_DAPI.tif,/exports/nieromics-hpc/emma/Cell_segmentation/saw_realign/Mask_images/D04923E2_mask_IF_test.tif
-o /exports/nieromics-hpc/emma/Cell_segmentation/saw_realign -w False
--sn D04923E2
-v /exports/nieromics-hpc/emma/Cell_segmentation/saw_realign/realigned_D04923E2/D04923E2_A/outs/feature_expression/D04923E2.raw.gef

2. I am running into issues regarding the resolution of the segmentation mask, which affects how the cells are segmented. Right now, the script from issue Aligning DAPI/mask images to regist.tif #254 requires the StarDist mask to have the same size as the ssDNA image loaded into saw count, which is quite low (16000x16000 pixels). Otherwise the image does not align to the .stereo file in StereoMap.
   However, this affects the resolution of the nuclei in the mask, and the 'borders' between nuclei, which are not as clear as in the masks generated by saw_count (resolution of 23000x23000), see images below:

Mask from saw_count run:

Mask from StarDist, in same resolution as ssDNA image used in saw_count run:

Is there any way to overcome these issues? Is there any way to easily align the mask to the .stereo file, preferrably without decreasing its resolution?
Many thanks for your reply & help!

[Jinghao-Chen]

Hi, could you try aligning the images for cell segmentation with the ssDNA images first, and then use regist.tif to perform cell segmentation with Stardist?