membrain-pick surfroama doesn't show the particles from .h5 file

[amineuron]

First of all, thank you for the great effort on all the membrain packages, I was able to train and predict points successfully.

That said, I noticed an issues:

When running membrain_pick surforama, the code fails because QtPointIO has no attribute _assign_orientations_from_nearest_triangles, which looks like a method that is not implemented.

details:
membrain_pick surforama --h5-path predict_output/Position_22_6_Position_22_6m16.h5 --tomogram-path ./Position_22_6.mrc
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ /home/amin/miniconda3/envs/membrain_all/lib/python3.9/site-packages/membrain_pick/cli/surforama_cli.py:74 in surforama │
│ │
│ 71 │ │ │ surforama_widget = initialize_surforama_widget( │
│ 72 │ │ │ │ points, faces, volume_layer, viewer, normal_offset=normal_offset │
│ 73 │ │ │ ) │
│ ❱ 74 │ │ │ display_cluster_centers( │
│ 75 │ │ │ │ viewer, mesh_data, pixel_size, surforama_widget, point_size=point_size │
│ 76 │ │ │ ) │
│ 77 │ │ else: │
│ │
│ /home/amin/miniconda3/envs/membrain_all/lib/python3.9/site-packages/membrain_pick/napari_utils/surforama_cli_utils.py:148 in display_cluster_centers │
│ │
│ 145 │ │ │ │
│ 146 │ │ │ surforama_widget.picking_widget.enabled = True │
│ 147 │ │ │ point_io = surforama_widget.point_writer_widget │
│ ❱ 148 │ │ │ initialize_points( │
│ 149 │ │ │ │ point_io=point_io, │
│ 150 │ │ │ │ point_coordinates=cluster_centers, │
│ 151 │ │ │ │ point_size=point_size, │
│ │
│ /home/amin/miniconda3/envs/membrain_all/lib/python3.9/site-packages/membrain_pick/napari_utils/surforama_cli_utils.py:107 in initialize_points │
│ │
│ 104 ): │
│ 105 │ │
│ 106 │ │
│ ❱ 107 │ normal_data, up_data = point_io._assign_orientations_from_nearest_triangles( │
│ 108 │ │ point_coordinates=point_coordinates │
│ 109 │ ) │
│ 110 │ features_table = { │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
AttributeError: 'QtPointIO' object has no attribute '_assign_orientations_from_nearest_triangles'

I could show these points on the surfaces how you have shown in the colab notebook, but can see them with membrain-pick

membrain-pick/src/membrain_pick/napari_utils/surforama_cli_utils.py

Line 9 in 75275c3

from surforama.gui.qt_point_io import QtPointIO

I also had question when we pick the points on extended Surfogram like (+7vx), the training consider this expantion while just got the mesh and points as an input?

Cheers,
Amin

[amineuron]

so the shortcut just to see the predicted points without orientation u can edit surforama_cli.py like this:

line 74-76

Instead of:

display_cluster_centers(
viewer, mesh_data, pixel_size, surforama_widget, point_size=point_size
)

Use:

display_cluster_centers_as_points(
viewer, mesh_data, pixel_size, point_size=point_size
)

this helps that u can see the output like colab,

now after visualization still the question remain that these point are on the inital mesh and if we use extenation of Surfogram then I dont know how this would be handled in training or predication cause now If we expand a bit the surface the points all covered under surface

[LorenzLamm]

Hi @amineuron ,
This is an issue I haven't managed to cleanly fix yet.
However, if you follow the surforama installation instructions here: Link
you should not encounter this issue anymore. The key here is to switch to the star_file_loading_adjustments branch before installation, where the _assign_orientations_from_nearest_triangles function is properly defined.

Please let me know if this works for you.

Regarding your question about mesh distances: MemBrain-pick's predictions will always be on the surface of the mesh. Internally, during training also points from extended meshes (i.e. further from the actual membrane surface) will be projected to the surface. This is a normalization step that is supposed to facilitate the localization task for the network.
If you would like to pick points that are further away from the surface (or particularly where the density of interest is further away from the surface), you can control the spatial extent the network can see with the step_number and step_size parameters in the convert_file command (for more info, see also here.

However, as mentioned, MemBrain-pick will always pick directly on the membrane surface. If you would like to get an offset from the membrane, I suggest that you could dilate the membrane segmentation before conversion to a mesh. Alternatively, you could also try to add an offset (given by the mesh normal vector) to each point.

I hope this helps.

Best,
Lorenz

[amineuron]

Hi Lorenz,

Thanks a lot for your detailed explanation! 🙏

I tried applying dilation maps before mesh conversion and then trained the network for 5 epochs, selecting the model with the best validation score. However, the predictions turned out quite strange it seems like the network is predicting points between the particles, almost as if the inverse of the score map would be correct. I’m not sure what might be causing this behavior.

Regarding the step_number and step_size parameters, I’m unsure how to adjust these properly in relation to the dilation. In my case, the proteins are quite small, and the tomograms are binned by 4 (voxel size ≈ 6 Å). After dilation by 8 vx, the surface thickness increases significantly (which effectively doubles the surface thickness on both sides 16 vx).

Do you have any suggestions or rules of thumb for setting step_number and step_size when dealing with such dilated surfaces? Also, would it be possible to dilate meshes and save the mesh directly, then picking on that? (cause dilatation is relatively slow on big tomograms)

Thanks again for your help!

Best,
Amin

[LorenzLamm]

Hi @amineuron ,

this is interesting. Did you take a look at the predicted score maps? So the points after clustering do not correspond to these predicted score maps?

I would always recommend training longer than 5 epochs btw - I got good results with something between 100-200 epochs. But if your scoremaps look good already after 5, even better :)

Regarding step number and step size: If you dilated the segmentation already to an extent where your particles are supposed to be, then you can choose step numbers centered around zero (e.g. -10 to 10). If you would like to go a bit further away from the membrane, you would need to lower this range (e.g. -20 to 0).
The extent of these steps is then defined via the step size parameter. So with step numbers -20 to 0 and step size 5, you will cover a distance of 10 (0.5 * 20) Angstrom away from the membrane, divided into 20 steps.

If your particles are very small, I guess you should pick a rather low step size here.

But it's always best to inspect how the inputs to the network actually look via the surforama interface: https://github.com/CellArchLab/membrain-pick/blob/master/docs/Surforama_Inspection.md#1-inspecting-membrain-pick-input

You can do this with the .h5 container you would feed into training / prediction. And then the rule of thumb should be: If you can see it well by eye, the model should also be able to pick it up.

I hope that helps :)