[Question] How to make an interactive image+3d gui?

[sweeneychris]

I was curious how you might make an application that combines image-based GUIs with the 3d scene GUI, for example a GUI that allows SAM prompts on an image with a 3d gui that shows a 3d point cloud points corresponding to segmented pixels. Any advice or pointers would be appreciated. Thanks for a great and very useful library!

[brentyi]

Hi Chris!

For the interactive image part, can you see if #383 has what you're looking for? You could use that to prompt SAM, project out the segmentation masks, and then visualize in 3D with scene.add_point_cloud().

This example also has some references for raycasting and working with clicks on 3D objects. For something hackier you could also put the image into the 3D scene and click on it with the 3D click tools.