Interactivity and Widget Synchronization in trame-vtklocal

[aghouas]

From reading through the source code of vtkWasmSceneManager and attempting to add a custom interaction style on the server, it appears that the client-side part of this library is using a vtkWebAssemblyRenderWindowInteractor that is not kept in sync with the server-side RenderWindowInteractor. This makes it challenging to process click events on the server or utilize interactive VTK widgets.

Expected Behavior

• The client-side interactor should synchronize with the server-side interactor to allow event handling on the server.
• Click events on the mesh should be processable on the server to enable point and cell picking.
• Interactive widgets, such as vtkOrientationMarkerWidget, should function properly without requiring significant workarounds.

Current Behavior

• Click events are not easily propagated to the server, making it difficult to implement point and cell picking.
• VTK interactive widgets do not function properly due to the desynchronized interactors.

Questions

1. Is there any documentation or an example demonstrating a workaround for this issue?
2. Is this a known limitation that is being worked on?
3. Are there recommended approaches for enabling point and cell picking in a trame-vtklocal setup?
4. Is it possible to use the vtkOrientationMarkerWidget with the current architecture?

Use Case

I am trying to enable point and cell picking in my application, allowing users to click on a mesh and see the selected cells and points highlighted. Additionally, I would like to use the vtkOrientationMarkerWidget for improved navigation.

Any guidance or suggestions would be greatly appreciated!

[jourdain]

0. This is not an issue, the interactors should not sync automatically.
1. No, but you could add an example of vtkOrientationMarkerWidget inside here so we could look at it and make sure it works
2. Yes, and we are working on ways to enable method calls on the WASM side so we could enable picking.
3. Not yet, just some workaround
4. It should be possible, but you may need to use a nightly build of vtk like for that example. pip install "vtk==9.4.20250215.dev0" --extra-index-url https://wheels.vtk.org

[aghouas]

Thanks for the quick response.

Do you have a timeline for interactivity features such as picking?

In the meantime, I’m working on the following workaround:

1. Register an observer for LeftButtonPressEvent with the render window interactor on the client side.
2. Capture the click event and transform the client screen coordinates to server screen coordinates (it looks like the server render window defaults to 300x300).
3. On the server side, I pass the transformed coordinates to a CellPicker instance. However, I'm running into an issue here, which I suspect is due to the client and server cameras being out of sync.

Any insights on this would be greatly appreciated!

[aghouas]

• Not yet, just some workaround

If you could share some info on this work around that would be great!

[jourdain]

What you are doing is most of the gist. But like you said, you need to keep the camera in sync along with the view size.
So you can use the SizeObserver for synching the size and the camera listener for keeping the camera in sync.

[aghouas]

Got it thanks! So far I've been writing a custom client in react. I know how to add observer code there. However, how to I add an observer for LeftButtonPressEvent if I am using vuetify / writing my client code in python ?

[jourdain]

Does that example helps?

[aghouas]

Thanks, yes this was very helpful!

However, I am running to another small issue now. I would like to trigger cell picking when the left button is released. Unfortunately, it seems this event is not fired by the wasm render window interactor. It works for LeftButtonPressEvent but not for LeftButtonReleaseEvent. Any idea why that is?

{
               # this works 
                "LeftButtonPressEvent": {
                    "pick_point": {
                        "p1": (
                            rwi_wasm_id,
                            "EventPosition",
                        ),
                    },
                },
               # This does not :(
                 "LeftButtonReleaseEvent": {
                    "pick_point": {
                        "p2": (
                            rwi_wasm_id,
                            "EventPosition",
                        ),
                    },
                },
            }

[jourdain]

Not sure, do you mind sharing a test/small example showing the issue?

[jourdain]

So the deal is that the trackball style does not trigger LeftButtonReleaseEvent event but EndInteractionEvent instead.
So you will have to track if pressed is pending before processing it...

[aghouas]

A got it! thanks

[Mashiatmm]

Thanks for the quick response.

Do you have a timeline for interactivity features such as picking?

In the meantime, I’m working on the following workaround:

1. Register an observer for LeftButtonPressEvent with the render window interactor on the client side.
2. Capture the click event and transform the client screen coordinates to server screen coordinates (it looks like the server render window defaults to 300x300).
3. On the server side, I pass the transformed coordinates to a CellPicker instance. However, I'm running into an issue here, which I suspect is due to the client and server cameras being out of sync.

Any insights on this would be greatly appreciated!

Hi @aghouas,
how do you transform the client screen coordinates to server screen coordinates? Also a snippet of your implementation of this picking feature for trame vtklocalview python will be extremely helpful.

[jourdain]

You can find a picking example here