Robot on VR does not update automatically

[fredshentu]

I would like to thank you for open-sourcing this awesome project! It's a super cool concept.

I have successfully set up the Quest 3 app and Windows environment. When I move my left-hand controller, I can see the joint information updating and sending to the Quest 3.

ARCap/data_processing/quest_robot_module.py

Line 416 in d50affb

msg += f",{hand_q_feedback[0]:.3f},{hand_q_feedback[1]:.3f}"

I am confident that the VR IP/host IP I entered is correct. Additionally, when I press button A on the right controller, the robot in VR updates slightly.

Despite this setup, I'm encountering an issue where the joint information does not seem to be updating fully or as expected.

Do you know what could be the possible issue?

Thank you!

[Ericcsr]

Thanks for your interest! I am not super clear about your issue discription so here are some questions:

1. Which robot are you using: Gripper, Leap hand or bimanual?
2. When you mentioned joint information didn't update correctly, did you saw a green frame which indicate the system is indeed in data collection mode.
3. "joint information does not seem to be updating fully", does this indicate the virtual robot freeze and not tracking controller or it keep jittering but response to different controller positions.

[fredshentu]

sorry for the late reply I was busy with other project the past weeks.

1. I am using gripper, since I don't have the Rokoko glove, I just hardcode the left_positions to np.zeros([4,3]) - 0.03
2. if I press X it will show recording with a red box.
3. actually i print the arm_q arm_q, hand_q, wrist_pos, wrist_orn = quest.solve_system_world(wrist_pos, wrist_orn, hand_tip_pose) here, the arm_q is updating, just the vertual robot is freezing and will jitter if I press A

do you know what could cause this issue?

[fredshentu]

since the rokoko glove are pretty expensive it will be great if you can provide a test script which allows people to try this new type of visual feedback (like simply moving a robot) without further hardware integration. I feel I've removed the rokoko dependency but for some reason the robot does not render (can confirm that ik works and new robot arm_q is constantly sending to the quest3)

[Ericcsr]

I think the cause may be data_collection_server is waiting for rokoko update in rokoko.receive(), if you have a hard coded hand position you can comment out everything related to rokoko. I will also add support for running code without rokoko using quest’s hand tracker recently, hopefully it will be more accessible

[fredshentu]

I’ve already commented out everything related to rokoko. It would be fantastic if you could provide a script that allows users without a Rokoko glove to try the system. Thanks in advance!

[fredshentu]

I found something very interesting: the exact same code can run on one of my computer (robot update regularly) but for the other two the robot will simple not move. And if I press A the robot will keep jittering. I can see the computer side is sending changing qpos to the quest constantly, but the robot on the quest is frozen.

Have you seen similar behavior? Would love hear your suggestion.

[Ericcsr]

I created a branch no_glove with use quest's hand tracking to replace rokoko, so far collected data can result in successful replay on real robot, haven't tested policy training with it.